Interface WebCrawlerLimits.Builder

  • Method Details

    • maxPages

      WebCrawlerLimits.Builder maxPages(Integer maxPages)

      The max number of web pages crawled from your source URLs, up to 25,000 pages. If the web pages exceed this limit, the data source sync will fail and no web pages will be ingested.

      Parameters:
      maxPages - The max number of web pages crawled from your source URLs, up to 25,000 pages. If the web pages exceed this limit, the data source sync will fail and no web pages will be ingested.
      Returns:
      Returns a reference to this object so that method calls can be chained together.
    • rateLimit

      WebCrawlerLimits.Builder rateLimit(Integer rateLimit)

      The max rate at which pages are crawled, up to 300 per minute per host.

      Parameters:
      rateLimit - The max rate at which pages are crawled, up to 300 per minute per host.
      Returns:
      Returns a reference to this object so that method calls can be chained together.