Skip to main content
The Profound Knowledge Base Fetcher is an HTTP agent that reads a website’s sitemap when you import that site into a Knowledge Base. It also re-downloads PDF files already in a Knowledge Base to check whether they have changed.

​
Summary

​
What it does

When you import a website with the Sitemap tab of Crawl Website, the fetcher requests the sitemap URL you enter and any child sitemaps listed in a sitemap index. Profound queues the page URLs it finds for import. The fetcher only reads sitemap files, so it runs when the Sitemap URL ends in .xml or .xml.gz. If you enter a plain domain instead, Profound finds the site’s pages another way and this fetcher isn’t involved. You can leave out https://, and the fetcher adds it. If the first attempt finds no pages and the address has no www., the fetcher tries again with www. added. With auto-sync on, the fetcher reads the sitemap again on every sync run to pick up new or removed pages. It also downloads each PDF that was imported from the web on that cadence and compares a fingerprint of its bytes with the previous version. Profound processes only the files that changed.

​
What it doesn’t do

  • Fetch HTML pages, images, scripts, or stylesheets. Page content is fetched separately
  • Read robots.txt
  • Submit forms, follow login flows, or access authenticated areas
  • Run JavaScript or use a headless browser
  • Reach private or internal network targets

​
Request behavior

​
Headers sent

PDF fingerprint requests send Accept-Encoding: identity instead.

​
How to identify the bot

Use the exact User-Agent header:
All Profound bots use User-Agent strings that start with Profound, so a prefix match on Profound is a good way to identify all Profound traffic, current and future.

​
How to block

Block the fetcher using any of the following methods:
  • A web application firewall (WAF) or content delivery network (CDN) rule matching User-Agent containing ProfoundKnowledgeBase
  • A WAF or CDN rule matching User-Agent containing the prefix Profound

​
Reporting abuse

If you observe behavior that doesn’t match this documentation, such as requests for HTML pages, request rates that look like a crawl, or activity from an unrecognized Profound* User-Agent, report it to security@tryprofound.com with example log lines. Profound treats these reports as security issues and responds to all of them.