> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tryprofound.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Profound Knowledge Base Fetcher

> What the bot fetches, when it runs, and how site operators can identify or block it

The Profound Knowledge Base Fetcher is an HTTP agent that reads a website's sitemap when you import that site into a [Knowledge Base](https://docs.tryprofound.com/context-manager/manage-knowledge-bases.md). It also re-downloads PDF files already in a Knowledge Base to check whether they have changed.

## Summary

| | |
| - | - |
| **User-Agent string** | `ProfoundKnowledgeBase/1.0 (+https://docs.tryprofound.com/bots)` |
| **Trigger** | User-initiated sitemap imports, and auto-sync when enabled for an imported folder or page |
| **Schedule** | Only on the auto-sync cadence you choose for an imported folder or page: daily, weekly, or monthly. Off by default |
| **Resources fetched** | Sitemap files and child sitemaps they reference, and PDF files already imported into a Knowledge Base |
| **Network origin** | AWS, no fixed IP allow-list available |

## What it does

When you import a website with the **Sitemap** tab of **Crawl Website**, the fetcher requests the sitemap URL you enter and any child sitemaps listed in a sitemap index. Profound queues the page URLs it finds for import.

The fetcher only reads sitemap files, so it runs when the **Sitemap URL** ends in `.xml` or `.xml.gz`. If you enter a plain domain instead, Profound finds the site's pages another way and this fetcher isn't involved.

You can leave out `https://`, and the fetcher adds it. If the first attempt finds no pages and the address has no `www.`, the fetcher tries again with `www.` added.

With auto-sync on, the fetcher reads the sitemap again on every sync run to pick up new or removed pages. It also downloads each PDF that was imported from the web on that cadence and compares a fingerprint of its bytes with the previous version. Profound processes only the files that changed.

## What it doesn't do

* **Fetch HTML pages**, images, scripts, or stylesheets. Page content is fetched separately
* **Read `robots.txt`**
* **Submit forms**, follow login flows, or access authenticated areas
* **Run JavaScript** or use a headless browser
* **Reach private or internal network targets**

## Request behavior

| | |
| - | - |
| **Per request timeout** | 20 seconds |
| **Redirects** | Followed manually, up to 5 hops |
| **Concurrency** | Sitemap files are fetched one at a time. During auto-sync, up to 10 documents sync at once, so up to 10 PDF downloads can run in parallel |
| **Sitemap limits** | Up to 50 sitemap files and 3 levels of nesting per import, 20 MB per file |
| **PDF limit** | 200 MB per file |
| **Caching** | None. Each import or sync issues fresh requests |

### Headers sent

```txt wrap theme={null}
User-Agent: ProfoundKnowledgeBase/1.0 (+https://docs.tryprofound.com/bots)
Accept-Encoding: gzip, deflate
```

PDF fingerprint requests send `Accept-Encoding: identity` instead.

## How to identify the bot

Use the exact User-Agent header:

```txt wrap theme={null}
User-Agent: ProfoundKnowledgeBase/1.0 (+https://docs.tryprofound.com/bots)
```

> [!TIP]
>
> All Profound bots use User-Agent strings that start with `Profound`, so a prefix match on `Profound` is a good way to identify all Profound traffic, current and future.

## How to block

Block the fetcher using any of the following methods:

* A web application firewall (WAF) or content delivery network (CDN) rule matching User-Agent containing `ProfoundKnowledgeBase`
* A WAF or CDN rule matching User-Agent containing the prefix `Profound`

## Reporting abuse

If you observe behavior that doesn't match this documentation, such as requests for HTML pages, request rates that look like a crawl, or activity from an unrecognized `Profound*` User-Agent, report it to [security@tryprofound.com](mailto:security@tryprofound.com) with example log lines.

Profound treats these reports as security issues and responds to all of them.