Introduction

Profound operates a small number of automated HTTP agents that fetch publicly available data from the web on behalf of our customers. This section documents each bot, what it does, when it runs, and how site operators can identify, allow, or block it.

​

Active bots

​

Bot operating rules

Every Profound bot follows the same operating rules:

• Public data only: Our bots fetch resources that are intended to be publicly readable (e.g. sitemap.xml, robots.txt, public HTML pages). They never attempt authenticated areas, never submit forms, and never bypass access controls.
• User-Agent identification: Every bot sends a stable, descriptive User-Agent header with a name that begins with Profound. Site operators can match on that prefix to identify all Profound traffic.
• robots.txt awareness: Our bots read robots.txt for any host they fetch from. Bots that crawl HTML pages honor Disallow directives.
• No persistent crawling without disclosure: Any bot that runs on a schedule, recursively follows links, or maintains an index has a dedicated page in this section documenting its behaviour in detail.