# Web crawler

### Web crawler

Crawl any public or authenticated website to ingest documentation directly from the web.

| Parameter                   | Description                                                                                                                                                             |
|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **URLs**                    | The starting URL(s) for the crawler (e.g., `https://docs.company.com`).                                                                                              |
| **URL prefix filters**      | The domain(s) from which data should be ingested. For example, if crawling from `docs.stripe.com/getting-started`, set the prefix to `docs.stripe.com`. Without this, the crawl job will not complete successfully. |
| **Enable recursive crawling**| Crawls all reachable links from the starting URLs. Strongly recommended.                                                                                              |
| **Include Images**          | Ingest and cite images found on crawled pages.                                                                                                                         |
| **Ignore URL parameters**   | Strips query parameters from URLs (e.g., `?ref=nav`). Strongly recommended to avoid duplicate page ingestion.                                                         |

**Optional parameters**

| Parameter                     | Description                                                                                                                                                                                             |
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Include / Exclude Regex**   | Regex patterns to include or exclude specific URLs during crawling.                                                                                                                                  |
| **Enable anchor tags**        | Uses HTML anchor tags to better segment page content, enabling more precise citations.                                                                                                               |
| **Ignore HTML classes**       | Class names for page elements to exclude from ingestion (e.g., navbars, footers).                                                                                                                     |
| **Add headers to crawlers**   | Custom HTTP headers for authenticated pages (e.g., `Authorization: Bearer <token>`).                                                                                                                  |
| **Enable dynamic crawling**    | Enables JavaScript rendering before crawling. Use for pages that load content dynamically.                                                                                                            |
| **Bypass Cloudflare verification**| Experimental option for Cloudflare-protected pages. May not always succeed.                                                                                                                  |
| **Enhance your data sources with knowledge graph** | Builds a knowledge graph from crawled pages to improve answer quality. Recommended when recursive crawling is enabled.                                                                           |