Web crawler
The Web Crawler tab indexes pages from a website into your Content Store, so an agent can retrieve knowledge from a public site alongside your own documents.
By the end of this page you will have set up a crawler, controlled its scope, and know how to run and manage it.
Prerequisites
- Admin or Developer role.
- A Content Store open on the Web Crawler tab.
Create a crawler
Click New Crawler to open the four-step wizard (Create Web Crawler - Step N of 4).
Step 1 — Name and URL
Enter a Crawler Name and the Website URL to start from.
Step 2 — Crawl settings
Set the scope and behavior:
- Crawl Depth — how many link levels deep to follow.
- Maximum Pages — a cap on how many pages to index.
- Respect robots.txt — honor the site’s crawl rules.
- Render JavaScript — use a headless browser for JavaScript-heavy sites (single-page apps).
Step 3 — URL patterns (optional)
Add Include Patterns to restrict the crawl to matching URLs, and Exclude Patterns to skip sections you don’t want.
Step 4 — Review
Review the configuration and create the crawler.
Run and manage a crawler
Each crawler shows controls and live progress:
- Start / Pause — run the crawl or pause it. Progress shows the percentage complete, the current URL, and pages crawled and indexed.
- Expand a crawler to see its crawled pages, each with an indexing status; remove a page if you don’t want it indexed.
- Use a crawler’s menu to Edit Settings or Delete Crawler.
Start with a shallow depth and a modest page cap, run it, then check the Search tab to confirm the store returns useful results before widening the crawl.
Common issues
- Fewer pages than expected — the crawl stopped at the Maximum Pages cap or Crawl Depth limit, or your Include/Exclude Patterns filtered pages out.
- A JavaScript-heavy page came back empty — enable Render JavaScript so the crawler loads the page in a headless browser.
- Pages you didn’t want were indexed — add an Exclude Pattern and reindex.
Next
- Search — test what the store returns.