What your agent knows
Your agent can only answer from what you have given it. This page covers where that comes from, how to choose what goes in, and why leaving things out is often the better decision.
Where knowledge comes from
Three routes, and you can mix them freely:
- Crawling your website. Point us at your address and we read your public pages. This is what most people use and it needs nothing from you but the URL.
- Uploading documents. PDF, Markdown, plain text or HTML. Useful for things that are not on your website: internal policies, the answers you currently paste into emails, a price list you have not published.
- Pasting text. The quickest way to add a single answer you keep having to give.
Anything you add is what the agent has. It does not know your business beyond this, and it does not go looking elsewhere. That is deliberate: an agent that invents plausible answers is worse than one that says it does not know and offers to fetch a human.
How the crawl finds pages
Two ways at once, so that neither one's blind spots matter much:
- Your sitemap. We look for it the way a search engine does: your
robots.txtfirst, then/sitemap.xml. If you have one, this is the most complete list of what you publish. - Your links. We start at the address you gave us and follow links within the same domain.
If you have no sitemap, nothing breaks; we simply follow links. If you do have one, we tend to find pages that are published but not prominently linked.
We only read pages on the same domain, and we leave a second between requests so we are never a burden on your server. Images, stylesheets and downloads are ignored: the agent reads words.
Choosing which pages to read
Under Knowledge, open Choose which pages and you get two boxes. One rule per line.
| Box | What it does |
|---|---|
| Skip these pages | Anything matching is never read. Everything else is. |
| Only read these pages | If you put anything here, only matching pages are read. Leave it empty unless you really want a narrow slice. |
Skipping always wins over including, so you can include a whole section and still carve one part out of it.
Writing a rule
Rules are matched against the part of the address after your domain.
| Rule | Matches |
|---|---|
| /blog/* | Everything under /blog/. The * is a wildcard. |
| cart | A rule with no wildcard matches anywhere in the address, so this catches /cart, /your-cart and /shop/cart/step-2. |
| /terms | Also loose, for the same reason: it catches /terms but also /terms-of-sale. If you meant only the one page, see the next row. |
| */terms | Only addresses ending in /terms. Adding a wildcard anchors the rule, so /terms-of-sale is left alone. |
| */tag/* | Any tag listing, wherever it sits. |
The wildcard is anchored, which is worth knowing: /daily/* matches
/daily/anything but not /holidays. If you want a loose
match, leave the wildcard out.
Preview before you commit
Next to the Crawl button there is a Preview. It reads your sitemap and shows what we would pull in, grouped by section, with a count for each. Nothing is read and nothing is stored.
Each group has a Skip these button that writes the right rule for you, so you do not have to work out the pattern yourself. The count updates as you go, so you can see the effect before pressing Crawl.
This is the step worth not skipping
Most surprises come from sites that publish far more than their owner pictures. Tag pages, author pages, paginated archives and dated posts add up fast, and they are rarely what customers ask about. Thirty seconds in Preview usually saves a great deal of room.
What to do, by type of site
A small business site, up to about 25 pages
Read everything. There is nothing to gain from filtering, and a page you thought was irrelevant is often the one someone asks about.
A site with a blog or news archive
This is the common case, and the one filters exist for. Hundreds of dated posts will drown out your delivery policy in the agent's memory, and they use up your plan. Skip the archive:
/blog/* /news/* */tag/* */author/*
Keep the posts only if customers genuinely ask about their content. "How do I re-season a cast iron pan" is worth keeping if you sell cookware and wrote a guide about it. Last year's company news is not.
A shop
Skip the machinery of buying, which changes constantly and answers nothing:
/cart* /checkout* /my-account* */page/*
Keep your delivery, returns, sizing and FAQ pages. Those are what people actually ask about, and they are usually a handful of pages carrying most of the value.
A documentation or support site
Use the second box for once. If your docs live under one path, read only those and let everything else go:
Only read these pages:
/docs/*
/help/*
A site with a members area
Anything behind a login is invisible to us anyway, since we read your site as a stranger would. What matters is skipping the login and account pages so the agent does not offer them as answers:
/login* /account* /dashboard*
More than one website
You can point one account at several websites. Each one you add is read separately and keeps
its own "skip these pages" and "only read these pages" boxes. Patterns are paths,
so typing /blog/ under one site affects that site only, even if another site on the
same account also has a /blog/. A crawl never wanders onto a different domain, so
there is no ambiguity about which site a rule belongs to: it belongs to the one whose box you
typed it in.
What they do share is the knowledge itself. Everything read from every site lands in one pool, and one agent answers on all of them. A visitor on your second site can therefore be given an answer drawn from a page on your first.
That is usually what you want when the sites are one business: a main site plus a documentation subdomain, or a shop plus a landing page. It is usually not what you want when they are unrelated businesses, because the agent will happily explain one to a customer of the other, and they share one name, one greeting and one monthly allowance. For unrelated sites, use a separate account for each.
How much your plan holds
Your limit is the total amount of text across everything you have added, crawled and uploaded together. It is not a page count, because pages vary wildly: fifty product pages might be smaller than three long guides.
| Plan | Knowledge | Roughly |
|---|---|---|
| Free | 2 MB | a few hundred ordinary pages |
| Starter | 15 MB | a substantial site with its documentation |
| Standard | 30 MB | a large site and a document library |
| Pro | 60 MB | more than most businesses will ever write |
The Knowledge page shows how much you have used. Only words count: images, styling and page furniture do not.
Running out is not the only reason to be selective. An agent searching a hundred relevant pages finds the right answer more reliably than one searching two thousand pages of which most are noise. Filtering is as much about accuracy as it is about space.
Staying up to date
We re-check your crawled pages on a schedule, weekly by default. You can set it to nightly, monthly, or off entirely under Knowledge.
Re-checking is cheap and invisible. Pages that have not changed are recognised and skipped, so a weekly check on a site that has not moved costs nothing and changes nothing. Your agent keeps answering throughout: there is never a moment where it has forgotten your site.
Your Monday digest tells you what happened, so you are not left guessing whether it ran.
Turn it off if your site is mid-redesign, if it is a staging copy, or if you have curated your knowledge by hand and do not want a crawl adding back what you deliberately trimmed.
Pages you delete
When you remove a page from your site, the next re-check removes it from your agent too. That matters more than it sounds: a refund policy you replaced six months ago is worse than no policy at all, because the agent will quote it confidently.
There is one deliberate exception. If your site is unreachable when we look, or pages return server errors, we remove nothing. A site being briefly down must never be mistaken for its pages having been deleted. In that case we simply try again on the next scheduled check.
If you want something gone immediately rather than at the next check, use Re-crawl on that source, or delete the source entirely and add it again.