Finding documents automatically for an AI research platform, with a person still in charge

The problem

A market-intelligence consultancy runs an AI research platform on Azure. Its clients ask questions in plain language and receive answers grounded only in a curated library of trusted documents. Another development partner built the chat interface and the API; we built one component, the service that finds documents and brings them into that library.

Before it existed, the library grew one uploaded file or one web address at a time. Analysts had no systematic way of finding new material across the sites they trust, and no way to check whether what came in was worth keeping.

The risk

The quick version of this tool is a crawler that fetches everything and asks a model whether each page is relevant, and that version has some serious problems. A crawler that will follow any address it is given can be pointed at internal systems (an attack known as server-side request forgery, or SSRF). A model call on every page has no natural ceiling on cost. And nobody can tell whether the crawler is finding what it should, because a crawl that quietly misses most of a site looks exactly the same as a site with very little on it.

For a platform whose whole promise is trustworthy answers, a library filled by unchecked automation would undermine the product.

The decision

We treated discovery as something to measure rather than assume. The service finds documents by several independent routes (site crawls and sitemaps, RSS feeds and the Internet Archive), so that one route can check another. That turned out to matter a great deal: on one government site, a live crawl found no PDFs at all, while the archive found more than 1,600.

We split the AI’s judgement into two separate scores, one for document quality and one for topic relevance, and only let the system approve or discard anything on its own when both agree. Anything less certain goes to a person. Analysts edit the classification policy themselves, in plain English, and re-judging reuses documents that have already been collected, so changing their minds does not cost them a fresh crawl.

Around all that, we added the safeguards a first version usually lacks. There is an SSRF guard that resolves and pins each address, a daily limit on AI spending for each workspace, and polite fetching that respects robots.txt (overriding it is a recorded decision). Managed identity means no API keys are deployed, and the code refuses to start if a live search index has been configured as a test target.

The outcome

The library now grows through systematic discovery, with confidence intervals and recall estimates showing how complete it is, and a person still makes the final call on anything ambiguous. More than 300 automated tests must pass before every deployment. A side-by-side comparison is also built into the service, which shows numeric questions answered correctly from tables through SQL in cases where document retrieval confidently returns the wrong figures.