A data source is what you can query: tables with fields from which Analytics Agent builds reports and the agent checks results. In Data Agent, sources need no manual setup: they appear on their own as soon as a ready database shows up in the data.
Automatic discovery
Every Lakehouse folder with a SQLite database file at its root becomes a source named {project}_{source} — for example, market_prices. The source points at the folder’s newest database and opens read-only: scripts write the data, queries never change it. If the database is deleted, the source disappears at the next rescan.
Data freshness
A rescan happens after every script run finishes in the source’s folder, and also when an unknown name is looked up. So fresh data that a script just collected is available for queries immediately — nothing needs a manual “refresh”.
Overrides and external databases
Sometimes a discovered source needs changing: point it at a different database, fix its title or settings — or connect a database that does not live in the Lakehouse. JSON files in the overrides folder (inside the /data volume) do this. One broken file does not break the registry: it is skipped with a warning, and the other sources keep working.
Querying the data
Today SQLite sources are queryable. Only read-only queries (SELECT) are allowed: the agent checks data after loading, and you use it in your analysis. The table schema is available right away — you can see which fields and types sit in every table.
The interface is a read-only dashboard
The Data Sources section lists the sources, each one’s details and its table count. Registration is the agent’s job — sources cannot be created or edited in the interface, and so cannot be accidentally broken either.
The point in plain words
Connecting a database used to mean configuring connections, access rights and formats. Here, none of that: a script builds a database in the Lakehouse, and the source is ready to use. One principle — “put a database in, get a source” — covers new tasks too.
Where a source’s material comes from
A source appears where a ready database lies, and a database is built by a script. The script usually takes its material directly — from APIs, files or exports. But when bulk collection is needed — hundreds of competitor pages, dozens of search queries, thousands of short texts — harvesters step in: they search the internet, download pages and process texts with models. HTML pages are saved as clean Markdown text, with the original page kept beside it.
Harvesters are separate services: configured once in the Harvesters section, launched by the agent. That way material is gathered fast and in volume, and the scripts turn it into Lakehouse data — and eventually into the sources reports are built on.
See also: Harvesters, Lakehouse, Runs, Data for Analytics Agent.