Fetching turns the links you found into material for analysis: pages are saved, converted to clean Markdown and laid out by date. For stubborn sites there are five fetch methods, escalating from a plain request to a full browser.
Five methods
- Direct HTTP — a plain request for the page: the cheapest and fastest way, works for most sites, but gives up where the content is drawn by scripts or a bot protection stands in the way;
- Proxied HTTP — the same request routed through a proxy (an intermediary server): helps when a site limits access by country or by request rate;
- Browser Emulation — a local browser (Playwright) on your server: loads pages with their JavaScript. The browser is downloaded automatically on first use; of the local methods it is the heaviest on resources;
- Nodriver — a local undetectable browser: the successor of undetected-chromedriver and the best local option for pages with bot protection. The Python package and Chromium download automatically on first use — nothing to install by hand;
- Remote Browser — a remote browser: an external browser-control service (the Playwright MCP or Zendriver protocol). Best suited for sites with bot protection and heavy JavaScript.
The methods are connected as services in the Services section. When a page cannot be fetched with one method, the system tries the next one by priority on its own — the usual chain goes from simple to stubborn: Direct HTTP → Proxied HTTP → Browser Emulation → Nodriver → Remote Browser. Getting around blocks and bot protections happens without the user’s involvement.
Conversion to Markdown
Every fetched page is converted to clean Markdown with Trafilatura: the content stays, while navigation, ads and markup go away. Everything downstream — chunking, extraction, reading — works on that text, not on the HTML.
Storage
Files are laid out by date: {RESEARCH_AGENT_FETCH_FOLDER}/yyyy-MM-dd/{domain}_{slug}_{hash8}.md — by default /data/fetch on the persistent volume. The database stores each file’s absolute path and size, and the interface serves the content through a protected file endpoint — the Markdown can be read right in the app.
Fetched pages are temporary history, not an archive: a background sweeper deletes them two days after the fetch (the window is configured with the RESEARCH_AGENT_FETCH_RETENTION_HOURS variable). Only the finished reports in the library live longer — they are what all of this is for, so they are the thing worth keeping.
Fetch sessions
The Fetches section shows fetch sessions: statuses, history and results, re-runs. A long batch can be stopped — the stop button sits on the session and on each of its rows; the app asks for confirmation first, and the pages already downloaded stay. A report in the library shows its search sessions and downloaded sources — tracing where each fact came from takes one click.
Access from external agents
Fetching is available from the outside via POST /api/agent/fetch with the X-API-Key header — external agents get the same pipeline with fallback methods and Markdown conversion.