---
title: "Harvesters"
id: "1801"
type: "page"
slug: "harvesters"
published_at: "2026-09-20T21:51:53+00:00"
modified_at: "2026-09-21T01:36:28+00:00"
url: "https://xedant.com/agents/data/docs/harvesters"
markdown_url: "https://xedant.com/agents/data/docs/harvesters.md"
excerpt: "Harvesters are data collectors: the services the agent uses to search the internet in bulk,…"
---

# Harvesters

[https://xedant.com/agents/data/docs/harvesters.md](https://xedant.com/agents/data/docs/harvesters.md)

Harvesters are data collectors: the services the agent uses to search the internet in bulk, download pages and process texts with models. The name comes from the product’s interface; the idea is simple — instead of walking sites one by one, the agent launches one big harvest, and Data Agent itself watches the pace, the sites’ limits and the cost accounting.

## Why you need it

A regular export takes data from one place: an API address, a table in a database, a file on a site. Bulk collection is when there are many pages, queries or texts: hundreds of catalog items, dozens of search queries, thousands of short descriptions. That is what collectors are for:

- compare competitor prices and assortments across hundreds of pages;
- gather reviews and discussions from many places;
- collect material for analysis: product descriptions, listings, publications;
- process a large volume of text with a model in one go — label, shorten, translate.

A harvester is configured once. After that the agent decides when to use it: you ask for data in chat, and it picks the right service, runs the harvest in large batches and puts the result away.

## Three kinds of harvesters

Harvesters fall into three groups: **Search**, **Download** and **Completion**. Each group covers its own step — find, fetch, process.

### Search — find on the internet

- **Serper (Google SERP)** — Google search results via the service’s key;
- **Brave Search** — search through the Brave Search API;
- **Tavily** — web search for AI tasks, able to return a ready model answer at once;
- **Keenable** — web search built for AI agents;
- **SearXNG** — a metasearch engine on your own server;
- **DuckDuckGo** — free search without a key, but with mandatory rate limiting;
- **JSON SEO** — Yandex, Google and Bing results, paid per results page.

### Download — fetch the page

- **Direct HTTP** — a plain page request: the cheapest and fastest way; gives up where content is rendered by scripts or bot protection stands;
- **Proxied HTTP** — the same through a proxy: helps when a site limits access by country or request rate;
- **Browser Emulation** — a local browser (Playwright) on your server: renders script-driven pages, downloads automatically on first use;
- **Nodriver** — a local undetectable browser: the best option for bot-protected pages; the package and Chromium download by themselves;
- **Remote Browser** — a remote browser-management service: for heavily protected, script-heavy sites.

HTML pages are saved as clean Markdown: Trafilatura does the conversion, so the text stays while navigation and ads drop out. The original page is saved beside it — you can always check what was on the site.

### Completion — process with a model

A completion harvester is any model with an OpenAI Chat Completions API: OpenAI, OpenRouter, DeepSeek and others. It exists for bulk text processing: label data, extract the fields you need, translate, shorten. Every request goes out in large batches, and the cost is counted in tokens.

## How to configure a harvester

Harvesters are configured in the Harvesters section. The common fields are the same for all:

- name — how you will recognize it;
- type — chosen at creation and never changed after;
- the “enabled” switch — a disabled harvester takes no part in the work;
- priority — the lower the number, the earlier the harvester is tried;
- maximum concurrency — how many tasks go out at once;
- rate limit — how many requests per period (empty — no limit).

Below are the type’s own settings: the service key, the base URL, a proxy, the timeout, the model name, the maximum output tokens, the temperature and more. The interface shows exactly the fields the chosen type needs.

Keys and passwords are stored masked: the list and the form show only `••••`, and “reveal” is a separate action. Instead of the key itself you can give a reference to a server environment variable (for example `$MY_KEY`): the secret then does not sit in the application database but is read from the environment at call time. If the variable is unset, the harvester honestly names the missing value.

Every harvester has a Test button: it makes a real probe call — a small search query, a download of example.com, or a tiny completion — and shows the service’s response. That is the fastest way to check that a key works and the settings are right. A harvester can be cloned with its settings, or deleted when it is no longer needed.

## Order and fallback

Harvesters of one group are ordered by priority: the one with the lower number is tried first. If a service does not answer or returns an error, the job moves to the next one — a harvest is not ruined by a single failure. This way one harvest can run through several search engines, or a paid service can be backed up with a free one.

When a harvest is pinned to a specific harvester, no fallback applies: the job either completes through it or fails with its error. That matters when one exact source is important — say, to keep a month-over-month comparison on identical rules.

## Harvests — what came out of it

A harvest is a bulk collection job: a batch of queries, pages or texts. The agent starts it at your request from chat, and the Harvests section shows what is happening with it. Next to every job you see the category, the state, how many items are already processed, the volumes and the cost, plus the expected finish time.

The state changes as the work goes: pending, running, completed or failed. Inside a job, results come by category:

- for search — the found links per query, the best on top;
- for download — a table of files with a view of the text and the original page;
- for completion — attempts with the question and the answer, token counts and cost.

The job queue lives in the application database, so a server restart does not lose it: the harvest continues from the same place. Results can be read again as many times as you like — they are not consumed by repeated reads.

## Where downloaded files live

Downloaded pages and documents go into the `/data/harvest` folder (movable with the `DATA_AGENT_HARVEST_FOLDER` variable). Inside are folders by date, and the file name is built from the domain and the page title. Every page has two files side by side: the Markdown text and the original HTML. Other kinds of documents — a PDF, for example — are stored as-is, with their own extension.

Downloaded files are temporary material, not an archive: the sweeper deletes them after two days (the period is the `DATA_AGENT_HARVEST_RETENTION_HOURS` variable). The paths and sizes stay in the database, but what the harvest was started for should be saved separately — in the Lakehouse. Long-living data is the scripts’ job, not the harvesters’.

## What it costs

Bulk collection spends real money: requests to paid search services and calls to models are not free. To keep costs under control, every harvester carries its own prices:

- search — the price per query;
- download — the price per page;
- completion — prices per million tokens: input, output, reading from the temporary storage and writing to it;
- the statistics show what has been spent and on what.

Priority helps save: cheap models go first, expensive ones as backup. Light tasks like labelling go to an inexpensive model, and the heavy parsing reaches the powerful one only when the first could not cope.

## Harvester statistics

Every harvester has a detailed card. It shows:

- how many tasks were handled and the share of successes;
- the average duration and the time nine out of ten tasks fit into;
- the total cost and when the service was last used;
- a two-week per-day strip and a log of recent calls with a jump to the job itself.

A new harvester shows all zeros — that is normal until tasks start flowing through it. As soon as they do, the statistics fill in by themselves.

See also: [Data Sources](/agents/data/docs/sources)
, [Scripts](/agents/data/docs/scripts)
, [Xedant Agent Integration](/agents/data/docs/agent)
.

[← Back to the documentation index](/agents/data/docs)