Installation

Don’t want to read all this?

Just drop a link to https://xedant.com/agents/data/install.md or to https://xedant.com/llms.txt into any AI chat (Claude, ChatGPT, etc.) and ask it to generate the config files and commands. It will read the docs, ask you a few questions about your setup, and hand you a ready-to-use configuration. Save time — let the model do the reading for you.

You can also reach me on Telegram — I’m always glad to help. And that’s not just politeness — I genuinely enjoy talking to like-minded people, especially if you love coding as much as I do.

Data Agent is a web application for collecting and preparing data that deploys on your server as a single Docker container. The Python environment that runs the agent’s scripts is already built into the image. The scripts folder is mounted separately — it is the same folder Xedant Agent works on, so both products see the same files.

Minimum requirements

Any Linux host with Docker is enough. Everything else is already in the image:

the application web server (ASP.NET Core 9);
a Python environment with all the data libraries (pandas, pyarrow, SQLAlchemy);
tools for file operations on data.

No database server is needed: by default the application’s own data lives in the built-in SQLite database in the /data volume. PostgreSQL connects with one environment variable if you already run one.

Docker Compose

Create a compose.yml file and run docker compose up -d:

services:
  dataagent:
    image: xedant/data-agent:min-latest
    container_name: dataagent
    ports:
      - "8080:8080"
    volumes:
      - ./data:/data
      # The scripts folder — the agent works on it directly (AGENT_PROJECT_PATH)
      - ../../scripts:/project/scripts
      # Lakehouse root — data organized by project and source
      - /lakehouse:/lakehouse
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ASPNETCORE_URLS=http://+:8080
      # Empty = SQLite /data/data.db; a connection string = PostgreSQL
      - DATA_AGENT_DB=
      - DATA_AGENT_BRAND=xedant
      # Connection to Xedant Agent for chat
      - AGENT_API_URL=
      - AGENT_API_KEY=
      # SHA-256 hash of the key for the agent APIs /api/agent/*
      - DATA_AGENT_API_KEY=
      # Python environment baked into the image
      - DATA_AGENT_PYTHON=/app/venv/bin/python
    restart: unless-stopped
    sysctls:
      fs.inotify.max_user_watches: "524288"
      fs.inotify.max_user_instances: "512"

After startup the interface is available on port 8080: http://localhost:8080.

There are three mounts in the compose file:

./data:/data — the application’s permanent data: database, keys, source overrides;
../../scripts:/project/scripts — the scripts folder: the very one Xedant Agent works on (its AGENT_PROJECT_PATH), so the agent and the app see the same files;
/lakehouse:/lakehouse — the data root, organized by project and source.

The sysctls raise the file-watching limits (inotify) — they are needed so changes the agent makes to scripts are picked up instantly, without a restart. Without these limits, some changes can go unnoticed when the file count is large.

The Python environment is built right into the image: on every start the container compares the dependencies from requirements.txt and installs what is new on its own. For the agent to use a new library, adding it to that one file is enough.

Environment variables

DATA_AGENT_BRAND — brand and interface language: xedant (English, the default) or pastukhov (Russian).
DATA_AGENT_DB — path to the SQLite database or a PostgreSQL connection string. Empty = SQLite /data/data.db.
AGENT_API_URL and AGENT_API_KEY — the address and API key of Xedant Agent for chat. Without them the Chats section is hidden; everything else works.
DATA_AGENT_API_KEY — the SHA-256 hash of the key the agent uses to call the agent APIs /api/agent/*. Without it those APIs stay closed.
DATA_AGENT_PYTHON — the Python interpreter for script runs (by default /app/venv/bin/python, the one baked into the image).
DATA_AGENT_HARVEST_FOLDER — the folder for files downloaded by harvesters (by default /data/harvest).
DATA_AGENT_HARVEST_RETENTION_HOURS — how many hours to keep downloaded files before the automatic cleanup (by default 48).
DATA_AGENT_PLAYWRIGHT_BROWSERS — the folder where browsers for page downloading are installed (by default /data/ms-playwright).
DATA_AGENT_CLICKHOUSE_URL — a ClickHouse connection string: optional statistics upload for script runs. Empty = disabled.
DATA_AGENT_INSTANCE — the instance name for statistics (by default, the machine name).
AGENT_LICENSE — the product license key (or activate it in the interface).

Important: DATA_AGENT_API_KEY is the SHA-256 hash of the key, not the key itself. The hash is computed with echo -n "key" | sha256sum, and that value is what Xedant Agent checks against the X-API-Key header it sends. The raw key is never stored in the variable.

Unlike the other agents, the login and password are not set in environment variables. On first launch the application is open for registration: the first account created becomes the administrator. After that, registration closes. Passwords are stored as SHA-256 hashes, sessions work through JWT tokens (90 days), and interface settings — theme, language, font size — are chosen by each user separately.

Setup after launch

Chat with the agent: set AGENT_API_URL and AGENT_API_KEY — the Chats section and the agent panel on every page appear. The list of models and skills comes from the agent itself.
Agent APIs (/api/agent/*): set DATA_AGENT_API_KEY — the agent will manage scripts, runs, the schedule and data queries from its chat. For this, Xedant Agent must work on the same scripts/ folder (its AGENT_PROJECT_PATH).
License: activated with a key in the interface (the license dialog) or the AGENT_LICENSE variable. Without a valid license, sending messages to the agent and managing the prompts queue do not work; runs, the scheduler, the Lakehouse and browsing data do work.

Bulk harvesting: browsers and packages

Harvesters can pull pages even from sites that defend themselves against bots. Everything needed for that installs itself on first use: the Python packages — trafilatura (converting pages to text) and nodriver (an undetectable browser) — into the Python environment, and the browsers themselves into the /data/ms-playwright folder (moved with the DATA_AGENT_PLAYWRIGHT_BROWSERS variable). There is nothing to install manually.

The first such harvest after a fresh install takes longer — it waits for the installation. After that everything works instantly. Downloaded files live in the /data/harvest folder (moved with DATA_AGENT_HARVEST_FOLDER), and the sweeper deletes them after two days (the period is DATA_AGENT_HARVEST_RETENTION_HOURS). The folder sits inside the /data volume, so it survives container recreation.

Data storage

Everything permanent lives in the /data volume and survives container recreation:

/data — the application database, JWT keys, source overrides;
/lakehouse — script data, by project and source: Parquet, SQLite, README, logs;
/data/harvest — pages and documents downloaded by harvesters: in date folders, with a retention period (Markdown and the raw HTML side by side);
/project/scripts — the plain scripts folder: copy it, version it in Git, or give it to the agent as a separate repository.

Upgrading

An upgrade is two commands: git pull, then docker compose pull && docker compose up -d. The /data volume survives the upgrade; schema migrations run automatically at startup. To roll back, pin the previous image tag and restore the /data backup taken before the upgrade.

← Back to the Data Agent home