Installation

Don’t want to read all this?

Just drop a link to https://xedant.com/agents/data/install.md or to https://xedant.com/llms.txt into any AI chat (Claude, ChatGPT, etc.) and ask it to generate the config files and commands. It will read the docs, ask you a few questions about your setup, and hand you a ready-to-use configuration. Save time — let the model do the reading for you.

You can also reach me on Telegram — I’m always glad to help. And that’s not just politeness — I genuinely enjoy talking to like-minded people, especially if you love coding as much as I do.

Data Agent is a web application for collecting and preparing data that deploys on your server as a single Docker container. The Python environment that runs the agent’s scripts is already built into the image. The directory with the agent’s content (scripts and source settings) is mounted separately — it is the same directory Xedant Agent works on, so both products see the same files.

Minimum requirements

Any Linux host with Docker is enough. Everything else is already in the image:

the application web server (ASP.NET Core 9);
a Python environment with all the data libraries (pandas, pyarrow, SQLAlchemy);
tools for file operations on data.

No database server is needed: by default the application’s own data lives in the built-in SQLite database in the /apps/data volume. PostgreSQL connects with one environment variable if you already run one.

The product is sparing with server resources. While idle the processor is not loaded: the scheduler and the harvesters do not poll the database on a timer — they wake up when there is work. After 15 minutes without activity the memory in use is released, and everything comes back on its own at the next task. The script search index itself lives on disk and is reused between runs, so it does not hold memory while nobody is searching. That is why Data Agent sits comfortably alongside other services on one server.

Docker Compose

Create a compose.yml file and run docker compose up -d:

services:
  dataagent:
    image: xedant/data-agent:min-latest
    container_name: dataagent
    ports:
      - "8080:8080"
    volumes:
      # The application's permanent data: database, keys, license, browsers
      - ./data:/apps/data
      # Version-controlled directory: agent scripts (scripts/) and source settings (sources/)
      - ../../apps/data:/project/apps/data
      # Lakehouse root — data organized by project and source
      - /data:/data
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ASPNETCORE_URLS=http://+:8080
      # Empty = SQLite /apps/data/data.db; a connection string = PostgreSQL
      - DATA_APP_DB=
      - DATA_APP_BRAND=xedant
      # Connection to Xedant Agent for chat
      - AGENT_API_URL=
      - AGENT_API_KEY=
      # SHA-256 hash of the key for the agent APIs /api/agent/*
      - DATA_APP_HASH=
      # Python environment baked into the image
      - DATA_APP_PYTHON=/app/venv/bin/python
    restart: unless-stopped
    sysctls:
      fs.inotify.max_user_watches: "524288"
      fs.inotify.max_user_instances: "512"

After startup the interface is available on port 8080: http://localhost:8080.

There are three mounts in the compose file:

./data:/apps/data — the application’s permanent data: database, keys, license, browsers;
../../apps/data:/project/apps/data — the version-controlled directory: the agent’s scripts (scripts/) and source settings (sources/). It is the same host directory Xedant Agent works on, so the agent and the app see the same files;
/data:/data — the Lakehouse root: collected data organized by project and source.

The sysctls raise the file-watching limits (inotify) — they are needed so changes the agent makes to scripts are picked up instantly, without a restart. Without these limits, some changes can go unnoticed when the file count is large.

The Python environment is built right into the image: on start the container compares the library list from requirements.txt and installs what is missing, and while running the app re-checks the file before every script run and installs whatever changed — with no restart or rebuild. The agent only needs to add a library to the file; everything else happens on its own.

Environment variables

DATA_APP_BRAND — brand and interface language: xedant (English, the default) or pastukhov (Russian).
DATA_APP_DB — path to the SQLite database or a PostgreSQL connection string. Empty = SQLite /apps/data/data.db.
AGENT_API_URL and AGENT_API_KEY — the address and API key of Xedant Agent for chat. Without them the Chats section is hidden; everything else works.
DATA_APP_HASH — the SHA-256 hash of the key the agent uses to call the agent APIs /api/agent/*. Without it those APIs stay closed.
DATA_APP_PYTHON — the Python interpreter for script runs (by default /app/venv/bin/python, the one baked into the image).
DATA_APP_DATA — the folder of the application’s permanent data: database, keys, license, browsers (by default /apps/data);
DATA_APP_CONFIG_PATH — the directory with version-controlled content: scripts and source settings (by default /project/apps/data);
DATA_APP_LAKEHOUSE_PATH — the Lakehouse root where collected data lives (by default /data);
DATA_APP_SCRIPTS_PATH — the folder of the agent’s scripts (by default /project/apps/data/scripts);
DATA_APP_TMP_FOLDER — temporary files: run work folders and service data (by default /tmp/agent, swept automatically);
DATA_APP_ENV_FILE — the path to the file with environment variables (by default /project/.env, see below);
DATA_APP_HARVEST_FOLDER — the folder for files downloaded by harvesters (by default /tmp/agent/harvest).
DATA_APP_HARVEST_RETENTION_HOURS — how many hours to keep downloaded files before the automatic cleanup (by default 48).
DATA_APP_PLAYWRIGHT_BROWSERS — the folder where browsers for page downloading are installed (by default /apps/data/ms-playwright).
DATA_APP_SEARCH_INDEX_FOLDER — the folder of the script search index (by default /apps/data/search-index);
DATA_APP_CLICKHOUSE_URL — a ClickHouse connection string: optional statistics upload for script runs. Empty = disabled.
DATA_APP_INSTANCE — the instance name for statistics (by default, the machine name);
DATA_APP_BASE_PATH — hosting in a subfolder of your domain (for example, /dataagent); empty = the app at the domain root;
DATA_APP_SSL and related — enabling HTTPS with your own certificate; empty = HTTP only;
DATA_APP_ERROR_REPORTING — 0 disables automatic error reports (on by default);
AGENT_LICENSE — the product license key (or activate it in the interface).

Important: DATA_APP_HASH is the SHA-256 hash of the key, not the key itself. The hash is computed with echo -n "key" | sha256sum, and that value is what Xedant Agent checks against the X-API-Key header it sends. The raw key is never stored in the variable.

The .env file

Environment variables do not have to live only in the compose file. The project root can hold a .env file — plain KEY=VALUE lines. The app reads it on start and re-reads it on the fly: change a line, and the new value applies without a restart. This is also the file for passwords and keys to external services: scripts receive them at run time and so never need them in code. One file holds both the app settings and the source secrets.

Two rules: a value already set in the container environment always beats the file (the file never overrides variables from compose), and the file itself — passwords included — never goes into Git. The file path can be changed with the DATA_APP_ENV_FILE variable.

By default the login and password are not set in advance. On first launch the application is open for registration: the first account created becomes the administrator. After that, registration closes. If you would rather prepare access beforehand (for example, for an unattended deployment), set the pair DATA_APP_LOGIN and DATA_APP_PASSWORD: the login in plain text, the password as its SHA-256 hash (a generator is on the sign-in page). Both variables must be set; without them self-registration applies. Passwords are stored as SHA-256 hashes, sessions work through JWT tokens (90 days), and interface settings — theme, language, font size — are chosen by each user separately.

Setup after launch

Chat with the agent: set AGENT_API_URL and AGENT_API_KEY — the Chats section and the agent panel on every page appear. The list of models and skills comes from the agent itself.
Agent APIs (/api/agent/*): set DATA_APP_HASH — the agent will manage scripts, runs, the schedule and data queries from its chat. For this, Xedant Agent must work on the same directory the scripts live in (apps/data/scripts) — both sides see the same files.
License: activated with a key in the interface (the license dialog) or the AGENT_LICENSE variable. Without a valid license, sending messages to the agent and managing the prompts queue do not work; runs, the scheduler, the Lakehouse and browsing data do work.

Bulk harvesting: browsers and packages

Harvesters can pull pages even from sites that defend themselves against bots. Everything needed for that installs itself on first use: the Python packages — trafilatura (converting pages to text) and nodriver (an undetectable browser) — into the Python environment, and the browsers themselves into the /apps/data/ms-playwright folder (moved with the DATA_APP_PLAYWRIGHT_BROWSERS variable). There is nothing to install manually.

The first such harvest after a fresh install takes longer — it waits for the installation. After that everything works instantly. Downloaded files live in the /tmp/agent/harvest folder (moved with DATA_APP_HARVEST_FOLDER), and the sweeper deletes them after two days (the period is DATA_APP_HARVEST_RETENTION_HOURS). The folder sits in the container’s temporary storage, so recreating the container clears it — downloaded files are meant to be transient material.

Data storage

The data is split across four places — by how long it lives:

/apps/data (a volume) — the application’s permanent data: database, keys, license, browsers, and the script search index. Survives container recreation. The index is not data but a search accelerator: the folder can be deleted and it will be rebuilt (changed with the DATA_APP_SEARCH_INDEX_FOLDER variable);
/data — the Lakehouse root: collected data by project and source (Parquet, SQLite, README, logs). A plain folder on your disk, moved by copying;
/project/apps/data — the directory with version-controlled content: the agent’s scripts (scripts/) and source settings (sources/). Copy it, version it in Git, or give it to the agent as a separate repository;
/tmp/agent — temporary files: run work folders and pages downloaded by harvesters. Recreating the container clears them — by design.

Upgrading

An upgrade is two commands: git pull, then docker compose pull && docker compose up -d. The /apps/data volume survives the upgrade; schema migrations run automatically at startup. To roll back, pin the previous image tag and restore the /apps/data backup taken before the upgrade.

← Back to the Data Agent home