---
title: "Installation"
id: "1790"
type: "page"
slug: "install"
published_at: "2026-09-20T21:51:48+00:00"
modified_at: "2026-09-21T01:36:28+00:00"
url: "https://xedant.com/agents/data/install"
markdown_url: "https://xedant.com/agents/data/install.md"
excerpt: "Data Agent is a web application for collecting and preparing data that deploys on your…"
---

# Installation

[https://xedant.com/agents/data/install.md](https://xedant.com/agents/data/install.md)

Don’t want to read all this? Just drop a link to [https://xedant.com/agents/data/install.md](https://xedant.com/agents/data/install.md)
 or to [https://xedant.com/llms.txt](https://xedant.com/llms.txt)
 into any AI chat (Claude, ChatGPT, etc.) and ask it to generate the config files and commands. It will read the docs, ask you a few questions about your setup, and hand you a ready-to-use configuration. Save time — let the model do the reading for you.

 You can also reach me on Telegram — I’m always glad to help. And that’s not just politeness — I genuinely enjoy talking to like-minded people, especially if you love coding as much as I do.

 Data Agent is a web application for collecting and preparing data that deploys on your server as a single Docker container. The Python environment that runs the agent’s scripts is already built into the image. The scripts folder is mounted separately — it is the same folder Xedant Agent works on, so both products see the same files.

## Minimum requirements

Any Linux host with Docker is enough. Everything else is already in the image:

- the application web server (ASP.NET Core 9);
- a Python environment with all the data libraries (pandas, pyarrow, SQLAlchemy);
- tools for file operations on data.

No database server is needed: by default the application’s own data lives in the built-in SQLite database in the `/data` volume. PostgreSQL connects with one environment variable if you already run one.

## Docker Compose

Create a `compose.yml` file and run `docker compose up -d`:

```
services:
  dataagent:
    image: xedant/data-agent:min-latest
    container_name: dataagent
    ports:
      - "8080:8080"
    volumes:
      - ./data:/data
# The scripts folder — the agent works on it directly (AGENT_PROJECT_PATH)
      - ../../scripts:/project/scripts
# Lakehouse root — data organized by project and source
      - /lakehouse:/lakehouse
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ASPNETCORE_URLS=http://+:8080
# Empty = SQLite /data/data.db; a connection string = PostgreSQL
      - DATA_AGENT_DB=
      - DATA_AGENT_BRAND=xedant
# Connection to Xedant Agent for chat
      - AGENT_API_URL=
      - AGENT_API_KEY=
# SHA-256 hash of the key for the agent APIs /api/agent/*
      - DATA_AGENT_API_KEY=
# Python environment baked into the image
      - DATA_AGENT_PYTHON=/app/venv/bin/python
    restart: unless-stopped
    sysctls:
      fs.inotify.max_user_watches: "524288"
      fs.inotify.max_user_instances: "512"
```

After startup the interface is available on port `8080`: `http://localhost:8080`.

There are three mounts in the compose file:

- `./data:/data` — the application’s permanent data: database, keys, source overrides;
- `../../scripts:/project/scripts` — the scripts folder: the very one Xedant Agent works on (its `AGENT_PROJECT_PATH`), so the agent and the app see the same files;
- `/lakehouse:/lakehouse` — the data root, organized by project and source.

The sysctls raise the file-watching limits (inotify) — they are needed so changes the agent makes to scripts are picked up instantly, without a restart. Without these limits, some changes can go unnoticed when the file count is large.

The Python environment is built right into the image: on every start the container compares the dependencies from `requirements.txt` and installs what is new on its own. For the agent to use a new library, adding it to that one file is enough.

## Environment variables

- **DATA_AGENT_BRAND** — brand and interface language: `xedant` (English, the default) or `pastukhov` (Russian).
- **DATA_AGENT_DB** — path to the SQLite database or a PostgreSQL connection string. Empty = SQLite `/data/data.db`.
- **AGENT_API_URL** and **AGENT_API_KEY** — the address and API key of Xedant Agent for chat. Without them the Chats section is hidden; everything else works.
- **DATA_AGENT_API_KEY** — the SHA-256 hash of the key the agent uses to call the agent APIs `/api/agent/*`. Without it those APIs stay closed.
- **DATA_AGENT_PYTHON** — the Python interpreter for script runs (by default `/app/venv/bin/python`, the one baked into the image).
- **DATA_AGENT_HARVEST_FOLDER** — the folder for files downloaded by harvesters (by default `/data/harvest`).
- **DATA_AGENT_HARVEST_RETENTION_HOURS** — how many hours to keep downloaded files before the automatic cleanup (by default 48).
- **DATA_AGENT_PLAYWRIGHT_BROWSERS** — the folder where browsers for page downloading are installed (by default `/data/ms-playwright`).
- **DATA_AGENT_CLICKHOUSE_URL** — a ClickHouse connection string: optional statistics upload for script runs. Empty = disabled.
- **DATA_AGENT_INSTANCE** — the instance name for statistics (by default, the machine name).
- **AGENT_LICENSE** — the product license key (or activate it in the interface).

Important: `DATA_AGENT_API_KEY` is the **SHA-256 hash** of the key, not the key itself. The hash is computed with `echo -n "key" | sha256sum`, and that value is what Xedant Agent checks against the `X-API-Key` header it sends. The raw key is never stored in the variable.

## First sign-in — self-registration

Unlike the other agents, the login and password are not set in environment variables. On first launch the application is open for registration: **the first account created becomes the administrator**. After that, registration closes. Passwords are stored as SHA-256 hashes, sessions work through JWT tokens (90 days), and interface settings — theme, language, font size — are chosen by each user separately.

## Setup after launch

- **Chat with the agent**: set `AGENT_API_URL` and `AGENT_API_KEY` — the Chats section and the agent panel on every page appear. The list of models and skills comes from the agent itself.
- **Agent APIs** (`/api/agent/*`): set `DATA_AGENT_API_KEY` — the agent will manage scripts, runs, the schedule and data queries from its chat. For this, Xedant Agent must work on the same `scripts/` folder (its `AGENT_PROJECT_PATH`).
- **License**: activated with a key in the interface (the license dialog) or the `AGENT_LICENSE` variable. Without a valid license, sending messages to the agent and managing the prompts queue do not work; runs, the scheduler, the Lakehouse and browsing data do work.

## Bulk harvesting: browsers and packages

Harvesters can pull pages even from sites that defend themselves against bots. Everything needed for that installs itself on first use: the Python packages — `trafilatura` (converting pages to text) and `nodriver` (an undetectable browser) — into the Python environment, and the browsers themselves into the `/data/ms-playwright` folder (moved with the `DATA_AGENT_PLAYWRIGHT_BROWSERS` variable). There is nothing to install manually.

The first such harvest after a fresh install takes longer — it waits for the installation. After that everything works instantly. Downloaded files live in the `/data/harvest` folder (moved with `DATA_AGENT_HARVEST_FOLDER`), and the sweeper deletes them after two days (the period is `DATA_AGENT_HARVEST_RETENTION_HOURS`). The folder sits inside the `/data` volume, so it survives container recreation.

## Data storage

Everything permanent lives in the `/data` volume and survives container recreation:

- `/data` — the application database, JWT keys, source overrides;
- `/lakehouse` — script data, by project and source: Parquet, SQLite, README, logs;
- `/data/harvest` — pages and documents downloaded by harvesters: in date folders, with a retention period (Markdown and the raw HTML side by side);
- `/project/scripts` — the plain scripts folder: copy it, version it in Git, or give it to the agent as a separate repository.

## Upgrading

An upgrade is two commands: `git pull`, then `docker compose pull && docker compose up -d`. The `/data` volume survives the upgrade; schema migrations run automatically at startup. To roll back, pin the previous image tag and restore the `/data` backup taken before the upgrade.

[← Back to the Data Agent home](/agents/data)