Scripts

The scripts folder is the heart of Data Agent. It is a plain folder of plain files on your server: one folder per data source. All scripts are written and maintained by the agent — you read and run them; there is nothing to edit by hand. The interface reads the folder directly, so the agent’s changes are visible instantly.

Folder structure

Every data source is a folder scripts/{project}/{source}/ that holds:

• README.md — the source’s mandatory description: where the data comes from, what is inside, which fields. It also becomes the folder’s page in the interface;
• extract.py — the export script: pulls data from the external service and saves it as Parquet;
• load.py — the build script: shapes the exported data into a single SQLite database for queries and visualization;
• optionally download.py — downloading files (when the export goes through files), test.py — checks, PLAN.md — notes.

The folder’s rules

• Parquet is always the raw data: what the export script pulled is never touched;
• SQLite is the derived database: rebuilt by the load.py script on every run;
• run logs are written next to the data in the Lakehouse, not in the scripts folder;
• passwords and keys live only in the .env file, never in the code: they reach the process as environment variables on every run;
• a new library is added to requirements.txt — the container installs it by itself on startup;
• keys for external services are better set up as harvesters rather than stored in code: the agent then takes a ready-made service, and the secrets live in one place.

The tree in the interface

The Scripts section shows folders and files as a tree. A folder’s page is its README description, formatted like a document (title, description, notes). A file’s page is the code with syntax highlighting; for Markdown, a finished document. Breadcrumbs at the top always show where you are.

Search, unread marks, icons

• search across folders and files works as you type — results appear immediately;
• unread changes stay highlighted until you open them;
• every folder can be given an icon — that is the section’s only write besides read marks; the interface never touches the code itself.

Live updates

File watching is built into the application: the agent fixes a script — the fix appears on screen instantly, without a reload. The same for new folders, descriptions and notes: you always see the current state of the folder.

Actions from the page

• a permanent link to a folder or file — easy to share;
• the “Ask the agent” button — opens a chat with the file’s path already pasted in: ask for a fix or an explanation;
• for runnable files — a run button and the run history right on the page.

Distribution and moving

The scripts folder is self-contained: copy it, pack it into an archive, or keep it as its own Git repository — the agent then works right on the repository. Moving to another server is simply moving the folder.

See also: Harvesters, Runs, Scheduler, Chat with the Agent.

← Back to the documentation index