All script runs — manual, scheduled, started from the agent’s chat and fired by afterRun hooks — go through the same engine: the same rules, queue, environment and history. That makes behavior predictable: what works on a schedule works the same way from chat.
Every run receives facts about itself from the app — who called it and why — and the script can return a ready result that is saved with the history. Details below.
How to run
The Run button sits on the page of the script to run. Only Python (.py) and shell (.sh) files are runnable. A run is accepted instantly: its record is created right away, while the process itself may wait for a free slot in the queue if parallel runs are limited.
Live output
The script’s output streams into the run’s console — you see every line as it appears, not after the finish. A run can be stopped with the Stop button if the script hangs or goes wrong.
One environment for every run
Every run gets a deterministic environment: the same Python interpreter (the one built into the image or set in settings), with the scripts root as the working directory. The .env file is re-read on every run, so edits to secrets or settings apply without restarting the application. Passwords reach the process as environment variables — they are not visible in the code.
Run context
A script never has to guess who started it or with which settings. Every run comes with a ready set of facts: the path to the script itself, the run number, where the run came from (manual, schedule, agent or hook), the arguments and the path to its own work folder. The data arrives as one object on input and is duplicated as environment variables — the script picks whichever way is convenient.
DATA_APP_RUN_ID— the run number;DATA_APP_SCRIPT— the script being run;DATA_APP_TRIGGER— where the run came from;DATA_APP_SCHEDULE_ID— the schedule entry, when the run came from one;DATA_APP_WORK_DIR— the script’s own work folder for state.
The private work folder matters. It lives in permanent storage and survives restarts: it is the right home for the “last extract date”, cursors and any other state the script uses to know where to continue from. The temporary folder must not be used for that — the sweeper clears it. If a script reads and writes none of this, it works exactly as before — nothing changes.
Run result
A script can return its outcome as ready data, not only as lines in the log. When the last thing in the output is a JSON object, Data Agent saves it together with the run. The run’s history entry gains a “Result” block with “Show result” and “Copy result” buttons — you see right away what exactly was collected, without reading the whole log. For example, a script can return the number of new rows, the date of the last extract or a list of the errors found.
If the script prints only text, no result is saved — that is fine too. The result is a small summary, not the data itself: large extracts are not meant for it, their place is the Lakehouse, and the result just carries a link to them.
Where a run came from
Every run carries a trigger label: started manually, by schedule, by the agent from chat, or by an afterRun hook (the “Hook” label). Agent runs are additionally tied to an ETL session and the chat — the run shows the session’s badge, which leads back to the chat to see what it was about. A hook run, the other way round, names the parent run it started after.
History and logs
All runs are kept in the application database. The Runs section is a filterable list: every run opens its details, full output and logs. Even if a script ran a week ago, its result can be viewed at any moment.
Limits and states
- max run time: by default a run is not limited — data extracts can run for hours. The time is set in settings, on a schedule entry, or on a specific run started by the agent; a script that runs longer is then stopped automatically;
- parallel run limit: too many simultaneous scripts do not overload the server — the rest wait in the queue. The queue is not endless either: while all slots are taken, at most 50 runs may wait, and the next run is refused with the message “the run queue is full” — protection against an accidental burst of runs;
- run states: running, finished successfully, finished with an error, stopped by the user, waiting in the queue, skipped;
- the global switch: settings contain an “Enable script execution” toggle. When it is off, no run starts — not manually, not by schedule, not by the agent, not by a hook; every attempt is journaled with the “Skipped” state and a reason. This is handy when you need to pause all data work for a while, for example during a move.
After a run — fresh data
After a successful run, the source’s folder in the Lakehouse is rescanned: new and updated data becomes available for queries immediately, with no manual setup. The source appears or refreshes automatically — details on the Data Sources page.