Python Scripts

Analytics Agent can be extended with your own Python scripts: they sit as plain files next to the settings and plug into data processing. No rework of the application itself is needed — drop in a file and the new capability already works, without a rebuild or a restart. A script saves the day where the built-in tools fall short: connect a service that has no ready-made source, compute or rename the results of a query, calculate an alert condition your own way, or deliver a notification by a non-standard route.

This is a capability for those ready to write code; the rest of the product never needs scripts. And a script does not have to be complex: often it is ten lines.

Where the scripts live

All scripts live in the /project/apps/analytics/scripts/ folder — in the same persistent volume as the documents and data sources. One script is one JSON settings file:

{
  "title": "Orders from your system",
  "notes": "What the script does",
  "file": "orders.py",
  "timeoutSec": 120,
  "env": { "CRM_BASE_URL": "https://crm.example.com" },
  "args": { "pageSize": 100 }
}

code — either right in the settings file or in a separate .py file next to it, whichever is more convenient;
title and notes — so that half a year later it is still clear what this is and why;
timeout — how many seconds to wait for the answer (60 by default, 10 minutes at the ceiling);
environment variables — fixed values for the run: addresses, access keys. You can also list which of the server’s environment variables to pass to the script;
arguments — ready-made values for every run; they can be overridden at a particular connection point.

The libraries your scripts need are listed in one shared file, scripts/requirements.txt. The Python environment is prepared by the application on the first run — no separate setup.

Four connection points

The same script plugs into four places of the analytics pipeline:

Data source — queries go to the script instead of a database: any service that has a Python client fits. Such a source works everywhere the regular ones do: in charts, tables, validation, alert rules and the PDF export (details in Data Sources);
Query result processing — the script receives the finished table and returns the same kind of table, re-keyed, filtered, enriched or computed (details in Queries);
Alert condition — a computation of your own right inside the alert rule: a complex formula, external data, a scoring model (details in Alerts);
Notification delivery — a channel that is not among the ready ones: the script receives the notification text and decides itself where and how to send it (details in Alerts).

An example. The script calls an external service at the address taken from the query text and returns a table — columns and rows. From that moment the service works as a regular data source: you can build a chart on it, compute a metric and set an alert, with nothing else to configure.

How a run goes

the input is a single JSON document: the connection point, the query data, the arguments. It is read with the familiar json.load(sys.stdin);
the answer is JSON too: the last complete object the script printed. Progress messages may be printed as well — they do no harm;
every script has its own working folder that persists between runs: a convenient place for files, cursors, and intermediate results;
the data size is capped: when there is a lot of data, pass a reference or a file in the working folder instead of the data itself;
the number of concurrent runs is limited — so scripts do not overload the server.

The Scripts section

The sidebar has a “Scripts” section. It is a screen for watching the runs: definitions are read here, not edited — editing the file is the change.

the environment state at the top: whether Python is ready, whether the library set has been assembled, how many runs and failures the day saw, what happened last time;
the script cards: title, where the code comes from, when it last ran;
the run journal: time, connection point, the result or a clear failure reason. Filters by status and by connection point;
a test run: set the arguments and the data, press “Run” — the result appears in the journal right away, just like a regular run;
the journal is kept for a set term (30 days by default) and cleans itself up.

When a script breaks

A script failure never silently swaps the data: the report, chart or alert shows a clear error — what exactly failed (the script was not found, the wait timed out, the answer did not parse) — and never shows something of its own instead. The failure reason is always visible in the run journal.

Runs can be turned off with one switch — then every connection point honestly reports that scripts are disabled instead of pretending everything is fine. The first run can be slow: that is when the environment assembles and the libraries install.

The trust boundary

Scripts are your own code, and they run with the application’s rights: there is no isolation between a script and the server. So keep them in version control together with the project, and never run other people’s files. When full isolation is needed, run the whole application in a separate container — which is exactly what the standard installation does.

Keys and passwords for scripts are stored in the script settings as plain text, just like database connection passwords — worth considering when granting access to the settings folder.

← Back to the documentation index