Python Scripts

Research Agent can be extended with your own Python scripts: they sit as plain files in the research folder and plug into the pipeline — in place of a search provider, in place of a fetching method, in place of a model, as well as for processing Markdown and extraction results. No rebuild or restart of the application is needed: drop in a file and the new capability already works. A script saves the day where the built-in tools fall short: connect a service that has no ready-made provider, fetch a page your own way, clean up markup before storage, or filter extraction rows before writing.

This is a capability for those ready to write code; the rest of the product never needs scripts — and often a script is ten lines.

Where scripts live

All definitions live in the /project/apps/research/scripts/ folder — in the same persistent volume as the reports and the service settings. One script is one settings file, whose name is the script’s key (for example, clean-markdown.yml):

title: Clean up Markdown
notes: Strips menus and service blocks before storage
enabled: true
timeoutSec: 30
code: |
  import json, sys
  ctx = json.load(sys.stdin)
  print(json.dumps({"markdown": ctx["payload"]["markdown"]}))

code — either right in the settings file or in a separate .py file next to it, whichever is more convenient. The separate file is read from disk on every run, so an edit to it shows up immediately;
title and notes — so that half a year later it is still clear what this is and why;
enabled — a disabled script does not serve the places connected to it;
timeout — how many seconds to wait for the answer (60 by default, 10 minutes at the ceiling);
environment variables — fixed values for the run: addresses, access keys. You can also list which of the server’s environment variables to pass to the script;
arguments — ready-made values for every run; they can be overridden at a particular connection point.

The folder is re-read on the fly: a file edit applies to the next run in under a second, without restarting the application. The product will not let you delete a script that is still connected to a service — disconnect the service first.

The libraries your scripts need are listed in one shared file, scripts/requirements.txt. The application installs them on its own before the next run — no separate environment setup.

Three service connection points

A script is connected in the Services section like any other service — with a priority, concurrency, a request limit, and fallbacks, just like any provider:

Python script (search) — your own search source: the script receives a query and returns the pages it found. Fits any service that has no ready-made provider (details in Search);
Python script (fetch) — your own way to fetch a page: the script receives an address and returns the page’s content (details in Fetching Pages);
Python script (LLM) — your own path to a model: the script receives a model request and returns the answer. This is how a gateway with a non-standard protocol connects (details in Data Extraction).

Such a service has two settings fields: “Script key” — the file name without the extension, required; and “Arguments” — an optional set of values the script receives at run time. If the script does not answer or answers with an error, the pipeline behaves as with any other service: it moves on to the next one by priority.

Two transforms for the whole product

Besides services, there are two places where a single script is wired in for the whole product. They are set in the settings.yml settings file (the scripts: section) — not in the interface:

Markdown processing (onMarkdown) — runs after a page is fetched and converted to Markdown, but before storage: the script receives the text and returns the corrected one. The processed text is what gets saved, while the raw page stays untouched;
Extraction processing (onExtract) — runs after the extraction is parsed and before the rows are written: the script can filter, rename, or extend the records.

Both transforms are forgiving: if the script breaks, fetching and extraction still finish — on the unprocessed data — and the failure stays in the journal. One error in markup cleaning does not cancel the work already done.

How a run goes

every run is a separate Python process: it starts, delivers the answer, and exits;
the input is a single JSON document: the connection point, the script’s name, where the run came from, the arguments, the data. It is read with the familiar json.load(sys.stdin);
the answer is JSON too: the last complete object the script printed. Progress messages may be printed as well — they do no harm;
every script has its own working folder that persists between runs: a convenient place for files, cursors, and intermediate results;
the data size is capped: when there is a lot of data, pass a reference or a file in the working folder instead of the data itself;
the process receives none of the application’s secrets: only the variables you listed yourself in its settings are available to it.

Execution rules

The common rules are set in the same settings file, in the scripts: section, and apply on the fly:

the global kill switch — one toggle forbids all runs; every connected place reports honestly that scripts are disabled instead of pretending everything is fine;
the timeout — 60 seconds per run by default, 10 minutes at the ceiling; when it expires, the process is stopped;
concurrent runs — no more than two processes at once by default, so scripts do not overload the server;
the output cap — 1 MiB per run; larger data stays in the working folder, and the script returns a reference to the file.

The Scripts section

The sidebar has a “Scripts” section — a screen for watching the runs: definitions are read here, not edited; editing the file is the change.

the environment state at the top: whether Python is ready, whether the library set has been assembled, how many runs and failures the day saw, what happened last time;
the list of scripts: title, where the code comes from, what it is connected to, when it last ran;
the run journal: time, connection point, where the run came from, the object, duration, the result or a clear failure reason. Filters by status and by connection point;
a test run: set the arguments and the data, press “Test script” — the result appears in the journal right away, just like a regular run;
the journal is kept for 30 days and cleans itself up.

Failure reasons are named in plain words that need no decoding: “Runtime unavailable” (no working Python or the libraries did not assemble), “Timeout”, “Invalid output” (the script printed no complete JSON), “Too large”, “Canceled”, “Disabled”. A script failure never silently swaps the data: for a connected service it looks like a service error, for a transform like a warning in the journal.

The trust boundary

Scripts are your own code, and they run with the application’s rights: there is no isolation between a script and the server. So keep them in version control together with the project folder, and never run other people’s files. When full isolation is needed, run the whole application in a separate container — which is exactly what the standard installation does. The script’s secret variables in the settings file deserve the same care as database connection passwords.

← Back to documentation