---
title: "Data Sources & Visualisations"
id: "2209"
type: "page"
slug: "sources"
published_at: "2026-10-15T15:12:28+00:00"
modified_at: "2026-10-22T16:38:23+00:00"
url: "https://xedant.com/agents/portal/docs/sources/"
markdown_url: "https://xedant.com/agents/portal/docs/sources.md"
excerpt: "Besides text and pictures, a portal page can show your own data: stock on hand,…"
---

# Data Sources & Visualisations

[https://xedant.com/agents/portal/docs/sources/.md](https://xedant.com/agents/portal/docs/sources/.md)

Besides text and pictures, a portal page can show your own data: stock on hand, revenue by day, service status, warehouse load. The data comes straight from your systems — a database or a small calculation script — instead of being typed in by hand. That is how live summaries appear on the portal: the number on the page is always the same as in the source.

## Why it matters

An ordinary summary page goes stale quickly: someone has to update the numbers in it. Here it works the other way round. You describe once where to take the data, and the portal asks the source on every view. A manager opens the page and sees the current state of things rather than yesterday’s export.

There is a second way to take data — a calculation script on Python (see [Dynamic Python Blocks](/agents/portal/docs/python)
). It is good when the data has to be computed or collected piece by piece. Data sources are simpler: if you have a database to take ready rows from, one query description is enough — with no program at all.

## Where the sources live

A source is an ordinary file with a connection description. All of them live in the `sources/` folder at the portal’s root, one file per source:

```
portal/
├── sources/
│   ├── warehouse.json            the warehouse
│   ├── billing.json              issued invoices
│   └── uptime.json               service availability
├── scripts/                      calculation scripts
└── news/                         portal pages
```

Source files are part of the content: the managing agent edits them, they land in the version history and they move together with the portal’s folder to another server. There is no separate screen for sources in the interface — as with pages, their description is kept by the agent. The `sources/` folder follows the portal’s content folder: point the portal at another content folder (its `PORTAL_APP_PORTAL_PATH` setting) and the sources move with it.

The connection data (the database address, the login, the password) is written right in the source file. The content folder is therefore the place for secrets: only you and the agent should have access to it, and a file with a password is not worth publishing in an open repository.

The format is the one Analytics Agent uses, taken as is: a source file copied from an Analytics Agent deployment into this folder is a valid way to create a source. There is no separate address for creating sources either — the agent writes the file, and the watcher on the `sources/` folder picks the edit up without a restart. One broken file does not take the others down: the portal names exactly that file and keeps working with the sources that read.

## Four kinds of source

- **SQLite** — a database in a single file. It fits when the data already lies in a file on the same server; nothing extra has to be brought up. The path to the file is required, and the connection may be opened read-only.
- **PostgreSQL** — the common database for applications. The portal needs the host, the database name, the username and the password; the port is 5432 and a protected connection is on by default.
- **ClickHouse** — a database for large volumes and fast summaries. The settings are the same as PostgreSQL’s, the port differs — 8123 — and the protected connection is off by default.
- **A Python source** — for when there is no ready database: the data is collected by a small script from the `scripts/` folder, for example one that calls an external service. The query to such a source becomes the script’s input rather than a piece of SQL.

Every source can also be given a limit on how long to wait for an answer: 30 seconds by default, from one second up to five minutes. If an external service takes its time, the page will not wait for it forever.

## What you may ask a source

A question to a source is a query: which columns and rows to take. The query lives in the page’s description next to the visualisation, and the reader neither writes nor sees it. That is also where the safety comes from: a visitor of the public site cannot ask the database for anything extra — they see only what the agent described in advance.

- **Read only.** A source answers a selection of data, not changes: nothing can be altered or deleted from the portal. Only `SELECT` and `WITH` are accepted — with no write commands, no comments, no several statements in one query and up to 50 kilobytes of text. A query that tries to change something does not even reach the database: it is refused before any connection is opened.
- **30 seconds at most.** If the data does not arrive in time, the query is stopped and the reader sees an understandable message instead of a never-ending download. A separate query can be given its own limit.
- **10 000 rows at most.** A result that is too large is cut off, and the answer says so honestly — the portal does not hide that it did not show everything. A separate query can be given its own row limit.
- **One answer in one shape.** Whichever engine answers, the result comes back as one understandable structure: columns with their types and rows of values. A query that failed is still answered with an explanation rather than a technical error page.

If several pieces of data are needed — a summary by day and a breakdown by region, say — several visualisations are placed on the page, each with its own query. They all look at the same source.

## The Python source and result transformation

For a Python source the query is not SQL but the program’s input: the script receives the query text, the page’s variables and the declared parameters, and must end with one JSON object holding `columns` and `rows`. The read-only gate does not apply here — the text is the script’s input, not a request to a database.

After the query, a transformation can be declared with the same script. It receives the table that has just been computed, and its answer replaces the result — so the rows shown to the reader can be reshaped, filtered or recalculated. A failed transformation fails the query: the reader never gets raw rows as if the transformation had worked. Both the source script and the transformation run in the portal’s own Python environment, the same one the calculation scripts use (see [Scripts & Dependencies](/agents/portal/docs/scripts)
).

## What visualisations exist

The result of a query can be shown in different ways — 33 kinds in all. The kind is chosen when the page is assembled, and the agent inserts the visualisation as is: the block is carried over from Analytics Agent unchanged, and the portal draws it itself.

- **Numbers and gauges** — a tile with a measure; a gauge with a needle; a bar gauge; a scale showing plan and fact against a target.
- **Tables and cards** — a data table; a card with the details of a single object.
- **Charts** — lines, columns, dots, a combined chart, a doughnut, a histogram, a box plot, a violin, candles, a radar, parallel coordinates, a slope.
- **Comparison** — a dumbbell (two values side by side), a waterfall (how a value changed step by step), a Pareto (what makes up the bulk), a tornado (what influences more), a funnel (where stages are lost).
- **Time and order** — a state strip, a status history, a Gantt chart (stage timelines), a calendar with days shaded by intensity.
- **Connections, shares and maps** — Sankey flows, a treemap, a node graph, Venn circles, a heat map, a geographic map of regions.

Like any block, a visualisation has a title, a width and action buttons. If the portal meets a kind it does not know, the reader sees no error: a calm note with the kind’s name appears in the block’s place. The block types are described in the [Blocks](/agents/portal/docs/blocks)
 section.

## How a visualisation gets its data and drills by value

A reader never sends a query to the portal: it is looked up by where it lives — the page, the block and the query’s name inside the block. The server reads the definition from the page’s description on disk and runs it against the source the definition names. An unknown page, block or query degrades into an understandable answer naming what was not found, never into a broken block.

A summary can be made live not only on refresh but on selection as well. The page settings can declare a drill variable — its name and how to label it to the reader; then a click on a chart mark or on a table column marked for drilling substitutes the chosen value, and every block whose queries use it recalculates at once. That is how the big picture turns into a breakdown: sales by region first, a click on a region, and the numbers for that region appear next to it. A **Clear filter** button above the page brings the whole picture back.

## Freshness and short-lived caching

Under every visualisation you can see when the data was updated — **Updated 14:02** — and there is a refresh button. Data is refreshed by itself when the agent edits the page, when the chosen value changes, or when an action answers “refresh”.

It is not always reasonable to trouble the database on every view: a heavy query takes resources. For that, a block can be given a storage period for its result — from zero to 86 400 seconds, that is up to a day. With 300 seconds the data is recomputed no more often than once in five minutes, and in between the ready result is shown. By default there is no storage at all: the query runs on every view. A failed query is never stored — it is tried again on the next refresh. Editing a source file or the query description itself drops the stored result at once, and signed-in readers never share a stored result with anonymous visitors.

## The public site

A page with a summary can be public too: visitors see the same numbers on the site at `/p/`, with no sign-in. That is convenient for open statistics, a catalogue with stock levels or a “how we work” page. Each visitor gets the data through their own request to the portal and cannot ask for anything else: which data to show is decided by the page’s description, not by the visitor.

A draft stays closed: a query against a closed page is not executed from the outside at all, and the visitor sees the same “page not found” answer as for a typo in the address. Publishing a page also publishes the results of its queries (see [Public Site](/agents/portal/docs/public-site)
).

## When something goes wrong, and what the agent does

One failed visualisation does not break the page — the other blocks draw as usual. The reader sees an understandable note: **Failed to execute query**, **No query configured** or **No data**; the details and the hint of what to fix are shown to the agent only. A broken source file does not get in the others’ way either: the portal names exactly that file and keeps working with the sources that read. The findings are `SOURCE_FILE_INVALID` (the file does not parse), `SOURCE_TYPE_UNKNOWN` (the type is not one of the four) and `SOURCE_FIELD_REQUIRED` (a required field is missing — the finding names the field).

All the work with sources is done by the managing agent, with ordinary files and requests to the portal:

- **creates a source** — describes the connection in a new file and checks that it reads;
- **looks at the list of sources** — what is connected, what reads, and where the error in a description or in a file is;
- **tests a query before writing it into a page** — so the agent makes sure the query works and the columns are named as needed before people see the summary;
- **assembles the visualisation** — puts the block on the page and sets the kind, the query, the storage period and the drill variable.

Next: what else the agent controls — in the [Agent Management](/agents/portal/docs/management)
 section; how source files and queries are checked — in [Page Validation](/agents/portal/docs/validation)
; how to compute data with a script — in [Dynamic Python Blocks](/agents/portal/docs/python)
.

[← Back to the documentation index](/agents/portal/docs)