Data for Analytics

Data serves as the data source for Analytics. Together they form a collect → prepare → present pipeline: Data regularly collects data and shapes it into one consistent form, while Analytics builds reports, charts and dashboards on top. Each product does its own part, and both run on your server.

How it works

Data scripts export data to Parquet — that is the raw data, the source of truth. Then the load.py script builds it into a single SQLite database — this is the database Analytics uses for visualization. The Parquet stays beside it: the data can always be verified against it without touching the database.

Reading is separated from writing

Analytics works with Data’s data read-only: it builds reports, charts and dashboards on the ready databases, and queries never change them — the data can be viewed, not damaged. The analyst never edits the databases or the raw files directly.

When a report needs to be added, fixed or extended — a missing column, say, or a weekly recalculation — Analytics does not touch the files itself: it hands the work to Data through its protected API (key access, every call verified). Through that API it puts or replaces a script, runs it, refreshes the data and sets a schedule when needed — Data validates and does everything itself. Fresh data appears, and the store stays in order.

What you need to do

  1. Ask the agent in chat to collect the data you need — it will create the source, the scripts and run them.
  2. After the load.py run, the database appears in the Lakehouse and the source registers automatically (how — on the Data Sources page).
  3. In Analytics, connect that source — see the Data Sources section of its documentation — and build reports and charts.

Example: the demo pipeline

Data ships with a demonstration pipeline — a Forex market data simulator: the forex-simulator/api folder with extract.py and load.py. After a run, the forex-simulator_api.sqlite database appears in the Lakehouse — a ready example of what Analytics receives: sources, schemas, tables.

Consistency rules

  • Parquet is always the source of truth: the raw data is never rebuilt;
  • SQLite is the derived database: rebuilt on every run, so it always matches the latest raw data;
  • a README next to the database describes it: where the data comes from, the structure, the fields — Analytics and a human read the data the same way;
  • only Data changes the data: Analytics reads the ready databases and hands every addition or fix through its protected API, where each call is verified.

The analysis templates in Analytics work through the same pairing (a library of 257 ready-made solutions: forecasts, segments, scoring, anomalies): the agent adapts their scripts to your data and runs them on Data through its API. Details in the Analysis Templates section.

More about Analytics’s capabilities is on the Analytics page.

See also: Lakehouse, Data Sources.

← Back to the documentation index