Lakehouse

The Lakehouse is a file interface to your data: a folder tree in the browser, viewing and querying files without configuring data sources. Drop a file into the folder — and you are already building a report on it.

Structure

/lakehouse/
├── ProjectName/
│   ├── DataSource1/
│   │   ├── file.md
│   │   ├── data.parquet
│   │   ├── database.sqlite
│   │   └── subfolder/
│   └── DataSource2/
└── AnotherProject/

The structure is: project → source → files and subfolders. In the interface, paths look like /lakehouse/{project}/{source}, and the home page shows all project-source combinations, sorted by the freshness of the files. The root is set by the ANALYTICS_LAKEHOUSE_PATH variable.

Supported formats

• Markdown — viewed as ready documentation for the data: table descriptions, glossaries, how-tos;
• Parquet — read through Apache Arrow: the file’s schema and queries without configuring sources;
• SQLite — queries to database files through the shared query pipeline, like a regular source;
• Excel — .xls and .xlsx tables, compressed ones included: you can pick the sheet, and the data pages through;
• PDF — the document opens right in the browser, no download needed;
• Logs and compressed text — .log, .txt, .csv and other text files, including .gz archives: paged viewing from the start of the file.

API

• folder and file contents;
• the schema of Parquet and SQLite files;
• queries to files right from the file browser.

The scenario

Exported data to Parquet, assembled a SQLite file, wrote documentation in Markdown — just put it all into the Lakehouse folder. The files are instantly visible in the browser, queries work on them, and reports plug them in as data. SQLite files are especially handy: a database from the folder root immediately becomes a ready source, no setup needed. For Python analytics the files are also readable from Jupyter blocks — details in the Jupyter Notebooks section.

The Lakehouse is usually filled by Data Agent: its scripts put Parquet files here and assemble a SQLite database from them. How this pairing works is described on the Data for Analytics Agent page.

← Back to the documentation index