Alerts are a way to ask the system to watch your data instead of you. You set a rule: a query against the data and a condition. The system checks it on a schedule, and when the condition holds, it sends a notification wherever you pointed — Telegram, Slack, email and another two dozen channels. Checking metrics by hand is no longer needed: the system itself tells you when something goes wrong.
Alert rules
A rule is an ordinary SQL query against your data source plus a condition. Example: a “too few orders” rule counts the orders in the last hour and fires when there are fewer than ten. Conditions are simple: a threshold “above/below a number”, a range of values, a math expression — or a summary across groups of data.
- rules are grouped into groups, and each group gets a check interval — from 10 seconds to once a day;
- before firing you can set a wait period: for example, signal only if the problem has held for two minutes straight, so one-off fluctuations don’t create noise;
- after recovery the signal can be held for a while, so the alarm doesn’t blink on every change;
- any rule can be tested manually: it runs the query, shows what would have fired, and sends nothing;
- a rule can be paused or duplicated — for example, to make a copy pointed at another source.
What happens on firing
Every rule lives a life: calm → pending → firing → recovered. The transitions are visible in the list of the rule’s “instances” (one per data group) and are written into history: when it fired, what the values were, what changed. There are also special states:
- No data — the query returned an empty result. That alone can be an alarm: if counters from a server stopped coming in, something broke;
- Error — the check could not run (the data source went down, for example). You can choose whether to treat it as an alarm or just mark it.
A rule’s state is visible in the reports themselves: a block can be linked to a rule — the panel shows at once whether it is calm or firing. Reading the report and keeping a finger on the pulse happen in one place.
Where notifications go
Recipients are set in the “Contact points” section. One contact point is one recipient (your team’s Telegram chat, for example), and it can carry several delivery channels at once: Telegram and email together. 23 channels are supported:
- agent chat (Agent Chat): the notification arrives right into the analyst’s AI chat — with the model and skill you specify;
- messengers and chats: Telegram, Slack, Discord, Teams, Google Chat, WhatsApp-like ones (Line, WeCom), DingTalk, Threema, Webex;
- email, webhook, SMS services and on-call systems: email, webhook, PagerDuty, Opsgenie, VictorOps, Pushover, Amazon SNS;
- tools and queues: Jira, Sensu, Kafka, MQTT, Alertmanager and more.
Every contact point has a test button: it sends a real test notification, so you can be sure everything is set up right — before the first alarm ever happens.
Who gets what, and when
“Notification policies” are a tree of rules deciding which alarm goes to whom. For example: all alarms go to the shared chat, and alarms with the “database” label also go to the on-call team. Policies can:
- Group — similar alarms are collected into one message and arrive as a single notification, not ten;
- Repeat — if an alarm hasn’t cleared, the system reminds you after a set interval;
- Inherit — set a common rule for everyone and extend it for individual teams: new policies pick up their parents’ settings.
Silences
Not every alarm needs immediate attention. Two mechanisms let you mute notifications:
- Mute timings — a recurring schedule: at night, on weekends and holidays notifications are not sent. Time is counted in your timezone, and intervals may cross midnight;
- Silences — a one-off quiet period by labels: “go quiet for two hours at lunch” for a specific server or rule. A silence starts immediately and runs until the time you set; you can end it early at any moment.
Silences are easy to enable right from the list of fired alarms: one button creates a silence for this rule and this data.
Message templates
Notification text is configured with templates: the title, the essence, links to the report and the rule. Every channel has a ready default template (for Telegram, Slack, email and others), and you can override it with your own — for example, to have a table with the numbers right in the email.
History and the delivery log
The “History” section shows everything that happened:
- firing history — state transitions of every rule with dates, values and comments;
- delivery log — which notifications went out, through which channel, to whom, and whether delivery succeeded;
- retries on failure — if a channel didn’t answer, delivery retries with a gradually increasing pause, and every attempt is visible in the log;
- retention — history is kept for a set number of days (30 by default) and cleaned up automatically.
Recording check results
A separate rule type is “metric recording”: a rule that simply runs a query on a schedule and stores the result in the Analytics Agent database. From the saved data you can build your own charts and dashboards — and other alert rules on top of them. Storage is limited by a retention period (90 days by default).
All alerting settings are stored as plain JSON files in the data/alerting folder — like reports, they are easy to version and move between installations.