A source is a place the news comes from. You do not configure them by hand in forms: every source is a small text settings file that your agent creates and edits. You simply say what to connect, and it sorts out the fields and checks that the source answers.
What a source is
Every source has a key — a short name it is referred to by everywhere else: in lists, in rules, in event cards. Next to the key live the address, the checking schedule, when needed its own headers or access through a proxy server, and also tags — handy for filtering events later.
The “setup wizard” in this product is not a separate screen but your agent. If you do not know which feed to take from a site, just name the site’s address: the agent can find feeds on the page by itself.
Fifteen source types
- Feed — a regular news feed in RSS 2.0, Atom 1.0 or JSON Feed 1.1 format. The format is recognized by itself; you do not need to name it. The simplest and most reliable source type.
- Page — watching a page as a whole. No material is collected here: the point is to notice the change of the page itself. Fits “about the company” pages, pricing, delivery terms.
- List — a page without a feed turns into a stream of records: you say which blocks of the page count as products or listings, and the agent parses them into separate items. That is how catalogs and classified boards work.
- Service query — the source calls a service at an address and parses the answer. Needed when a service has a programmatic interface but no ready feed.
- Search — a saved search query: “everything mentioning the company name”. You can limit domains, freshness and language — so that years-old junk does not get into the material.
- Sitemap — a site’s page-index file. New pages appearing in it arrive as material. Convenient for sites that do not serve news as a feed but always have a sitemap.
- Public Telegram channel — the agent reads the latest posts at the channel’s open address and separately marks posts that were deleted inside the watched window.
- Intake — a passive source: nothing is polled, material arrives from outside. That is how newsletters connect — your mail service forwards them to the intake address.
- Document — watching a file: the agent notices a change of size and content (by checksum). Fits price lists and regulations published as files.
- Reddit — a public community or a selection of communities.
- YouTube — a channel, an author or a playlist: new videos arrive as material with the description.
- Podcasts — episode releases arrive as material with the episode description.
- Fediverse — an account on Mastodon and similar networks: the agent reads the public posts.
- Reviews — a public reviews page is parsed into separate reviews, and the score (say, “4.6”) becomes a value you can watch: “tell me when the average drops below 4.0”.
- Your own collector — when a site serves data in an unusual way (for example, through a separate programmatic service), the source becomes your own Python code: it calls whatever is needed and returns a list of items. Such a source then lives like all the others — with its own schedule, state and check journal.
What all sources share
- Schedule — how often to check the source. A check can be paused and run immediately.
- Health — the source’s state: “active”, “a streak of failures”, “error”, “paused”, “waiting for the first check”. Next to it, the check journal: what was found, how many new items, what the site answered.
- Rate limit — how many requests per minute this source is allowed. One request per minute by default, so as not to annoy other people’s sites.
- Its own access — an individual source can get its own proxy server, its own request headers and its own access method, without touching the rest.
- Tags — the source’s labels (“competitors”, “suppliers”, “industry”). Rules and alerts are later built on them.
Finding feeds and moving the source list
The agent can find feeds by a site’s address — including for YouTube, GitHub, Reddit and Mastodon, where the feed hides behind a non-obvious address.
The source list can be exported into an OPML file — a simple list of addresses understood by nearly every news-reading program — and such a file can be imported back. Useful when moving: the source list transfers in one go. But honestly: OPML carries the list only. Schedules, tags and the already collected material do not get into that file.
Honest limits
- Public Telegram channels only. Private channels need a separate bot or special access — that is not in the first version.
- Sites that need a login open with your saved cookies. The product performs no automatic logins by username and password: when a session dies, the source honestly reports that access is lost instead of pretending “no changes”.
- X (Twitter), Instagram and TikTok are not supported. Their terms make collection unprofitable and fragile — better not to promise what breaks in a month.
Next: how the agent walks the sources, stores pages and filters repeats — in the Collection, Politeness & Archive section.