A single action or rule proves little: a real check is the whole journey, from signing in to the page you need, with all the checks along the way. A test assembles such a journey from ready-made pieces and runs as many times as you like. Below is what a test consists of and how to assemble one.
What a test is
A sequence of steps executed in order. A step comes in four kinds:
- Open an address — go to the page you need.
- Run an action — walk a recorded scenario: sign in, fill in a form, open a section.
- Check the rules — run the checks on the page where the test arrived.
- Script — the step is executed not in the browser but on the server: that is how an API address, a database or a file gets checked. The step names the script and its constant arguments; the script’s checks land in issues on equal terms with rules.
A step has a “continue on error” switch. Usually it is off: when a step fails, going further is pointless — the result would be wrong anyway. It is turned on for optional steps: a popup that appears only for some visitors, for example.
The script for a step comes from the project. If the named script does not exist or is disabled, the test will not save: the product names the guilty step and the reason. A disabled script never passes validation silently.
A step can also carry a capture declaration — an assignment of what to capture for the visual check: the snapshot kind (the visible part of the page, the whole page, a separate block or element), a human-readable name, how to find the element on the page, which areas to mask, which variation the snapshot belongs to and the threshold to compare at. What this gives you — in the Visual Testing section.
On a run, a test with declared captures becomes a build: the product captures what you specified and compares the snapshots against baselines. A test without declarations works as before — with rule checks.
Assembling a test
- On the “Tests” tab, create a new test and give it a clear name: “Order payment”.
- Add steps in order and pick ready actions and rules for them from the list.
- In a “check the rules” step the set can be narrowed: for example, check only the payment-page rules instead of all the project’s rules.
- Save. Every step is validated at once: if an action is deleted or not approved yet, the test will not save, and the product tells you exactly which step is at fault.
Validation on save is not pedantry: a test that references a nonexistent step would not run anyway — you would just learn it at the worst possible moment.
A test made entirely of scripts — no browser
When all of a test’s steps are scripts and the before/after phases pull in no actions, the test is executed on the server. No browser is needed at all for such a run: the runner line says “server”. The queue, the schedule and external access work as usual — but a mixed test, one with at least a single browser step, still waits for a browser with the addon.
Why this matters: new kinds of checks — server-side ones that have no page — are added as files in the project, not as changes to the product. That way API addresses, data in a database or files on disk get checked on the same schedule and with the same evidence as ordinary tests. Details — in the Scripts & Extensibility section.
How a test differs from a single rule
A rule checks the page it was run on. A test brings the browser to the right pages itself and checks each of them. So a rule answers “what is wrong on this page”, while a test answers “can the journey be walked at all, and is everything in order along it”. A rule works in the browser on the open page, a script on the server with no page at all; on failure, both file an issue.
Where a test lives
A test is a file tests/<id>.yml in the project folder, next to the actions and rules. That means it lands in the edit history and moves to another server together with the project. You can change it both through the interface and in the file — open it in the file dialog, and the list shows the current content. And the programmer agent can assemble or edit the same test over an access key — see the External API Access section.
Setup and cleanup
Before its steps, a test can run setup, and after them cleanup: sign into the right account before the start, put the data in order afterwards, for example. The list of “before” and “after” actions is set once for the project and fits every test. When a test has lists of its own, they fully replace the project’s — they are not mixed with them; an empty list means a deliberate refusal of the phase. Cleanup runs even when a step ended with an error — otherwise garbage would remain after a failed run.
The addresses and passwords a test needs come from Environments, Credentials & Variables.
Test version history
Every save of a test is remembered as a separate version: you can see what changed and when, and an earlier revision can be restored byte for byte. A run remembers which exact version it took — so old builds always tell you what was checked back then. Details — in the Test Versions & Steps section.
When a step goes stale
Interfaces change, and a step may stop working: a button was renamed, a field moved. The sign is simple — the run ends with an error on that step, while the page itself looks normal. What to do:
- look at the error message — it says which step failed;
- re-record the action on that page — the new action will memorize the elements again;
- or ask the agent to fix the action in words, when the edit is small.
How to run
A test can be started manually: with a button on the “Tests” tab in the app, or by picking the test in the addon panel. It then waits for an executor — a browser with the addon connected. Usually that is under a minute. When runs are needed regularly, the test goes on a schedule.
Next: what happens during a run and what stays after it — in the Runs section.