Runs are executed by a runner — the thing that opens pages and walks through a test’s steps. It used to be only Chrome with our addon, and a permanently powered-on computer had to be kept next to it. Now the runner is built into the product itself: a headless Chromium browser inside the container that executes runs right after installation — with no addon and no second machine. Below is when it is enough, and when Chrome is more convenient.
Why it is needed
The point is for “a test is queued” and “a test is running” to be one and the same event. A freshly installed product already knows how to execute runs: a schedule fires at night, when nobody is sitting at the computer, and the result is waiting for you in the summary by morning.
That used to be missing: pages were opened only by the addon in your Chrome, so runs depended on whether the computer was on and the browser was open. There was one exception — a test assembled entirely from scripts was executed by the server. Now browser runs also go without you.
The built-in runner or Chrome with the addon
The Chrome addon has not gone anywhere and is no longer required for runs. Here is how the two methods differ:
| What to compare | Built-in runner | Chrome with the addon |
| Where it works | in the product’s container, on the server | on your computer, in a regular Chrome |
| Is a separate computer needed | no | yes, if runs go on a schedule |
| Can you see the run going on | no — the browser is headless | yes — you can watch with your own eyes |
| Hand captures and action recording | no | yes |
| Runs, schedules and snapshots for visual checks | yes | yes |
The short rule: for schedules and everyday runs the built-in runner is enough, while Chrome with the addon is kept where eyes are needed — watch a test go, capture a screen by hand, record an action, or let the model work with a live page. What the addon can do is in the Chrome Extension section.
One queue, several runners
A run is claimed by whoever gets there first. If both the built-in runner and Chrome are online, one of them does it, and the other does not take that run: it never executes twice. In the run history, next to the runner’s name, you can see who exactly executed it.
Each runner has its own engine — that is the name of the browser a run happens in. The built-in one works on the Chromium engine, the addon on your Chrome. These are different engines, and they render pages slightly differently, so their baselines are separate: the product never compares a snapshot from one engine against a sample from the other — otherwise every build would look like a complete rework of the page. An engine is chosen per project and not mixed: switching to another engine gives new baselines, which have to be reviewed and approved once, after which comparisons are stable again. Details are in the Baselines section.
Two modes of operation
- On demand — this is the default. At idle there is no runner at all: no browser, no process serving it. When a run lands in the queue, the product starts the runner itself, and once the work is done it exits. The first run after a pause starts a couple of seconds slower; after that everything goes as usual.
- Resident — the runner works all the time, and after a long idle it restarts itself to give back the memory it took. This mode is chosen when runs go continuously or a consistently fast response is needed.
Why it is done this way: a server that is quiet almost always should not keep a browser in memory for no reason. At idle the product spends neither CPU nor extra memory, so it sits comfortably on a small machine that also runs other programs.
Settings
All of them are optional — runs work right after installation. Change them only with good reason: for example, to bring back the resident mode or to let the browser live longer in memory.
| Variable | What it sets | Default |
TEST_APP_RUNNER_ENABLED | enables the built-in runner: 0 — disable it entirely, then runs are left to Chrome with the addon only | 1 |
TEST_APP_RUNNER_ON_DEMAND | 1 — start on demand, no process at idle; 0 — a resident runner | 1 |
TEST_APP_RUNNER_RECYCLE_MINUTES | in resident mode: after how many minutes of full idle to restart the runner to give back memory; 0 — never restart | 30 |
TEST_APP_RUNNER_NAME | how the built-in runner is called in the runners list | Playwright (built-in) |
TEST_APP_RUNNER_POLL_SEC | how many seconds the server holds the runner’s request waiting for a run from the queue (no more than 30). A queued run is claimed within milliseconds — this value only paces the empty checks | 15 |
TEST_APP_RUNNER_HEADLESS | 1 — headless mode (the product carries only that browser build); 0 — headed, which needs a full build | 1 |
TEST_APP_RUNNER_SANDBOX | browser isolation: 0 — off (needed to work in a container), 1 — on | 0 |
TEST_APP_RUNNER_BROWSER_IDLE_SEC | after how many seconds without runs to close the browser and give back memory; 0 — never close | 600 |
The access key (TEST_APP_KEY) is taken by the runner from the environment variables or from the settings file, and it re-reads it every cycle. That means the key can be dropped into the file on the fly — no container restart is needed for that.
Where it is visible in the interface
- “Settings → UI testing” — a separate “Built-in runner” row: whether it is available and when it last checked in with the server. It is not to be confused with the “Addon connected” line — that one refers to Chrome only.
- “Operations” — in the runners list the built-in one carries a “built-in” badge. There you can also see whether it is online and which browser version it is using.
- Draining and resuming — work the same as for Chrome: a drained runner finishes the current run and stops taking new ones, and after the pause is lifted it queues up for runs again.
Honest limitations
- Headless only. The product ships one browser build — the headless one, so you cannot watch a run with your own eyes. Need to observe — keep Chrome with the addon. Headed mode would require adding a full browser build into the container by hand — the product does not have one.
- Browser isolation is off. That is needed for the browser to work in a container at all. It only opens the addresses you yourself set in tests.
- The browser is already part of the product. Nothing is downloaded when the container starts — that is a plus: the product works on a slow and even on an internet-closed server. The flip side — the browser is updated only together with the product, separately from it.
- One run at a time. Like Chrome, the built-in runner executes one run at a time. When there are many runs, runners are added: the built-in one plus Chrome with the addon on another computer, for example.
Next: how a run goes and what its result means — in the Runs section. Pause a runner or look at their list — in the Operations & Health section. And if you need hand captures, recordings and runs in a live window — in the Chrome Extension section.