# Expectation: the guide Source: https://expectation.sim001.net/guide/ . One file for LLMs; index at /llms.txt. --- # Introduction Expectation is acceptance testing where the behaviour itself is the source of truth. You describe a system as **expectations**: given some state, when something happens, then the system ends up in some state. Every step is written in a shared **vocabulary** of states and events, so expectations can be compared with each other mechanically, not just run. ```gherkin Expectation: Turning the dial up starts the kettle Given the kettle is off When the dial is above 55 % Then the kettle is heating ``` From a suite of expectations, Expectation: 1. **finds conflicts.** Two expectations that both apply, and promise incompatible things, are an error. It names the situation where they clash and suggests which one is the exception. You settle it with `Overrides:`, and overrides must form a DAG. 2. **builds a flow DAG** from the starting state. It shows what leads to what, warns about behaviour nothing can reach, and gives every expectation a shortest route from the start. 3. **compiles user stories into acceptance tests.** A story only describes the journey. At each step, the expectations it passes through supply the checks. A story that relies on unspecified behaviour, or contradicts an expectation, is rejected before anything runs. Generated tests reach whatever no story does. 4. **runs those tests** against your system through a small driver, and takes a screenshot at every step. 5. **writes the documentation**: a manual where stories are step-by-step tutorials and features are topic pages, with the test evidence folded away under "How this is tested", plus a full test report. ```text .expect files ─language─▶ Suite ─graph─▶ conflicts · override DAG · flow DAG │ │ └────────story────────────┴─▶ plans (acceptance tests) │ harness + driver ──┴─▶ report + screenshots ─docs─▶ HTML + Markdown ``` Writing a suite that also reads as good documentation takes some care: see [Feature docs that explain](feature-docs.md), and its [generated feature page](../smart-kettle/site/feature-boiling-when-it-is-cheap.html). ## Why Most acceptance tests are scripts: do this, check that. Nothing connects one script to the next, so two tests can quietly assume opposite things, and nobody notices which behaviour is never tested at all. Gherkin made scripts readable, but its steps are free text bound to code with regular expressions, so a tool can't tell what they mean. Expectation makes the steps mean something. Because every step is a fact about a declared state, or a declared event, the suite is a model of the system. That model can be checked before any test runs. A story doesn't have to restate every outcome, because the expectations already promise them. And the same model, written in the user's words, is good enough to publish as the user manual. ## What it is not - It doesn't drive your UI for you. You write a **driver** (usually a dozen closures) that sets states, makes events happen and reads states back. Anything you can control from Rust can be tested: a library, a game loop, a web app through its API or a browser, a device over a serial line. - It isn't a property-based tester. Generated tests follow the shortest routes through the flow graph to cover every expectation; they don't search for counterexamples. ## Where to go next - [Getting started](getting-started.md) runs a whole suite in five minutes. - [A worked example](kettle.md) walks through conflicts, exceptions, stories, coverage and a failing run. - [The language](language.md) is the reference for `.expect` files. - The [example manual](../example/site/index.html) is what the generated docs look like. --- # For AI agents Everything an agent needs is available as plain text, so it does not have to scrape HTML. | What | Where | | --- | --- | | Index of the docs, for LLMs | [`/llms.txt`](../llms.txt) | | The whole guide as one file | [`/llms-full.txt`](../llms-full.txt) | | The overview skill | [`/skills/expectation/SKILL.md`](../skills/expectation/SKILL.md), also [`/skill.md`](../skill.md) | | The feature-specs skill | [`/skills/write-feature-specs/SKILL.md`](../skills/write-feature-specs/SKILL.md) | | Guide source (Markdown) | [`book/src`](https://github.com/ptescher/expectation/tree/main/book/src) in the repository | To use a skill, save the `SKILL.md` to `.claude/skills//SKILL.md` in your project (or your agent's equivalent). ## The short version ```bash cargo install --git https://github.com/ptescher/expectation expectation_cli expectation check expectations # errors exit 1 expectation plan expectations --coverage # the tests the stories compile to expectation docs expectations docs # the manual ``` A suite is every `*.expect` file under a directory: a vocabulary of states and events, expectations (given, when, then), and stories that become tutorials and tests. [The language](language.md) is the reference. ## Rules that matter - Don't invent behaviour. Inspect the product; if something is undecided, write `Unresolved: ` in the description, mark it `Approval: proposed`, and ask. - Write the explanation and the expectations together, then check, plan, generate and read the pages. See [Feature docs that explain](feature-docs.md). - Report assumptions, claims with no evidence, steps the driver can't do yet and open decisions. "Not supported yet" is not "passing", and `Approval:` is separate from both `Status: pending` and the test results. - A manual of just the agreed behaviour: `expectation docs --approved-only`. --- # Getting started This chapter runs a complete suite against a tiny system: a coin-operated turnstile. It's the [`turnstile` example](https://github.com/ptescher/expectation/blob/main/crates/harness/examples/turnstile.rs) in the repository. ## Add the crates Expectation isn't on crates.io yet. Depend on the git repository: ```toml [dev-dependencies] expectation_harness = { git = "https://github.com/ptescher/expectation" } expectation_story = { git = "https://github.com/ptescher/expectation" } ``` For the command-line tool: ```bash cargo install --git https://github.com/ptescher/expectation expectation_cli ``` That installs a binary called `expectation`. ## Write the suite A suite is every `*.expect` file under a directory. Make `expectations/turnstile.expect`: ```gherkin Suite: Turnstile Vocabulary: State: the turnstile is locked | unlocked State: the coin box has 0..1000 coins Event: a visitor inserts a coin Event: a visitor pushes the arm Background: Given the turnstile is locked And the coin box has 0 coins Feature: Paying to go through Expectation: A coin unlocks it Given the turnstile is locked When a visitor inserts a coin Then the turnstile is unlocked And the coin box goes up by 1 Expectation: Pushing through locks it again Given the turnstile is unlocked When a visitor pushes the arm Then the turnstile is locked Expectation: Pushing a locked turnstile does nothing Given the turnstile is locked When a visitor pushes the arm Then the turnstile is locked Story: A day at the zoo As a visitor I want to pay once and go through So that I can see the animals When a visitor inserts a coin Then the turnstile is unlocked When a visitor pushes the arm Then the turnstile is locked ``` - The **vocabulary** declares the states the system can be in, and the events that can happen. Every other line must use these words. - The **Background** is the state every test starts from. - **Expectations** are the rules. `goes up by 1` is a relative Then: it checks the change, whatever the count was. - The **story** is a journey through them. ## Check it ```bash expectation check expectations ``` ```text warning[untested]: no story exercises «Pushing a locked turnstile does nothing» --> turnstile.expect:25 | Expectation: Pushing a locked turnstile does nothing = help: a story that would: When a visitor pushes the arm 0 errors, 1 warning Turnstile: 3 expectations in 1 feature, 1 story (1 test); 0 exceptions; flow DAG 3 edges, 2 levels deep (search truncated) ``` Nothing ran, but the suite was checked: every step uses the vocabulary, no two expectations disagree, and the story only relies on behaviour the expectations specify. The warning says one expectation has no story, and how a story could reach it. "Search truncated" is normal here: every coin makes a new state, so the search for what leads to what stops at its depth limit instead of counting to 1000. It has already found everything by then. Try breaking it. Add a fourth expectation: ```gherkin Expectation: A hard push forces it Given the turnstile is locked When a visitor pushes the arm Then the turnstile is unlocked ``` `check` now fails. Both this and «Pushing a locked turnstile does nothing» apply when a visitor pushes a locked turnstile, and they disagree about the turnstile. The error names that situation and suggests settling it with `Overrides:` if one is an exception to the other. Here neither is, so take the new one out again. ## Bind it to the system The runner talks to the system through a **driver**. `Bindings` builds one from closures, one per vocabulary term: ```rust use expectation_harness::{Acceptance, Bindings, RunOptions}; #[derive(Default)] struct Turnstile { locked: bool, coins: u32, } #[test] fn turnstile_behaves() { let mut driver = Bindings::new(Turnstile { locked: true, coins: 0 }) .observe("the turnstile", |t| if t.locked { "locked" } else { "unlocked" }) .observe("the coin box", |t| t.coins as f64) .arrange("the turnstile", |t, v| t.locked = v.as_str() == Some("locked")) .arrange("the coin box", |t, v| t.coins = v.as_f64().unwrap() as u32) .on("a visitor inserts a coin", |t| { t.coins += 1; t.locked = false; }) .on("a visitor pushes the arm", |t| t.locked = true) .on_reset(|t| *t = Turnstile { locked: true, coins: 0 }); Acceptance::load("expectations") .unwrap_or_else(|e| panic!("{e}")) .with_coverage() .assert_passes(&mut driver, &RunOptions::default()); } ``` - `observe` reads a state back. Named values are strings; numbers are `f64`. - `arrange` puts a state in place before a test starts (the Background, a scenario's Givens). - `on` makes an event happen. - `on_reset` runs before every test. `with_coverage()` adds a generated test for each expectation no story reaches, so all three are verified. `assert_passes` panics with every failing check if anything fails. ```bash cargo test ``` ## Write the docs Pass a `docs_dir` to take screenshots (if the driver has `on_screenshot`), then write the manual: ```rust let acceptance = Acceptance::load("expectations")?.with_coverage(); let opts = RunOptions { docs_dir: Some("docs".into()), ..Default::default() }; let report = acceptance.run(&mut driver, &opts); acceptance.docs(Some(&report)).write("docs")?; ``` `docs/README.md` is the manual in Markdown, and `docs/site/index.html` is the same as a website. The story is now a tutorial called "A day at the zoo". ## Next The [worked example](kettle.md) is a richer system, with time, exceptions, several stories and screenshots. --- # A worked example: the kettle The [`kettle` example](https://github.com/ptescher/expectation/tree/main/crates/harness/examples/kettle) is a suite of five files about an electric kettle, run against a toy kettle with time in it. This chapter walks through it: the vocabulary, a conflict and its exception, stories, generated coverage, a failing run, and the manual it writes. ```bash cargo run -p expectation_harness --example kettle ``` The manual it writes is published here: [the kettle manual](../example/site/index.html). ## The vocabulary `vocabulary.expect` declares every word the suite may use, and where every test starts: ```gherkin Vocabulary: State: the kettle is off | starting | heating | dry State: the dial is 0..100 % State: the preset is WARM | HOT | BOIL State: the child lock is on | off Event: the user presses start Event: the user presses reset Event: the user presses preset Event: the user presses lock Event: the water runs out Background: Given the kettle is off And the dial is 0 % And the preset is WARM And the child lock is off ``` The dial is a number, so steps can compare it: `above 55 %`, `between 20 and 40 %`. The others have named values. "The water runs out" is an event without a person: the docs will say "wait until the water runs out" instead of telling the reader to do it. ## Features and an exception `heating.expect` says how the kettle starts and stops: ```gherkin Feature: Heating The kettle starts off. Turn the dial past 55 % to start it, the way you would an old stovetop knob, or press start. Expectation: Turning the dial up starts the kettle Holding the dial above 55 % for a moment starts it. Given the kettle is off When the dial is above 55 % Then the kettle is heating ... ``` `child_lock.expect` adds a child lock, which keeps the dial from doing anything: ```gherkin Expectation: The child lock holds the kettle while the dial turns Given the kettle is off And the child lock is on When the dial is above 55 % Then the kettle is off ``` Both expectations apply when the kettle is off, the lock is on and the dial goes above 55 %, and they say opposite things. `expectation check` refuses the suite: ```text error[conflict]: «The child lock holds the kettle while the dial turns» and «Turning the dial up starts the kettle» disagree about the kettle --> child_lock.expect:15 | Expectation: The child lock holds the kettle while the dial turns = «The child lock holds the kettle while the dial turns» says the kettle is off --> heating.expect:5 | Expectation: Turning the dial up starts the kettle = «Turning the dial up starts the kettle» says the kettle is heating = note: both apply when the kettle is off and the child lock is on, and the dial is above 55 % = help: «The child lock holds the kettle while the dial turns» is the more specific; if it's an exception, add `Overrides: Turning the dial up starts the kettle` to it. Otherwise narrow a Given so they can't both apply ``` It is an exception, so the example says so: ```gherkin Expectation: The child lock holds the kettle while the dial turns Overrides: Turning the dial up starts the kettle Given the kettle is off And the child lock is on When the dial is above 55 % Then the kettle is off ``` Now the suite checks, and the docs for «Turning the dial up starts the kettle» carry a note: "**Except** when the child lock is on: see The child lock holds the kettle while the dial turns." ## Stories `stories.expect` has three stories. Here's one: ```gherkin Story: Keeping little hands off As a parent I want the child lock to keep the kettle off So that a curious toddler turning the dial doesn't start it Scenario: Turn the dial with the lock on, then unlock and boil When the user presses lock Then the child lock is on When the dial is 100 % Then the kettle is off When the dial is 0 % And the user presses lock Then the child lock is off When the dial is 80 % Then the kettle is heating ``` `expectation plan` shows the test it compiles to. Every check comes from an expectation that fires at that step, so the story's own Thens are only a statement of what the reader should see: ```text Keeping little hands off › Turn the dial with the lock on, then unlock and boil arrange the kettle is off, the dial is 0 %, the preset is WARM and the child lock is off 1. when the user presses lock fires «Lock turns the child lock on» check the child lock is on («Lock turns the child lock on») 2. when the dial is 100 % fires «The child lock holds the kettle while the dial turns» check the kettle is off («The child lock holds the kettle while the dial turns») 3. when the dial is 0 % and the user presses lock fires «Lock again turns it off» check the child lock is off («Lock again turns it off») 4. when the dial is 80 % fires «Turning the dial up starts the kettle» check the kettle is heating («Turning the dial up starts the kettle») ``` A story can't wander off the specification. Without the `Overrides:` line above, step 2 is an error too, because two expectations fire and disagree. A step where nothing is expected to happen is `unspecified`, and a Then that contradicts the expectations is `contradicts`. ## Coverage Four expectations have no story. `check` warns about each one, with a story that would reach it: ```text warning[untested]: no story exercises «Running dry stops the kettle» --> heating.expect:16 | Expectation: Running dry stops the kettle = help: a story that would: When the dial is above 55 % / When the water runs out ``` `Acceptance::with_coverage()` turns those routes into generated tests, so every expectation is verified even if no story mentions it: ```text ✓ First cup › Turn it up, then turn it off ✓ Keeping little hands off › Turn the dial with the lock on, then unlock and boil ✓ Choosing a preset › Step to HOT and start ✓ Reaching «The child lock ignores the start button» ✓ Reaching «Running dry stops the kettle» ✓ Reaching «HOT to BOIL» ✓ Reaching «BOIL back to WARM» ``` ## The driver [`world.rs`](https://github.com/ptescher/expectation/blob/main/crates/harness/examples/kettle/world.rs) is the toy kettle. Like a real one, it takes time: the dial has to stay above 55 % for 0.6 s before the kettle starts, and starting takes another 0.22 s. The driver binds each term and says how time passes: ```rust Bindings::new(World::new(lock_leaks)) .observe("the kettle", |w| w.phase_name()) .observe("the dial", |w| w.dial) .set("the dial", |w, v| w.dial = v.as_f64().unwrap()) .arrange("the kettle", |w, v| /* put it in that phase */) .on("the user presses lock", World::press_lock) // ... .on_settle(|w| (0..20).for_each(|_| w.tick(0.05))) // a second after each action .on_advance(|w| { w.tick(0.1); Progress::Advanced }) // then 0.1 s per extra look .on_screenshot(|w| Screenshot::svg(w.svg())) ``` Settling a full second after each action is what makes "Then the kettle is off" mean it *stays* off, not that it was off for the instant before it switched on. See [Drivers](drivers.md). ## A failing run `--bug` seeds a defect: the child lock lets the dial through. ```bash cargo run -p expectation_harness --example kettle -- out --bug ``` ```text ✗ Keeping little hands off › Turn the dial with the lock on, then unlock and boil at: When the dial is 100 % Then the kettle is off expected the kettle is off («The child lock holds the kettle while the dial turns»); observed heating ``` The failure names the step, the check, the expectation that promised it, and what the kettle actually did. The steps after it are skipped, not failed. In the docs, that expectation is marked failed, and so is the tutorial. ## The manual The run writes a manual. The story became this tutorial, in the reader's words: ```markdown # First cup ## Before you begin - The kettle is off. - The dial is 0 %. - The preset is WARM. - The child lock is off. ## Steps 1. **Set the dial to 80 %.** The kettle is heating. 2. **Set the dial to 0 %. Then press reset.** The kettle is off. ``` Each feature is a topic page, and the test evidence is collapsed under "How this is tested". The vocabulary here has no `Say:`, `Set:` or `Shows:` lines, so the wording is worked out from the steps. With them, step 1 could read "Turn the dial past halfway. The red light comes on." See [Writing specs that read well](writing-specs.md). Browse the result: [the kettle manual](../example/site/index.html). --- # One suite, three apps The [kettle demo](https://github.com/ptescher/expectation/tree/main/demos/kettle) is the same app built three times, in **GPUI**, **SwiftUI** and **React Native**, and held to one suite. Each app embeds the Expectation driver for its stack (see [Embedded drivers](remote.md)), and the same runner drives all three. Browse the result: [the demo manual](../demo/site/index.html). Its Platform menu switches between the three apps' manuals: the same tutorials, with each app's own screenshots and the instructions worded for it. ## One suite The behaviour is written once. The suite declares the platforms, and says where the wording differs: ```gherkin Vocabulary: Platform: GPUI | SwiftUI | React Native State: the kettle is off | starting | heating | dry Shows: heating: the display reads HEATING and the light turns red State: the dial is 0..100 % Set: drag the DIAL slider to {value} Event: the user presses start Say: click START Say[SwiftUI]: tap START Say[React Native]: tap START ``` Everything else (the features, the exception for the child lock, the three stories) is plain, so it's true on every platform. `expectation check` checks the suite as each platform reads it. ## Three apps Each app implements the kettle in its own language and binds the vocabulary to it: | | GPUI | SwiftUI | React Native | | --- | --- | --- | --- | | Kettle | `kettle.rs` | `Kettle.swift` | `kettle.ts` | | Driver | `expectation_remote::client` | `ExpectationDriver` (Swift package) | `expectation-driver` (npm) | | Commands run on | the GPUI foreground, every tick | the main actor | the JS thread | | Screenshots | the app, with `screencapture` | the app, from its key window | the runner, from the simulator | The SwiftUI binding, for example: ```swift let driver = ExpectationDriver(platform: "SwiftUI", app: "Kettle") .observe("the kettle") { .name(k.phase.name) } .observe("the dial") { .number(k.dial) } .set("the dial") { k.dial = $0.number ?? 0 } .on("the user presses start") { k.start() } .on("the water runs out") { k.runOut() } .onReset { k.restore() } driver.start() ``` ## Running it ```bash cd demos/kettle ./run.sh gpui ./run.sh swiftui ./run.sh react-native ``` Each run builds the app, starts `expectation-run --platform `, launches the app, which connects, and saves the results and screenshots. A runner for one platform turns away apps for the others, so leftover apps from an earlier run can't interfere. The site build then writes the manual for all three from what the runs saved: ```bash expectation docs demos/kettle/expectations demos/kettle/docs ``` ## What it shows The three apps are independent implementations, written in three languages. Nothing but the suite keeps them the same. The test reports show the same tests passing on each, and a behaviour change starts in the suite, where all three apps will be held to it. --- # The language A suite is every `*.expect` file under a directory. The format is line oriented, like Gherkin. A line's first word decides what it is, a line starting with `#` is a comment, and indentation is only for people. ```gherkin Vocabulary: State: the kettle is off | starting | heating | dry State: the dial is 0..100 % State: the preset is WARM | HOT | BOIL # How the docs say to change it: Set: press PRESET until the display reads {value} # What the reader sees: Shows: the display reads {value} Event: the user starts the kettle # How the docs say to do it: Say: press START # A setup, in one line: Phrase: the kettle is plugged in and full Say: fill the kettle and plug it in Given the kettle is off And the dial is 0 % And the preset is WARM Phrase: the user starts the kettle with the dial Say: turn the dial past 55 % and hold it there When the dial is 80 % Background: Given the kettle is plugged in and full Feature: Heating # Free text before the first step is a description: The kettle starts off. Expectation: Turning the dial up starts the kettle Given the kettle is off When the dial is above 55 % Then the kettle is heating Expectation: START starts the kettle Given the kettle is off When the user starts the kettle Then the kettle is heating Story: Your first cup As a tea drinker I want to start the kettle by turning the dial So that it works like the one I already have # The tutorial's introduction: Every kettle in the range has the same dial. Scenario: Dial up When the user starts the kettle with the dial Then the kettle is heating ``` ## Suites and sections An optional `Suite: ` line names the suite. Otherwise it's named after its directory. Each directory under the suite is a **section**: a nested set of features that share a setup. Its `Section: ` line (in any of its files) and the text under it name and describe it. A leading `01-` in a directory name only sets the order. A section's `Background:` is its setup, written over its parent's. So everything in `app/drawing/` starts where `app/` does, with the pen picked on top. Pages about a section start from that setup and don't re-explain how to get there, the way a manual assumes you've read the chapter before. A section's `Launch:` line, right under its `Section:` line, says how to start the program under test: `Launch: run cargo run -p kettle`. It is always shown first under "Before you begin" and never as a numbered step, every sub-section inherits it, and a sub-section's own `Launch:` replaces it (`Launch[iOS]:` for another platform). Drivers get it as `Driver::launch`, called once for the tests that share it, so a runner doesn't start its own server. A `Say:` that holds a shell command in backticks (one that starts with a program name or a path, like `` `cargo run` `` or `` `./app` ``) gets a `launch-in-step` warning. Each top-level section is a system of its own: stories and the flow graph only use the expectations in their own top-level section. ## Vocabulary | Line | Declares | | --- | --- | | `State: is \| \| …` | A state with named values. | | `State: is .. [unit]` | A number, optionally with a unit. | | `Event: ` | Something that happens. | | `Phrase: ` | A step of your own, made of other steps. | | `Term: ` | A word for the glossary, defined by the text under it. | | `Reference: Given … When … Then … ``` - `Given` is the state before. Several are joined with `And`. - `When` is at most one event, plus any states the outside world changes (such as the dial). - `Then` is the state after. States the Then doesn't mention keep their values. - An expectation with no `When` is an **invariant**: it holds whenever its Given does. - `Overrides: ` makes it an exception to another expectation: where both apply, this one wins. Overrides must form a DAG. - `Hands over to: ` says that when this fires, a different program (or part of one) takes over, and it starts in that setup: the launcher starts the sim, so afterwards the sim is "just started". A section whose Background is that setup is then reached from here instead of being a start of the state map, which goes on from the setup alone (the launcher's buttons say nothing about the sim). Stories and tests don't see it: they stay in the launcher, and check the expectation's Then. - `Docs: hidden` keeps it off its topic page. It's still tested, and it's in the test report. - `Status: pending` says the behaviour is written ahead of its implementation. A pending expectation stays in the model: it appears in the map and the state map, it can be reached, and conflicts with it (pending or required) are still errors. What changes is testing. A test that depends on it, because it fires it or because it is an invariant in force there, is not run; the report shows it as **pending**, which is neither passed, failed nor "not supported yet" (that one means the driver lacks a binding, and is still reported as such for everything else). Pending tests don't fail the run and never count as verified, and the expectation's own verdict is `pending`. Failures in the other tests still fail the run. Remove the line when the behaviour is built. Because an invariant applies wherever its Given holds, a pending invariant makes the tests in its section pending, so prefer pending transitions. - `Approval: approved` or `Approval: proposed` says whether the specification itself is agreed (see [Approval](#approval)). It says nothing about whether the behaviour is built or passes. Text before the first step is the expectation's description. ## Approval `Approval: approved` or `Approval: proposed` goes right under a `Feature:`, `Expectation:` or `Story:` line (before its description or steps) and says whether the specification itself has been agreed. It is for teams that keep drafts and agreed behaviour in one suite, and it changes nothing about checking, planning or running. - An expectation without its own `Approval:` has its feature's. Nothing says it, and unmarked counts as not approved when an approved-only manual is asked for (see [the manual](manual.md#approved-only-manuals)). - An approved expectation inside a proposed feature is an error (`approval-conflict`). - It is separate from `Status: pending`. A pending expectation is agreed but not built; a proposed one may be built and passing but is not agreed. ## Backgrounds that follow a setup A section's Background can say which setup it is reached from: ```gherkin Background: Follows: the launcher has just opened Given the station is connected to the TX15 And the other plane is silent ``` The section is reached from that setup, not a start of the state map: only the setup that nothing leads to is. Its "Before you begin" starts from the section whose setup it follows. How it's reached depends on whether the expectations can say: - **They can.** Where a chain of expectations gets from the followed setup to this one (a website's home page has a link to billing), that chain is the way in. The state map shows its real moves, and the docs give its steps. With coverage on, a generated test "Reaching «Billing»" arranges the followed setup, takes the route and checks everything the section's setup says. The section's own tests arrange its setup directly (a deep link), so this is the test that shows the way in gets to the same place. - **They change every state the setup needs, but nothing gets there.** That's an error (`follows-no-way-in`): the route it claims is broken. Changing the link so it goes somewhere else gives you this error. - **Some state the setup needs, nothing changes** (the station is connected to the TX15). The Background's steps (and their `Say:`) are what you do, and the state map has one move from the followed setup to the section, named by the section. That move is a claim about how to set up, which nothing tests. If nothing gets to the setup a section follows, that's a `follows-unreached` warning. `Hands over to:` is the same idea for what a tested expectation does. ## Stories ```gherkin Story: As a I want So that Scenario: Given … When … Then … ``` - The narrative is the "As a …", "I want …", "So that …" lines. - Any other text is the introduction, which opens the tutorial in the docs. - A story has one or more `Scenario:`s. Steps written straight under the story form a single scenario. - A scenario may start with `Given`s, which are arranged on top of the Background. - After that, each `When` starts a step, `And`/`But` add to it, and its `Then`s are what the story checks afterwards. ## Links and the glossary `Term: ` and the text under it define a word for the glossary. Any text can link to a term, a section, a feature, an expectation or a story with `[[its name]]`, and mark code with backticks. A term can point to documentation outside the suite with a `Reference:` line, either `Reference: | | …` declares the platforms. The first is the default. - `Say[P]:`, `Set[P]:` and `Shows[P]:` replace the plain line for platform `P`. - `Phrase[P]:` replaces the plain phrase of the same name for `P`. - `On: desktop, iOS` says that the section (the whole directory), feature, expectation or story is only about those platforms. Use it for what a platform doesn't have. An exception to an expectation that isn't on a platform is dropped there. - Tag only the platforms that differ from the default. Tagging the default is an error (`default-platform`), and so is a name that isn't declared (`unknown-platform`). ## Checking, running and docs `expectation check` checks the suite as each platform reads it, and every one must check clean. A driver reads the suite for its platform and runs its tests: ```rust let acceptance = Acceptance::load_for("expectations", "iOS")?; ``` Its results are its own (`results/app@iOS.tsv`), apart from those of runners that are the same everywhere (`api.tsv`). `Acceptance::publish(dir, runner, &report)` saves a run's results and merges them with what other runners saved. `expectation docs` writes one manual per platform: the default at the root and the others under `platforms//`. Every HTML page has a Platform menu that opens the same page in the other manual. --- # Writing specs that read well For how deep a feature description should go, with an example that has an outside service, an unavailable state and an open decision, see [Feature docs that explain](feature-docs.md). The generated manual is only as good as the words in the specs. A suite that tests well but reads badly is half done: the same lines are the user's documentation. This is a loop that works. ## 1. Start from what the user does Write a handful of high-level expectations in the user's words, before any detail. Name states for what the user sees ("the kettle is heating"), not for internals ("the relay is closed"). Name events for what the user means ("the user starts the kettle"), not for the control ("the user presses START"). The control goes in a `Say:`. ## 2. Let the checker shape the model ```bash expectation check expectations expectation graph expectations > flow.mmd ``` - A **conflict** is a question the spec hasn't answered. Decide which behaviour wins, and say so with `Overrides:`, or narrow a Given so they can't both apply. - An **unreachable** expectation means nothing gets the system into its Given. Add the expectation that does, or fix the Background. - **No way in** means a section's setup can't be reached from its parent's. The docs need that route to say "Before you begin". ## 3. Write stories as tutorials One goal per story, in the order a new user meets things. The narrative says who and why. The introduction is the tutorial's first paragraph. A story's Thens are what the reader should notice, not everything that's true; the expectations already check the rest. ```bash expectation plan expectations ``` shows each story as the test it compiles to, and errors when a story leans on behaviour the expectations don't promise. ## 4. Give the docs words Add `Say:` to events and phrases, `Set:` to states the reader changes, and `Shows:` to states the reader sees. Turn long setups into phrases ("the kettle is plugged in and full"), so "Before you begin" is one line, not a list of states. Define jargon with `Term:`. ## 5. Generate and read as a newcomer ```bash expectation docs expectations docs ``` Read `docs/README.md` and the site as someone who has never seen the system. Better: have someone else do it, or ask an AI agent to audit the pages as a reader and list what's vague, missing or confusing. ## 6. Fix the specs, then loop For each finding, change the spec, not the generated page: | Finding | Fix | | --- | --- | | A vague step | a better `Say:`, a phrase, or a `Then` | | A missing why | a description under the feature or expectation | | A confusing title | a better name | | A state dump in "Before you begin" | a setup phrase with a `Say:` | | A word the reader doesn't know | a `Term:` | Then check, regenerate and read again, until the docs read well. ## Style Keep reader-facing text short and plain. Simplified Technical English (ASD-STE100) is a good target: one instruction per sentence, active voice, present tense, common words, no more than about 20 words a sentence. The docs generator already writes instructions this way, so specs in the same style blend in. --- # Feature docs that explain A suite can pass every check and still make a thin manual. The usual cause is a description that lists test conditions and caveats and never says what the feature is for. This chapter is the depth to aim for, a workflow to get there, and a worked example you can run: ```bash expectation check examples/smart-kettle expectation docs examples/smart-kettle docs ``` The example is [`examples/smart-kettle`](https://github.com/ptescher/expectation/tree/main/examples/smart-kettle): a smart kettle that shows the electricity price from an outside provider and boils when the price is cheap. It has a feature that depends on an outside service, a state for when that service is unavailable, and a decision nobody has made yet. The manuals it generates are published: [with its drafts](../smart-kettle/site/index.html) and [approved only](../smart-kettle-approved/site/index.html). If you are an AI agent, there is a [skill](https://github.com/ptescher/expectation/tree/main/.claude/skills/write-feature-specs/SKILL.md) with this chapter's checklist. ## What a feature page needs The text under `Feature:` is the first thing a reader sees. Write it for someone who has never talked to you. For each user-facing feature, cover: 1. **What it does, who uses it, and what it helps them decide or do.** One or two sentences, in the user's words ("someone who makes tea every morning"). 2. **The normal workflow.** What the user does, what goes in, what comes out. 3. **How to read the result.** What a dot, a number, a banner or an empty list means, and what to do about it. 4. **The states that are not the happy path.** Empty, unavailable, invalid, partial, late; and any limit that matters, like "one cheap rate per boil". 5. **What is settled and what is not.** Say what is agreed. If something is undecided, write `Unresolved:` and the question, and mark the expectation `Approval: proposed`. Do not invent an answer to make the page read smoothly; ask for the decision. 6. **The promises the tests check.** These are the `Expectation:`s under the description. Leave filler out, not context. A paragraph per point is normal. A description that a reader could not follow without having sat in the planning meeting is too thin, however short. Each expectation can have a description too. Use it for the *why* of one rule ("A number that might be hours old is worse than none"). Stories get an introduction that says the goal and the situation; the narrative (`As a … I want … So that …`) is the reader's reason. ## An end-to-end example Start with what the user can see about things you do not control. The price comes from an outside provider, so its health is a state of the vocabulary, with a `Shows:` for how the kettle tells the user: ```gherkin Vocabulary: State: the price feed is live | delayed | unavailable Set: switch the test provider to {value} Shows: live: the price has a green dot Shows: delayed: the price has an amber dot and says when it was last updated Shows: unavailable: the price is replaced by the message "Prices unavailable" and a Retry button State: the cheap boil is off | waiting | boiling Event: the feed stops answering Say: switch the test provider off Term: Price provider The outside service that sends the electricity price. The kettle only uses the price the provider sends. Reference: Provider API | https://example.com/docs/prices ``` The feature description explains the feature, links the term, and covers the unavailable state in plain words: ```gherkin Feature: Seeing the electricity price Approval: approved The smart kettle is for people who want to spend less on hot drinks, such as someone who makes tea every morning and would rather boil when power is cheap. They open the app and read the price. They use it to decide whether to boil now or wait. The price comes from an outside provider ([[Price provider]]). The kettle shows what the provider sends and does not work out prices by itself. Next to the price, a dot says how much to trust it. Green means the price is current. Amber means the provider is late, so the price is old: the app says when it was last updated, and you should treat the price as a guide only. If the provider does not answer, the app has no current price to show. It says so and does not show an old price as if it were new. When the provider is back, tap Retry to get the price again. Expectation: A silent provider means no price The app says "Prices unavailable" and shows no number. A price that might be hours old is worse than none. Given the price feed is live When the feed stops answering Then the price feed is unavailable ``` The undecided part is written down, not guessed. This expectation is `proposed`, and its description says what the question is: ```gherkin Expectation: A delayed price can start the boil Approval: proposed Unresolved: should an old price start a boil? Boiling sooner may save money, but the real price may already have gone back up, and nobody has decided whether the kettle may start while the price is in doubt. Until it does, this is proposed and an approved manual leaves it out. Given the cheap boil is waiting And the price feed is delayed When the price drops to the cheap rate Then the cheap boil is boiling ``` And a story walks the journey, including the unavailable state, because that is when a reader most needs to know what the kettle does: ```gherkin Story: Boil when the power is cheap Approval: approved As a tea drinker I want the kettle to boil when electricity is cheap So that my morning cup costs less without me watching the price You make tea every morning and your power is cheaper at some hours. Instead of checking the price, you ask for a cheap boil. This tutorial also shows what happens when the price provider goes quiet, because that is when you most need to know what the kettle is doing. Scenario: Ask for a cheap boil, lose the provider, get the boil When the user asks for a cheap boil Then the cheap boil is waiting When the feed stops answering Then the price feed is unavailable And the cheap boil is waiting When the feed answers again Then the price feed is live When the price drops to the cheap rate Then the cheap boil is boiling ``` ### What it generates The feature page (`features/boiling-when-it-is-cheap.md`; the HTML site's page is the same) opens with the description, then one section per expectation, with the *why*, the instruction, and the test folded away: > # Boiling when it is cheap > > A cheap boil tells the kettle to wait until the price is low, so you do not have to watch > it. […] > > ## A cheap price starts the boil > > The kettle starts the moment a live price reaches your cheap rate. > > If the cheap boil is waiting and the price feed is live, have the test provider send a price > at or below your cheap rate. The display reads HEATING and a banner says the price is cheap. The tutorial turns the story into steps, with the setup first: > # Boil when the power is cheap > > You make tea every morning […] > > ## Before you begin > > - Switch the test provider to live. > - The cheap boil is off. > > ## Steps > > 1. **Fill the kettle, set the price you call cheap and tap Boil when cheap.** The Cheap boil > button reads "Waiting for a cheap price". > 2. **Switch the test provider off.** The price is replaced by the message "Prices > unavailable" and a Retry button. The Cheap boil button reads "Waiting for a cheap price". > 3. **Switch the test provider back on and tap Retry.** The price has a green dot. > 4. **Have the test provider send a price at or below your cheap rate.** The display reads > HEATING and a banner says the price is cheap. Where the model has no `Say:` or `Shows:` for something, the page falls back to a plain sentence made from the step. That is the cue to add one. ## The workflow 1. **Look at the product first.** Read the code, the UI, existing docs, tickets and the people who know it. Note what you can see and what you are assuming. 2. **Write the explanation and the expectations together.** The description, the vocabulary's `Say:`/`Set:`/`Shows:`, the expectations and one story per user goal. A description written afterwards drifts from what is tested. 3. **Check.** `expectation check ` must pass. Fix every error; read every warning. 4. **Plan with coverage.** `expectation plan --coverage` shows the tests each story becomes and the generated ones for expectations no story reaches. Add a story, or accept a generated test, for each `untested` warning. 5. **Generate the manual.** `expectation docs `, with results from a run if you have them. 6. **Read the pages as the reader.** Open the feature pages and tutorials. Ask what a newcomer would not understand, and what you would have to be told in person. 7. **Report what you found.** List the assumptions you made, the claims with no evidence (nothing in the code or product backs them), the steps the driver cannot do yet (they show as "not supported yet", not as passing), and the open decisions. Do not present any of those as verified behaviour. Mark undecided behaviour `Approval: proposed` and ask. ## Settled and proposed `Approval: approved` and `Approval: proposed` mark what is agreed and what is not; see [Approved-only manuals](manual.md#approved-only-manuals). A suite that does not use them is unchanged. Approval is about the *specification*; it is separate from `Status: pending` (the behaviour is not built yet) and from the test results. --- # How expectations are compared A condition (a Given, a Then, a trigger) stands for a set of states: every combination of values that satisfies it. Everything below comes down to three questions about those sets: - do two overlap? - does one imply the other? - what's left after an action overwrites some states? Named values are bit sets and numbers are intervals, so the answers are exact. They are never guesses from sampling. ## Outcome conflicts Two expectations **conflict** when the same action can set off both, their Givens overlap, and their Thens contradict. In the overlap, the system can't do both. The error names the overlap ("both apply when the kettle is off and the child lock is on, and the dial is above 55 %") and suggests which is the exception: the more specific one. ## Invariant conflicts An expectation with no `When` is an invariant: it holds whenever its Given does. It conflicts with an expectation that leaves the system where the invariant's Given holds but its Then can't, and with a Background that does the same. ## Exceptions An `Overrides:` between the two, directly or through a chain, settles a conflict. Where both apply, the overriding one's Then wins. - Overrides must form a DAG. A cycle is an error (`override-cycle`). - An override between two expectations that never disagree gets a warning (`unneeded-override`). - Settled conflicts are listed in the docs as the suite's exceptions, and the overridden expectation's page links to each one: "**Except** when the child lock is on: see …". ## Relative Thens Two different relative Thens ("goes up by 1" and "goes up by 2") always disagree. A relative one and an absolute one disagree unless the shared situation pins down the one value where they meet. ## The flow DAG Starting from each section's setup, and from each scenario's arranged start, a breadth-first search applies every transition that can fire, and sees what state it leaves. - It only uses expectations from the same top-level section. - An expectation's **level** is the fewest steps before it applies. Its parents are the expectations that take the last step on those shortest routes. - Levels only grow along edges, so the result is a DAG even though the behaviour loops (start, run dry, reset, start again). - Each expectation gets a shortest **route** from the start, which is a ready-made test for it. That's what generated coverage tests follow. - An expectation the search never reaches gets an `unreachable` warning: nothing can test it. The search has limits on depth and the number of states it visits. A counter that can keep going up ("the coin box goes up by 1") makes new states forever, so the search stops at its limit and says "search truncated". By then it has usually found every expectation. `expectation graph` prints the DAG as Mermaid (or DOT with `--dot`), and the manual's map page draws it, coloured by verdict. ## Ways into a section For each section, a goal-directed search finds the chain of expectations that gets from its parent's setup to its own: "press PEN", or "sign in, open a project, press PEN" for something deeper. That chain, without any step it works without, is the "Before you begin" of the section's page. If there is no such chain, it's a `no-way-in` warning, because then nothing explains how to get there. --- # From stories to acceptance tests A story describes a journey. It doesn't have to say everything that happens along the way, because the expectations already do. Planning walks each scenario through the expectations, and turns it into a test **plan**: what to arrange, what to do at each step, and what to check after it. ## What fires An expectation **fires** at a step when the current state implies its Given, and the step's action implies its trigger. - In a compound action ("When the dial is 0 % and the user presses reset"), the other changes happen first, then the event. - When an exception and the expectation it overrides both fire, only the exception's Then applies. - After a step, the state is the old state with the Thens of what fired written over it. ## What's checked Each step's checks are: - the Thens of the expectations that fired, - the Thens of invariants now in force, - and the story's own Thens. The story never has to restate what the expectations promise. The tests check it anyway, so a story that only says "When the dial is 80 %" still checks that the kettle is heating. ## What's rejected Planning is strict, because a story that relies on unspecified behaviour would test something nobody agreed on. | Code | | When | | --- | --- | --- | | `unclear-step` | error | The step *might* set off an expectation, but the story doesn't pin down whether. For example "the dial is at least 50 %" when an expectation needs "above 55 %". | | `unspecified` | error | An event happens and no expectation says what it does. | | `contradicts` | error | One of the story's Thens contradicts what the expectations say. | | `clash` | error | Two expectations fire and disagree, and nothing overrides. | | `unpromised` | warning | One of the story's Thens isn't promised by any expectation. It's checked anyway. | ## Where a test starts A test starts from its section's setup (the Backgrounds, from the root down), with its scenario's Givens written over it. The runner **arranges** that state before the first step, choosing a value inside each constraint: the first allowed name, or the middle of a range. ## Coverage An expectation that no story exercises gets an `untested` warning, with a route that would reach it: ```text warning[untested]: no story exercises «Running dry stops the kettle» = help: a story that would: When the dial is above 55 % / When the water runs out ``` `Acceptance::with_coverage()` (and `expectation plan --coverage`) adds a generated test for each one, following that route from the expectation's own section setup. It also adds a test for each section that `Follows:` a setup the expectations get from, taking that route in and checking the section's setup at the end. Generated tests are named "Reaching «…»". They appear in the test report, but not as tutorials. With coverage, every expectation gets a **verdict**: | Verdict | Means | | --- | --- | | verified | Every check it promised passed in at least one test, and none failed. | | failed | A check it promised failed. | | not supported yet | The tests that reach it can't run yet: the driver can't do a step, or a service isn't available. | | untested / not run | No test reaches it, or the tests haven't run. | --- # Drivers The runner talks to the system under test through a **driver**, in the vocabulary's terms: set this state, make this event happen, what is that state now. You can implement the `Driver` trait, or build one from closures with `Bindings`. ## Bindings `Bindings` wraps your system (any value, called the world) and one closure per vocabulary term: ```rust use expectation_harness::{Bindings, Progress, Screenshot}; let mut driver = Bindings::new(Kettle::default()) .observe("the kettle", |k| k.phase_name()) // read a state .observe("the dial", |k| k.dial) .set("the dial", |k, v| k.dial = v.as_f64().unwrap()) // change a state, as an action .arrange("the kettle", |k, v| k.force(v.as_str().unwrap())) // put a state in place before a test .on("the user presses reset", Kettle::reset) // make an event happen .on("the water runs out", Kettle::run_out) .on_reset(|k| *k = Kettle::default()) // before every test .on_settle(|k| k.run_for(1.0)) // after each action .on_advance(|k| { k.run_for(0.1); Progress::Advanced }) // between looks .on_screenshot(|k| Screenshot::svg(k.svg())); // after each step ``` Terms are matched the way steps are: ignoring case, extra spaces and a leading "the". | Method | Used for | | --- | --- | | `observe(subject, f)` | Reading a state. Return a `&str` for a named value, or a number. Named values match the vocabulary ignoring case. | | `set(subject, f)` | A `When` that changes a state ("When the dial is 80 %"). Also used to arrange, unless there's an `arrange`. | | `arrange(subject, f)` | Putting a state in place before the first step (the Background, a scenario's Givens). | | `on(event, f)` | Making an event happen. | | `on_reset(f)` | Putting the system back to the start before each test. | | `on_settle(f)` | Letting the system settle after each action, before anything is observed. | | `on_advance(f)` | Letting a little time pass before the runner looks again. | | `on_screenshot(f)` | A picture of the system after each step: `Screenshot::png`, `jpeg` or `svg`. | | `needs(&[…])`, `fresh(&[…])` | The [external services](services.md) the tests need. | The values passed to `set` and `arrange` are `Value`s: `v.as_str()` for a named value, `v.as_f64()` for a number. ## Time: settling and polling Real systems take time to react. For each step the runner: 1. acts: sets the states and makes the event happen; 2. lets the system **settle**; 3. observes every check. If they don't all hold, it **advances** and observes again, up to `RunOptions::patience` times (100 by default). The step passes as soon as every check holds at once. Polling is what lets "Then the kettle is heating" wait for the kettle to warm up. Settling is what makes "Then the kettle is off" mean that it *stays* off. Without it, the check could pass in the instant before the kettle switches on. Make the settle long enough for whatever should happen to happen, and for whatever shouldn't to show. A system with no time in it needs neither: the default `advance` returns `Progress::Static`, and the runner stops looking after the first observation. ## Not supported yet Before a test starts, the runner checks that the driver can do everything the plan needs: arrange each starting state, set each changed state, perform each event, observe each check. If it can't, the test is **not supported yet**: ```text The driver can't arrange the child lock, arrange the dial, arrange the kettle and 6 more things yet. ``` That isn't a failure. The system, or its driver, just doesn't do it yet. The test's steps get a placeholder instead of a screenshot, and the expectations only it would check are "not supported yet" in the docs. This lets a suite describe a whole product while its drivers catch up, and keeps the gap visible. A driver that can arrange some values of a state and not others overrides `Driver::can_arrange`. ## Controls: what exists, and what was used A driver can list the buttons, fields, switches and links a person can act on, and say which ones its actions used. Then the report shows every control the tests found and which of them no test ever used, as `interactions.md`, `site/interactions.html` and `interactions.json` (for CI). ```ignore Bindings::new(World::default()) .on_controls(|w| vec![Control::new("Settings", "export-log", "button", "Export log")]) .on_used(|w| std::mem::take(&mut w.used)) // Control::key()s since the last call ``` Say which vocabulary event using a control makes happen with `Control::mapped_to("the user presses lock")`, and the report joins the control to the behaviour specified for that event and the stories about it. Each control then shows its gaps, with the reason: *no event mapping*, *event not in the vocabulary*, *no expectation*, *only pending expectations*, *no authored story*, *never used by a test*, *no passing execution*. A story that performs the event in a section named like the control's view documents the control even if it couldn't run; it still isn't verification. The same event on a control in another view is a different row, and what covers one doesn't cover the other. An integration that knows controls no run will reach can list them with `Acceptance::with_inventory`. `Driver::controls` is asked after a test's start is arranged and after each step; the report keeps the union. `Driver::take_used` is emptied after arranging, so setting a test up doesn't count, and read after each step. Give each control an id that doesn't depend on its label, and the view it is in: the same id in two views is two controls. A story and a generated test are counted separately, and a control used only by tests that failed is not shown as working. The list is what the driver reported, not proof that nothing else is on screen. Turn it off with `RunOptions { controls: false, .. }` if listing is slow. ## The Driver trait For full control, implement `Driver` yourself. Only `set`, `perform` and `observe` are required: ```rust pub trait Driver { fn needs(&self) -> Vec { vec![] } fn fresh(&self) -> Vec { vec![] } fn reset(&mut self) -> Result<(), String> { Ok(()) } fn arrange(&mut self, var: &Variable, value: &Value) -> Result<(), String> { self.set(var, value) } fn set(&mut self, var: &Variable, value: &Value) -> Result<(), String>; fn perform(&mut self, event: &Event) -> Result<(), String>; fn observe(&mut self, var: &Variable) -> Result; fn can_arrange(&self, var: &Variable, value: &Value) -> bool { true } fn settle(&mut self) {} fn advance(&mut self) -> Progress { Progress::Static } fn screenshot(&mut self) -> Result, String> { Ok(None) } fn supports(&self, need: Need<'_>) -> bool { true } } ``` `Variable::key` and `Event`'s `key` are the normalised names, for matching. Calls may block, so a driver can talk to a browser, a device or another thread. Any `Err` fails the step with that message. ## Running ```rust let acceptance = Acceptance::load("expectations")?.with_coverage(); // Run, and panic with every failure if anything failed: acceptance.assert_passes(&mut driver, &RunOptions::default()); // Or keep the report: let opts = RunOptions { docs_dir: Some("docs".into()), ..Default::default() }; let report = acceptance.run(&mut driver, &opts); println!("{}", report.failures(&acceptance.suite, &acceptance.plans)); acceptance.docs(Some(&report)).write("docs")?; ``` With a `docs_dir`, screenshots go to `/screenshots//.`. ### Several runners `Acceptance::publish(dir, runner, &report)` saves a run's results as `dir/results/.tsv`, merges them with what other runners saved there, and writes the docs from the merged report. An API runner and a UI runner can each run their own part of one suite, and the manual shows both. `expectation docs` rebuilds the docs from saved results without running anything. Results are keyed by each test's name and a fingerprint of its plan, so results saved for an older version of a test are ignored, not misapplied. ## Running in parallel `run_parallel` runs the tests on `n` worker threads. Each worker builds its own driver (and its own services) by calling your closure with the worker's number, so nothing is shared: give each one its own app instance, port, database or screen. ```rust,ignore let report = acceptance.run_parallel(4, &RunOptions::default(), |n| { Worker::new(MyDriver::start(8000 + n as u16)) .with_services(services_for(n)) }); ``` Workers take the next test as they finish one, and the report is in the order of the tests. The closure runs on the worker's thread, so a driver need not be `Send`. --- # Embedded drivers Some systems can't be driven from inside a Rust test: a phone app in a simulator, a desktop app in its own process, a React Native bundle. For those, the driver goes **inside the app**, and the runner talks to it over a small protocol. ```text runner (Rust) app (any stack) ┌────────────────────────┐ POST /hello ┌─────────────────────────────┐ │ Acceptance + RemoteDriver│ ◀──────────────────── │ embedded Expectation driver │ │ listens on :4477 │ GET /next (long poll) │ observe / set / arrange / │ │ │ ────────────────────▶ │ on(event) / reset / │ │ │ POST /reply │ screenshot │ └────────────────────────┘ ◀──────────────────── └─────────────────────────────┘ ``` The app connects out to the runner, so it never hosts a server: that suits phones, sandboxes and simulators. The embedded driver says hello, long-polls for commands, runs each one on the app's own thread, and posts the answer. When no runner is listening it waits quietly and retries, so it can stay in development builds. ## In the app Each client binds the vocabulary to the app the same way `Bindings` does in Rust: one closure per term. ### Swift (iOS, macOS, SwiftUI, UIKit) ```swift // Package: https://github.com/ptescher/expectation, path clients/swift import ExpectationDriver let driver = ExpectationDriver(platform: "SwiftUI", app: "Kettle") .observe("the kettle") { .name(kettle.phase.name) } .observe("the dial") { .number(kettle.dial) } .set("the dial") { kettle.dial = $0.number ?? 0 } .arrange("the kettle") { kettle.force($0.string ?? "off") } .on("the user presses lock") { kettle.pressLock() } .onReset { kettle.restore() } driver.start() // connects only when EXPECTATION_URL is set ``` Handlers run on the main actor. On UIKit platforms, screenshots of the key window are taken for you. The app needs `NSAllowsLocalNetworking` under `NSAppTransportSecurity` in its Info.plist to reach the runner over plain HTTP. In the simulator, pass the runner's address with `SIMCTL_CHILD_EXPECTATION_URL=http://127.0.0.1:4477 xcrun simctl launch …`. ### JavaScript (React Native, web, Node) ```js // npm: clients/js in the repository (expectation-driver) import { ExpectationDriver } from 'expectation-driver'; const driver = new ExpectationDriver({ platform: 'React Native', app: 'Kettle' }) .observe('the kettle', () => kettle.phase) .set('the dial', v => kettle.setDial(v)) .on('the user presses lock', () => kettle.pressLock()) .onReset(() => kettle.restore()); if (__DEV__) driver.start('http://127.0.0.1:4477'); ``` Handlers may be async. There's no built-in screenshot in JavaScript: pass `onScreenshot` (returning a base64 PNG), or let the runner take them from outside. ### Rust (GPUI, egui, Bevy, anything with a main loop) ```rust use expectation_remote::client::{Client, Handlers}; let handlers = Handlers::new() .observe("the kettle", |v: &KettleView| v.kettle.phase_name()) .set("the dial", |v, x| v.kettle.dial = x.as_f64().unwrap_or(0.0)) .on("the user presses lock", |v| v.kettle.press_lock()); let client = Client::connect_env(handlers.hello("GPUI", "Kettle")); // On the UI thread, every frame or tick: while let Some(request) = client.try_recv() { handlers.handle(&mut view, request); } ``` The client runs on its own thread; `try_recv` hands each command to the thread that owns the app. ## In the runner ```rust use expectation_remote::RemoteDriver; let mut driver = RemoteDriver::listen("127.0.0.1:4477")? .settle_for(Duration::from_millis(1000)) .screenshot_with(|| simulator_screenshot()); // optional: from outside the app let app = driver.wait_for_app(Duration::from_secs(60))?; let acceptance = Acceptance::load_for("expectations", &app.platform)?.with_coverage(); let report = acceptance.run(&mut driver, &RunOptions::default()); ``` `RemoteDriver` knows what the app can do from its hello, so anything the app didn't bind is "not supported yet", not a failure. Time is real: it sleeps to settle after each action and between looks. If no app connects, every test is "not supported yet". Each command the runner sends gets 10 seconds to be answered. For apps whose actions take longer, `.command_timeout(Duration::from_secs(240))?` raises that deadline (it must be positive; there is no unlimited mode). It is separate from `wait_for_app`, which only covers the first hello, and from the settle and poll intervals. A timeout fails the command, naming it and the deadline, but it doesn't cancel the work in the app: let it finish, or stop it safely, before the next action or before ending the session. Or skip writing a runner and use `expectation-run`: ```bash cargo install --git https://github.com/ptescher/expectation expectation_remote expectation-run expectations --platform SwiftUI --docs docs --runner app ``` `--stylesheet CSS` and `--state-shot-ratio W:H` style the generated docs as they do for `expectation docs` (see [the manual](manual.md#your-own-theme)); bad values fail before the run. `--command-timeout SECONDS` sets that per-command deadline (`--wait` only covers connecting). It listens, waits for the app, runs the suite for that platform, prints the results, and with `--docs` saves them and writes the manual. `--screenshot-cmd 'xcrun simctl io booted screenshot "$1"'` takes screenshots from outside, for apps that don't take their own. ## The protocol JSON over HTTP/1.1. Anything that can make an HTTP request can be a client. | Request | Body | Answer | | --- | --- | --- | | `POST /hello` | `{"platform", "app", "observe": [..], "set": [..], "arrange": [..], "events": [..], "screenshot": bool, "reset": bool, "controls": bool}` | `{"session": n}` | | `GET /next?session=n` | | a command (200), nothing yet (204: ask again), or 410: say hello again | | `POST /reply?session=n` | `{"id", "value"}`, `{"id", "png"}`, `{"id", "svg"}`, `{"id"}` or `{"id", "error"}` | 204 | A command is one of: ```json {"id": 1, "op": "reset"} {"id": 2, "op": "arrange", "subject": "the kettle", "value": "off"} {"id": 3, "op": "set", "subject": "the dial", "value": 80} {"id": 4, "op": "perform", "event": "the user presses lock"} {"id": 5, "op": "observe", "subject": "the kettle"} {"id": 6, "op": "screenshot"} {"id": 7, "op": "controls"} {"id": 8, "op": "used"} ``` Terms are the app's own spelling from its hello. A named value is a JSON string and a number is a JSON number. `png` is base64. An app that says `"controls": true` can list what a person can act on, and the report then shows which controls no test used (see [Controls](drivers.md#controls-what-exists-and-what-was-used)). `controls` answers with `controls`, a list of `{"view", "id", "kind", "label", "event"}` (the last three optional; `event` is the vocabulary event using the control makes happen). `used` answers with `used`, a list of `{"view", "id"}` for the controls `set` and `perform` acted on since the last `used`, and empties it. In Rust that is `Handlers::on_controls` and `on_used`, in JavaScript `onControls` and `onUsed`, in Swift `controls` and `used`. ## Events: real taps or the app's actions? An embedded driver makes an event happen by calling the same action the button calls, not by synthesising a tap. That's fast, works the same on every stack, and tests the behaviour. It doesn't test that the button is wired to the action, or that it's visible. If that matters, pair the suite with a UI-automation smoke test (XCUITest, Espresso, Detox), or bind the event to a handler that drives the UI. --- # External services Tests sometimes need something that isn't part of the system under test: a database, a server, a container. The harness treats that as a **service**, so no test, driver or script has to start it. ```rust use expectation_harness::service::{Command, Docker, Ready, Services}; let mut services = Services::new() .with(Command::new("server", "./target/debug/server").ready(Ready::Tcp(8080))) .with(Docker::new("database", "postgres:16").port(5432, 55432) // a database is a container .env("POSTGRES_PASSWORD", "test").ready(Ready::Tcp(55432))); let mut driver = Bindings::new(world).needs(&["server", "database"]).fresh(&["database"]); let report = acceptance.run_with(&mut driver, &RunOptions::default(), &mut services); ``` ## Needs A driver says what it needs (`Driver::needs`, or `Bindings::needs`). Before each test, the runner makes sure each one is up: one that is up stays up, and one that died is started again. They all stop when `services` is dropped, the last started first. A test the driver can't run anyway starts nothing. ## Fresh What a test leaves behind (rows the last test wrote, a user it signed in) is cleared by naming the service in `Driver::fresh`. It's started afresh before each test that runs. ## Not supported yet, not failed A service that can't run on this machine (no Docker, no build) makes the tests that need it "not supported yet", and the reason says how to get it: ```text database: install PostgreSQL with `brew install postgresql` ``` A test suite on a laptop without Docker still runs everything else, and the gap is visible in the docs. ## Providers | Provider | Starts | | --- | --- | | `Command::new(name, program)` | A program, with `.arg`, `.env` and `.hint` (how to install it). Ready when `.ready(…)` says so. | | `Docker::new(name, image)` | A container, with `.port(inside, outside)`, `.env`, `.container(name)`. `.build_from(dir)` builds the image when it isn't there. On a Mac, it wakes Docker Desktop. | | `Present::new(name, path)` | Nothing: a build or a file that must exist. | | `Fallback::new(name, ways)` | The first of several ways that's available: a native build, else a container. | `Ready::Tcp(port)` waits until the port answers. A port that something else already holds is an error, not a ready service: whatever answers there is probably left over from an earlier run. Anything else implements the `Service` trait: a name, whether it can run here (`unavailable`), and `start`, which returns a `Running` that stops when it's dropped. --- # The generated manual `Docs::write` produces a manual, both as an HTML site in `site/` and as Markdown that GitHub renders. It reads like a project's user documentation, not a test report. The test evidence is all there, folded away. ```rust acceptance.docs(Some(&report)).write("docs")?; ``` or, from results that runs have saved in `docs/results/`: ```bash expectation docs expectations docs ``` ## Approved-only manuals Mark what is agreed with `Approval: approved` and what is not with `Approval: proposed` (see [the language](language.md#approval)), then write a manual of just the agreed part: ```bash expectation docs expectations docs-approved --approved-only --results docs/results ``` The full suite is still checked first and must be clean. Then only what is approved is kept, before the suite is read, so the manual is one where the rest was never written: - A feature stays if any of its expectations is approved (its own, or its feature's). Its description stays only if the feature itself is approved. A feature with nothing approved is gone, with its page, navigation entry and every link to it; a `[[link]]` to something left out reads as plain text. - A story is a tutorial only if it is approved itself. Its test page goes with it. - The vocabulary, terms and backgrounds stay. They aren't behaviour. - An approved story or expectation that depends on one that isn't (a story steps through a proposed expectation, or an approved expectation overrides one) is an error that names the problem, so the approved manual can't present proposed behaviour as agreed. Approve the other one, or leave this out. - Nothing marked approved means an empty manual, with a warning. `--results` says where the saved results are, since the output directory is usually not the one the full manual and its `results/` live in. Test results are what runs saw and are shown as they are: an approved specification may still fail, or not be testable yet. Without the flag, nothing changes, and neither do checking, planning or running. ## Your own theme The site loads a CSS file of yours after its built-in styles, so a team can give the reports its documentation's look without forking or editing generated files: ```bash expectation docs expectations docs --stylesheet brand.css ``` ```rust let acceptance = Acceptance::load("expectations")?.with_stylesheet("brand.css")?; ``` `expectation-run` takes `--stylesheet` too. The file is copied to `site/assets/custom.css` and linked from every page of every platform's manual: tutorials, test results, interaction coverage, both maps and the theme page. Put a file called `theme.css` in the suite's directory and it is used without the flag, so regenerating, or restyling from saved results without running a test, keeps the theme. Results, screenshots, links and pass/fail never depend on it. A stylesheet that is missing, empty or has unbalanced braces is an error, not a fallback. Without one, the site is exactly the built-in Day and Night. Override the colour, type and shape tokens (`--canvas`, `--ok`, `--font-sans`, `--radius-panel`, …; the generated theme page lists them all) or any component's CSS by selector. The built-in dark rule is more specific than `:root`, so set a token for Day (`:root`), for dark readers (`@media (prefers-color-scheme: dark) { :root:not([data-theme="day"]) { … } }`) and for `:root[data-theme="night"]`. The file is copied as written: a relative `url()` in it resolves beside `site/assets/custom.css`, so use `data:` URLs to stay self-contained and usable offline. For accessibility keep text at 4.5:1 against its surface in each mode, and keep result states distinguishable without colour: the words *verified*, *failed* and *not supported yet* stay on the page. The state map's cards draw each screenshot at the shape you give with `--state-shot-ratio` (`with_shot_ratio` in Rust), for example `9:19.5` for phone screens. See [the kettle manual](../example/site/index.html) for a whole one. ## Pages - **Tutorials**, one per story scenario: the story's introduction, **Before you begin** (the setup, in the specs' own words), then numbered steps. Each step says what to do (from the `Say:`s and `Set:`s), what you'll see (the story's Thens, or what the expectations promise) and shows the screenshot the test took there. - **Topic pages**, one per feature: its description, then each expectation as a sentence ("If the kettle is off, press START. The light comes on and the water starts to heat."), its exceptions as notes, a screenshot of it in action, and the tutorials that show it. - **Section pages**: the description, Before you begin (the setup, or the steps that get there from the parent section), the topics in it, and its tutorials as cards. - **Reference**: the glossary of `Term:`s, the map of the flow DAG, the theme, and the **test report**: every test and its result, and a page per test with each step, check and screenshot. - **Test vocabulary**: each state, event and phrase, and how the docs word it. On every page, the test details of each expectation (its Gherkin, what it follows and leads to, its verdict and the tests that cover it) are collapsed under "How this is tested". Generated coverage tests appear only in the test report. A step without a screenshot shows a "Not supported yet" or "No screenshot" placeholder. ## Markdown The Markdown has the same pages: ```text README.md the overview and contents tutorials/ one per scenario features/ one per feature glossary.md tests.md the test report, with a Mermaid map of the flow DAG tests/ one per test vocabulary.md screenshots/ site/ the HTML version ``` Pages read in order: each section, then its tutorials, its topics and its subsections. Each page's footer names the spec file to edit to change it. ## The outline By default the manual follows the sections. A suite can have an `outline.txt` beside its sections to shape it like a book: ```text Part: Start here Page: installing manual/installing.md Page: the-installer [desktop] Part: Brewing Section: The kettle ``` - **Parts** group the top-level sections. They're the navigation's labels and the README's contents. - The order is the order the reader meets them in, and so the previous and next links. - **Pages** are hand-written Markdown files in `manual/` (install, reference, troubleshooting), placed among the generated ones. A page's title is its first `# ` line. They can use `[[links]]` into the generated pages. `[desktop]` keeps a page out of another platform's manual. The sections themselves stay where they are, because their setups inherit from their parents. A top-level section that no part names goes into a last part, "More", with a warning. ## The look The site is plain HTML and CSS, with no scripts. It follows the reader's light or dark preference. The colours, type and corners come from the `expectation_theme` crate, and the theme page in each manual shows them. --- # The expectation command `expectation` checks a suite, shows its graph and plans, and writes its docs, without running anything. Running needs the system under test, so that happens through `expectation_harness` in your own tests or program, which saves the results for the docs. ```bash cargo install --git https://github.com/ptescher/expectation expectation_cli ``` Or, from a checkout: `cargo run -p expectation_cli -- `. ## check ```bash expectation check ``` Loads the suite, checks the vocabulary, conflicts, overrides and stories, and prints every diagnostic with where it is and what to do. A suite with platforms is checked as each platform reads it. Ends with a one-line summary: ```text Kettle: 11 expectations in 3 features, 3 stories (3 tests); 2 exceptions; flow DAG 12 edges, 3 levels deep ``` Exits 1 if there are errors, so it can run in CI. ## graph ```bash expectation graph # Mermaid expectation graph --dot # Graphviz DOT ``` Prints the flow DAG: each feature as a group, each expectation as a node (invariants as hexagons), arrows from the start and between expectations, dotted arrows for overrides, and red ones for conflicts nobody has settled. ## states ```bash expectation states # Mermaid state diagram expectation states --dot # Graphviz DOT expectation states --by "the screen,the dialog" # tell states apart by these ``` Prints the state map: every state the app can get into from each section's setup, and every action that moves it from one to another (`[*]` marks the starts). Where `graph` is about expectations, this is about the app: its screens and modes and how to get around them. A section whose setup another expectation `Hands over to:` is not a start: it is reached from there. States are told apart by the vocabulary's enumerated states unless `--by` names others, so a number like a dial's reading doesn't make a state per value. Name the state that is your app's screen to get one node per screen; actions that change only what `--by` leaves out show as staying put. The docs have the same map as `site/states.html`: drag to pan, scroll to zoom, and pick a state to see its screenshot (from a test step that was in it), what can be done there, the expectations behind each move, and the ways in. It works without JavaScript too. ## plan ```bash expectation plan expectation plan --coverage ``` Prints the acceptance test each story compiles to: what's arranged, each step, which expectations fire and what's checked. `--coverage` adds the generated tests for expectations no story reaches. ## docs ```bash expectation docs ``` Writes the manual (Markdown and the HTML site) to ``, using the results that runs have saved in `/results/`. Without results, every test shows as "not run". - `--approved-only` writes just what the suite marks `Approval: approved` (see [Approved-only manuals](manual.md#approved-only-manuals)); `--results ` reads the saved results from there instead of `/results`. - `--stylesheet ` restyles the HTML site with your CSS file (see [Your own theme](manual.md#your-own-theme)). A missing, empty or malformed file stops the command before anything is written. - `--state-shot-ratio ` sets the shape of the screenshots on the state map's cards, as width over height: `9:19.5` for a phone, `16:9`, or a number like `0.46`. Between 1:5 and 5:1. ## In your own project The CLI is also a library: `expectation_cli::main()` runs the command with the process's arguments. A workspace can ship the command as its own package, so `cargo run -p -- check expectations` builds it at the version the workspace locks: ```rust // tools/expectation/src/main.rs fn main() { expectation_cli::main(); } ``` ## Exit codes | Code | Means | | --- | --- | | 0 | Done, with no errors (there may be warnings). | | 1 | The suite has errors. | | 2 | A usage error, or a directory that can't be read. | --- # Diagnostics Every diagnostic has a code, a place and usually a hint. Errors stop the suite loading. Warnings don't, but most of them mean something won't be tested or documented. ```text error[bad-value]: `boiling` isn't a value of the kettle --> heating.expect:9 | Then the kettle is boiling = help: the kettle is one of: off, starting, heating, dry ``` ## The language | Code | | Means | | --- | --- | --- | | `syntax` | error | A line that isn't part of the language here: a keyword in the wrong place, or a missing name. | | `empty-suite` | error | The directory has no `.expect` files. | | `unknown-step` | error | A step that's neither a declared state nor a declared event. The hint suggests the closest. | | `bad-value` | error | A value the state doesn't have, a number outside its range, or a comparison that doesn't fit. | | `bad-state` | error | A state declaration that can't be read, or one declared twice differently. | | `bad-event` | error | An event with no name. | | `bad-phrase` | error | A phrase that mixes Givens and Whens, or has no steps. | | `bad-override` | error | `Overrides:` names an expectation that doesn't exist. | | `approval-conflict` | error | An expectation is `Approval: approved` inside a feature that is `proposed`. | | `approval-dependency` | error | With `--approved-only`: an approved expectation overrides one that isn't approved. | | `duplicate` | error | Two expectations, terms or other named things with the same name. | | `two-names` | error / warning | A section named twice (error), or the suite named twice (warning). | | `no-then` | error | An expectation with no Then. | | `contradiction` | error | Steps in one condition that can't hold together ("off" and "heating"). | | `misplaced-event` | error | An event in a Given or a Then. Events only happen in a When. | | `misplaced-shift` | error | A relative step ("goes up by 1") anywhere but an expectation's Then. | | `two-events` | error | One action with two events. Put a Then between them, or split the step. | | `empty-story` | error | A story or scenario with no steps. | | `maybe-step` | warning | Description text that looks like a misspelled step ("Gvien the kettle is off"). | | `says-nothing` | warning | A step that's true of every value ("off or heating" of a state with only those). | | `shows-what` | warning | A `Shows:` line that doesn't say which value it's for. | ## Platforms | Code | | Means | | --- | --- | --- | | `unknown-platform` | error | A tag or `On:` naming a platform that isn't declared, or any platform in a suite that declares none. | | `default-platform` | error | A tag for the default platform. Write its lines plainly. | ## Conflicts and the graph | Code | | Means | | --- | --- | --- | | `conflict` | error | Two expectations can both apply and promise contradicting things. The note names where they overlap; the hint suggests which is the exception. | | `override-cycle` | error | Overrides that go round in a circle. | | `unneeded-override` | warning | An override between two expectations that never disagree. | | `unreachable` | warning | No chain of expectations gets the system into this expectation's Given, so nothing can test it. | | `no-way-in` | warning | Nothing leads from a section's parent into the section, so the docs can't say how to get there. | | `follows-no-way-in` | error | A section `Follows:` a setup, the expectations change every state its own setup needs, and still no chain of them gets there. The route it claims is broken. | | `follows-unreached` | warning | Nothing gets to the setup a section `Follows:`: no section's setup has it, and no chain of expectations leads there. | ## Stories | Code | | Means | | --- | --- | --- | | `unclear-step` | error | The step might set off an expectation, and the story doesn't pin down whether. | | `unspecified` | error | An event happens and no expectation says what it does. | | `contradicts` | error | A Then in the story contradicts what the expectations say. | | `clash` | error | Two expectations fire at this step and disagree. Settle the conflict first. | | `unpromised` | warning | A Then in the story that no expectation promises. It's checked anyway. | | `untested` | warning | No story exercises this expectation. The hint is a story that would. | --- # Crates Expectation is a workspace of small crates with no dependencies outside the standard library. Most projects only need `expectation_harness` (and `expectation_story` for the report types). | Crate | Role | | --- | --- | | `expectation_model` | The vocabulary, the condition algebra, expectations and stories as plain data. | | `expectation_language` | The `.expect` format: parsing, platforms, and resolving steps against the vocabulary. | | `expectation_graph` | Conflicts, override precedence, the flow DAG, Mermaid and DOT export. | | `expectation_story` | Stories → acceptance-test plans, coverage, results and verdicts. | | `expectation_docs` | The manual, as an HTML site and Markdown. | | `expectation_theme` | The colours, type and shapes the docs and graphs are drawn in. | | `expectation_service` | External services a run needs: started once, stopped after, "not supported yet" when they can't be had. | | `expectation_harness` | The `Driver` trait, `Bindings`, the runner, and the `Acceptance` pipeline. Re-exports the service crate as `expectation_harness::service`. | | `expectation_remote` | Driving an app through a driver embedded in it: `RemoteDriver`, the Rust client, and the `expectation-run` command. Clients for Swift and JavaScript are in `clients/`. | | `expectation_cli` | The `expectation` command, as a binary and a library. | Each crate depends only on the ones before it: `language`, `graph` and `story` build on `model`; `docs` on those and `theme`; `harness` on everything, plus `service`; and `cli` on everything but `harness`. The [API reference](../api/expectation_harness/index.html) has every public type.