Introduction

Expectation is acceptance testing where the behaviour itself is the source of truth.

You describe a system as expectations: given some state, when something happens, then the system ends up in some state. Every step is written in a shared vocabulary of states and events, so expectations can be compared with each other mechanically, not just run.

Expectation: Turning the dial up starts the kettle
  Given the kettle is off
  When the dial is above 55 %
  Then the kettle is heating

From a suite of expectations, Expectation:

1. finds conflicts. Two expectations that both apply, and promise incompatible things, are an error. It names the situation where they clash and suggests which one is the exception. You settle it with Overrides:, and overrides must form a DAG.
2. builds a flow DAG from the starting state. It shows what leads to what, warns about behaviour nothing can reach, and gives every expectation a shortest route from the start.
3. compiles user stories into acceptance tests. A story only describes the journey. At each step, the expectations it passes through supply the checks. A story that relies on unspecified behaviour, or contradicts an expectation, is rejected before anything runs. Generated tests reach whatever no story does.
4. runs those tests against your system through a small driver, and takes a screenshot at every step.
5. writes the documentation: a manual where stories are step-by-step tutorials and features are topic pages, with the test evidence folded away under “How this is tested”, plus a full test report.

.expect files ─language─▶ Suite ─graph─▶ conflicts · override DAG · flow DAG
                            │                         │
                            └────────story────────────┴─▶ plans (acceptance tests)
                                                             │
                                          harness + driver ──┴─▶ report + screenshots ─docs─▶ HTML + Markdown

Why

Most acceptance tests are scripts: do this, check that. Nothing connects one script to the next, so two tests can quietly assume opposite things, and nobody notices which behaviour is never tested at all. Gherkin made scripts readable, but its steps are free text bound to code with regular expressions, so a tool can’t tell what they mean.

Expectation makes the steps mean something. Because every step is a fact about a declared state, or a declared event, the suite is a model of the system. That model can be checked before any test runs. A story doesn’t have to restate every outcome, because the expectations already promise them. And the same model, written in the user’s words, is good enough to publish as the user manual.

What it is not

• It doesn’t drive your UI for you. You write a driver (usually a dozen closures) that sets states, makes events happen and reads states back. Anything you can control from Rust can be tested: a library, a game loop, a web app through its API or a browser, a device over a serial line.
• It isn’t a property-based tester. Generated tests follow the shortest routes through the flow graph to cover every expectation; they don’t search for counterexamples.

Where to go next

• Getting started runs a whole suite in five minutes.
• A worked example walks through conflicts, exceptions, stories, coverage and a failing run.
• The language is the reference for .expect files.
• The example manual is what the generated docs look like.

Getting started

This chapter runs a complete suite against a tiny system: a coin-operated turnstile. It’s the turnstile example in the repository.

Add the crates

Expectation isn’t on crates.io yet. Depend on the git repository:

[dev-dependencies]
expectation_harness = { git = "https://github.com/ptescher/expectation" }
expectation_story = { git = "https://github.com/ptescher/expectation" }

For the command-line tool:

cargo install --git https://github.com/ptescher/expectation expectation_cli

That installs a binary called expectation.

Write the suite

A suite is every *.expect file under a directory. Make expectations/turnstile.expect:

Suite: Turnstile

Vocabulary:
  State: the turnstile is locked | unlocked
  State: the coin box has 0..1000 coins
  Event: a visitor inserts a coin
  Event: a visitor pushes the arm

Background:
  Given the turnstile is locked
  And the coin box has 0 coins

Feature: Paying to go through
  Expectation: A coin unlocks it
    Given the turnstile is locked
    When a visitor inserts a coin
    Then the turnstile is unlocked
    And the coin box goes up by 1

  Expectation: Pushing through locks it again
    Given the turnstile is unlocked
    When a visitor pushes the arm
    Then the turnstile is locked

  Expectation: Pushing a locked turnstile does nothing
    Given the turnstile is locked
    When a visitor pushes the arm
    Then the turnstile is locked

Story: A day at the zoo
  As a visitor
  I want to pay once and go through
  So that I can see the animals

  When a visitor inserts a coin
  Then the turnstile is unlocked
  When a visitor pushes the arm
  Then the turnstile is locked

• The vocabulary declares the states the system can be in, and the events that can happen. Every other line must use these words.
• The Background is the state every test starts from.
• Expectations are the rules. goes up by 1 is a relative Then: it checks the change, whatever the count was.
• The story is a journey through them.

Check it

expectation check expectations

warning[untested]: no story exercises «Pushing a locked turnstile does nothing»
  --> turnstile.expect:25
   |   Expectation: Pushing a locked turnstile does nothing
   = help: a story that would: When a visitor pushes the arm

0 errors, 1 warning
Turnstile: 3 expectations in 1 feature, 1 story (1 test); 0 exceptions; flow DAG 3 edges, 2 levels deep (search truncated)

Nothing ran, but the suite was checked: every step uses the vocabulary, no two expectations disagree, and the story only relies on behaviour the expectations specify. The warning says one expectation has no story, and how a story could reach it.

“Search truncated” is normal here: every coin makes a new state, so the search for what leads to what stops at its depth limit instead of counting to 1000. It has already found everything by then.

Try breaking it. Add a fourth expectation:

  Expectation: A hard push forces it
    Given the turnstile is locked
    When a visitor pushes the arm
    Then the turnstile is unlocked

check now fails. Both this and «Pushing a locked turnstile does nothing» apply when a visitor pushes a locked turnstile, and they disagree about the turnstile. The error names that situation and suggests settling it with Overrides: if one is an exception to the other. Here neither is, so take the new one out again.

Bind it to the system

The runner talks to the system through a driver. Bindings builds one from closures, one per vocabulary term:

#![allow(unused)]
fn main() {
use expectation_harness::{Acceptance, Bindings, RunOptions};

#[derive(Default)]
struct Turnstile {
    locked: bool,
    coins: u32,
}

#[test]
fn turnstile_behaves() {
    let mut driver = Bindings::new(Turnstile { locked: true, coins: 0 })
        .observe("the turnstile", |t| if t.locked { "locked" } else { "unlocked" })
        .observe("the coin box", |t| t.coins as f64)
        .arrange("the turnstile", |t, v| t.locked = v.as_str() == Some("locked"))
        .arrange("the coin box", |t, v| t.coins = v.as_f64().unwrap() as u32)
        .on("a visitor inserts a coin", |t| {
            t.coins += 1;
            t.locked = false;
        })
        .on("a visitor pushes the arm", |t| t.locked = true)
        .on_reset(|t| *t = Turnstile { locked: true, coins: 0 });

    Acceptance::load("expectations")
        .unwrap_or_else(|e| panic!("{e}"))
        .with_coverage()
        .assert_passes(&mut driver, &RunOptions::default());
}
}

• observe reads a state back. Named values are strings; numbers are f64.
• arrange puts a state in place before a test starts (the Background, a scenario’s Givens).
• on makes an event happen.
• on_reset runs before every test.

with_coverage() adds a generated test for each expectation no story reaches, so all three are verified. assert_passes panics with every failing check if anything fails.

cargo test

Write the docs

Pass a docs_dir to take screenshots (if the driver has on_screenshot), then write the manual:

#![allow(unused)]
fn main() {
let acceptance = Acceptance::load("expectations")?.with_coverage();
let opts = RunOptions { docs_dir: Some("docs".into()), ..Default::default() };
let report = acceptance.run(&mut driver, &opts);
acceptance.docs(Some(&report)).write("docs")?;
}

docs/README.md is the manual in Markdown, and docs/site/index.html is the same as a website. The story is now a tutorial called “A day at the zoo”.

Next

The worked example is a richer system, with time, exceptions, several stories and screenshots.

A worked example: the kettle

The kettle example is a suite of five files about an electric kettle, run against a toy kettle with time in it. This chapter walks through it: the vocabulary, a conflict and its exception, stories, generated coverage, a failing run, and the manual it writes.

cargo run -p expectation_harness --example kettle

The manual it writes is published here: the kettle manual.

The vocabulary

vocabulary.expect declares every word the suite may use, and where every test starts:

Vocabulary:
  State: the kettle is off | starting | heating | dry
  State: the dial is 0..100 %
  State: the preset is WARM | HOT | BOIL
  State: the child lock is on | off
  Event: the user presses start
  Event: the user presses reset
  Event: the user presses preset
  Event: the user presses lock
  Event: the water runs out

Background:
  Given the kettle is off
  And the dial is 0 %
  And the preset is WARM
  And the child lock is off

The dial is a number, so steps can compare it: above 55 %, between 20 and 40 %. The others have named values. “The water runs out” is an event without a person: the docs will say “wait until the water runs out” instead of telling the reader to do it.

Features and an exception

heating.expect says how the kettle starts and stops:

Feature: Heating
  The kettle starts off. Turn the dial past 55 % to start it, the way you would an old
  stovetop knob, or press start.

  Expectation: Turning the dial up starts the kettle
    Holding the dial above 55 % for a moment starts it.
    Given the kettle is off
    When the dial is above 55 %
    Then the kettle is heating
  ...

child_lock.expect adds a child lock, which keeps the dial from doing anything:

  Expectation: The child lock holds the kettle while the dial turns
    Given the kettle is off
    And the child lock is on
    When the dial is above 55 %
    Then the kettle is off

Both expectations apply when the kettle is off, the lock is on and the dial goes above 55 %, and they say opposite things. expectation check refuses the suite:

error[conflict]: «The child lock holds the kettle while the dial turns» and «Turning the dial up starts the kettle» disagree about the kettle
  --> child_lock.expect:15
   |   Expectation: The child lock holds the kettle while the dial turns
   = «The child lock holds the kettle while the dial turns» says the kettle is off
  --> heating.expect:5
   |   Expectation: Turning the dial up starts the kettle
   = «Turning the dial up starts the kettle» says the kettle is heating
   = note: both apply when the kettle is off and the child lock is on, and the dial is above 55 %
   = help: «The child lock holds the kettle while the dial turns» is the more specific; if it's an exception, add `Overrides: Turning the dial up starts the kettle` to it. Otherwise narrow a Given so they can't both apply

It is an exception, so the example says so:

  Expectation: The child lock holds the kettle while the dial turns
    Overrides: Turning the dial up starts the kettle
    Given the kettle is off
    And the child lock is on
    When the dial is above 55 %
    Then the kettle is off

Now the suite checks, and the docs for «Turning the dial up starts the kettle» carry a note: “Except when the child lock is on: see The child lock holds the kettle while the dial turns.”

Stories

stories.expect has three stories. Here’s one:

Story: Keeping little hands off
  As a parent
  I want the child lock to keep the kettle off
  So that a curious toddler turning the dial doesn't start it

  Scenario: Turn the dial with the lock on, then unlock and boil
    When the user presses lock
    Then the child lock is on
    When the dial is 100 %
    Then the kettle is off
    When the dial is 0 %
    And the user presses lock
    Then the child lock is off
    When the dial is 80 %
    Then the kettle is heating

expectation plan shows the test it compiles to. Every check comes from an expectation that fires at that step, so the story’s own Thens are only a statement of what the reader should see:

Keeping little hands off › Turn the dial with the lock on, then unlock and boil
  arrange  the kettle is off, the dial is 0 %, the preset is WARM and the child lock is off
  1. when   the user presses lock
       fires  «Lock turns the child lock on»
       check  the child lock is on  («Lock turns the child lock on»)
  2. when   the dial is 100 %
       fires  «The child lock holds the kettle while the dial turns»
       check  the kettle is off  («The child lock holds the kettle while the dial turns»)
  3. when   the dial is 0 % and the user presses lock
       fires  «Lock again turns it off»
       check  the child lock is off  («Lock again turns it off»)
  4. when   the dial is 80 %
       fires  «Turning the dial up starts the kettle»
       check  the kettle is heating  («Turning the dial up starts the kettle»)

A story can’t wander off the specification. Without the Overrides: line above, step 2 is an error too, because two expectations fire and disagree. A step where nothing is expected to happen is unspecified, and a Then that contradicts the expectations is contradicts.

Coverage

Four expectations have no story. check warns about each one, with a story that would reach it:

warning[untested]: no story exercises «Running dry stops the kettle»
  --> heating.expect:16
   |   Expectation: Running dry stops the kettle
   = help: a story that would: When the dial is above 55 % / When the water runs out

Acceptance::with_coverage() turns those routes into generated tests, so every expectation is verified even if no story mentions it:

✓ First cup › Turn it up, then turn it off
✓ Keeping little hands off › Turn the dial with the lock on, then unlock and boil
✓ Choosing a preset › Step to HOT and start
✓ Reaching «The child lock ignores the start button»
✓ Reaching «Running dry stops the kettle»
✓ Reaching «HOT to BOIL»
✓ Reaching «BOIL back to WARM»

The driver

world.rs is the toy kettle. Like a real one, it takes time: the dial has to stay above 55 % for 0.6 s before the kettle starts, and starting takes another 0.22 s. The driver binds each term and says how time passes:

#![allow(unused)]
fn main() {
Bindings::new(World::new(lock_leaks))
    .observe("the kettle", |w| w.phase_name())
    .observe("the dial", |w| w.dial)
    .set("the dial", |w, v| w.dial = v.as_f64().unwrap())
    .arrange("the kettle", |w, v| /* put it in that phase */)
    .on("the user presses lock", World::press_lock)
    // ...
    .on_settle(|w| (0..20).for_each(|_| w.tick(0.05)))   // a second after each action
    .on_advance(|w| { w.tick(0.1); Progress::Advanced }) // then 0.1 s per extra look
    .on_screenshot(|w| Screenshot::svg(w.svg()))
}

Settling a full second after each action is what makes “Then the kettle is off” mean it stays off, not that it was off for the instant before it switched on. See Drivers.

A failing run

--bug seeds a defect: the child lock lets the dial through.

cargo run -p expectation_harness --example kettle -- out --bug

✗ Keeping little hands off › Turn the dial with the lock on, then unlock and boil
    at: When the dial is 100 % Then the kettle is off
    expected the kettle is off («The child lock holds the kettle while the dial turns»); observed heating

The failure names the step, the check, the expectation that promised it, and what the kettle actually did. The steps after it are skipped, not failed. In the docs, that expectation is marked failed, and so is the tutorial.

The manual

The run writes a manual. The story became this tutorial, in the reader’s words:

# First cup

## Before you begin

- The kettle is off.
- The dial is 0 %.
- The preset is WARM.
- The child lock is off.

## Steps

1. **Set the dial to 80 %.**

   The kettle is heating.

   <screenshot>

2. **Set the dial to 0 %. Then press reset.**

   The kettle is off.

Each feature is a topic page, and the test evidence is collapsed under “How this is tested”. The vocabulary here has no Say:, Set: or Shows: lines, so the wording is worked out from the steps. With them, step 1 could read “Turn the dial past halfway. The red light comes on.” See Writing specs that read well.

Browse the result: the kettle manual.

One suite, three apps

The kettle demo is the same app built three times, in GPUI, SwiftUI and React Native, and held to one suite. Each app embeds the Expectation driver for its stack (see Embedded drivers), and the same runner drives all three.

Browse the result: the demo manual. Its Platform menu switches between the three apps’ manuals: the same tutorials, with each app’s own screenshots and the instructions worded for it.

One suite

The behaviour is written once. The suite declares the platforms, and says where the wording differs:

Vocabulary:
  Platform: GPUI | SwiftUI | React Native

  State: the kettle is off | starting | heating | dry
    Shows: heating: the display reads HEATING and the light turns red
  State: the dial is 0..100 %
    Set: drag the DIAL slider to {value}

  Event: the user presses start
    Say: click START
    Say[SwiftUI]: tap START
    Say[React Native]: tap START

Everything else (the features, the exception for the child lock, the three stories) is plain, so it’s true on every platform. expectation check checks the suite as each platform reads it.

Three apps

Each app implements the kettle in its own language and binds the vocabulary to it:

The SwiftUI binding, for example:

let driver = ExpectationDriver(platform: "SwiftUI", app: "Kettle")
    .observe("the kettle") { .name(k.phase.name) }
    .observe("the dial") { .number(k.dial) }
    .set("the dial") { k.dial = $0.number ?? 0 }
    .on("the user presses start") { k.start() }
    .on("the water runs out") { k.runOut() }
    .onReset { k.restore() }
driver.start()

Running it

cd demos/kettle
./run.sh gpui
./run.sh swiftui
./run.sh react-native

Each run builds the app, starts expectation-run --platform <name>, launches the app, which connects, and saves the results and screenshots. A runner for one platform turns away apps for the others, so leftover apps from an earlier run can’t interfere. The site build then writes the manual for all three from what the runs saved:

expectation docs demos/kettle/expectations demos/kettle/docs

What it shows

The three apps are independent implementations, written in three languages. Nothing but the suite keeps them the same. The test reports show the same tests passing on each, and a behaviour change starts in the suite, where all three apps will be held to it.

The language

A suite is every *.expect file under a directory. The format is line oriented, like Gherkin. A line’s first word decides what it is, a line starting with # is a comment, and indentation is only for people.

Vocabulary:
  State: the kettle is off | starting | heating | dry
  State: the dial is 0..100 %
  State: the preset is WARM | HOT | BOIL
    # How the docs say to change it:
    Set: press PRESET until the display reads {value}
    # What the reader sees:
    Shows: the display reads {value}
  Event: the user starts the kettle
    # How the docs say to do it:
    Say: press START

  # A setup, in one line:
  Phrase: the kettle is plugged in and full
    Say: fill the kettle and plug it in
    Given the kettle is off
    And the dial is 0 %
    And the preset is WARM

  Phrase: the user starts the kettle with the dial
    Say: turn the dial past 55 % and hold it there
    When the dial is 80 %

Background:
  Given the kettle is plugged in and full

Feature: Heating
  # Free text before the first step is a description:
  The kettle starts off.

  Expectation: Turning the dial up starts the kettle
    Given the kettle is off
    When the dial is above 55 %
    Then the kettle is heating

  Expectation: START starts the kettle
    Given the kettle is off
    When the user starts the kettle
    Then the kettle is heating

Story: Your first cup
  As a tea drinker
  I want to start the kettle by turning the dial
  So that it works like the one I already have

  # The tutorial's introduction:
  Every kettle in the range has the same dial.

  Scenario: Dial up
    When the user starts the kettle with the dial
    Then the kettle is heating

Suites and sections

An optional Suite: <name> line names the suite. Otherwise it’s named after its directory.

Each directory under the suite is a section: a nested set of features that share a setup. Its Section: <name> line (in any of its files) and the text under it name and describe it. A leading 01- in a directory name only sets the order.

A section’s Background: is its setup, written over its parent’s. So everything in app/drawing/ starts where app/ does, with the pen picked on top. Pages about a section start from that setup and don’t re-explain how to get there, the way a manual assumes you’ve read the chapter before.

Each top-level section is a system of its own: stories and the flow graph only use the expectations in their own top-level section.

Vocabulary

A state can use are, has or have instead of is (“the drawing has 0..650 shapes”), and steps about it read the same way. Vocabularies from all files merge into one, so a section can declare the words only it uses.

How the docs say things

Name events for what the user means, not the button they press: “the user starts the kettle”, not “the user presses START”. The button goes in lines under the declaration:

For a number, a Shows: can start with a comparison as steps write it: at least 99 %:.

Without these, the docs make do. “The user clicks the canvas” becomes “Click the canvas”, “the water runs out” becomes “Wait until the water runs out”, and a state becomes “Set the dial to 80 %”.

Phrases

A phrase is a step of your own, for saying more with less.

• A phrase of Givens is a setup (“the kettle is plugged in and full”). Use it in a Background, a Given, or at the start of a scenario.
• A phrase of Whens is a sequence of actions (“the user draws two shapes”, one When per action). An expectation can use one that’s a single action; a story step can use any. It becomes several test steps, but stays one numbered step in the docs.

A phrase’s Say: is how the docs put it. Phrases can use the phrases declared before them.

Steps

A step is either an event or a fact <subject> is [not] <predicate>. The predicate is:

• one or more values: off, off or heating, not dry;
• for numbers, a comparison: 80 %, above 55 %, below 10, at least 20, at most 40, between 20 and 40 %.

Matching ignores case, extra spaces and a leading “the”, and “1 shape” matches a unit of “shapes”. A step outside the vocabulary is an error, with a “did you mean” hint.

An expectation’s Then can also be relative: the drawing goes up by 1, goes down by 10 m, stays the same.

Expectations

Expectation: <name>
  <description>
  Overrides: <another expectation>
  Given …
  When …
  Then …

• Given is the state before. Several are joined with And.
• When is at most one event, plus any states the outside world changes (such as the dial).
• Then is the state after. States the Then doesn’t mention keep their values.
• An expectation with no When is an invariant: it holds whenever its Given does.
• Overrides: <name> makes it an exception to another expectation: where both apply, this one wins. Overrides must form a DAG.
• Docs: hidden keeps it off its topic page. It’s still tested, and it’s in the test report.

Text before the first step is the expectation’s description.

Stories

Story: <name>
  As a <someone>
  I want <something>
  So that <a reason>

  <introduction>

  Scenario: <name>
    Given …
    When …
    Then …

• The narrative is the “As a …”, “I want …”, “So that …” lines.
• Any other text is the introduction, which opens the tutorial in the docs.
• A story has one or more Scenario:s. Steps written straight under the story form a single scenario.
• A scenario may start with Givens, which are arranged on top of the Background.
• After that, each When starts a step, And/But add to it, and its Thens are what the story checks afterwards.

Links and the glossary

Term: <name> and the text under it define a word for the glossary. Any text can link to a term, a section, a feature, an expectation or a story with [[its name]], and mark code with backticks.

Platforms

Platform:, On: and the [platform] tags are covered in Platforms.

Platforms

One suite can describe a system that exists on several platforms: a desktop app and a phone app that do the same job, say. Everything written plainly is true on every platform. Where one differs, the suite says so, and each platform reads the suite as its own.

Vocabulary:
  # The first is the default: plain lines are about it.
  Platform: desktop | iOS

  Event: the user unlocks the door
    Say: press UNLOCK
    Say[iOS]: tap UNLOCK          # how the docs tell an iOS reader to do it
  State: the door is locked | unlocked
    Shows[iOS]: unlocked: the UNLOCKED badge turns green   # replaces the plain Shows lines

  Phrase: the app is open on a new account
    Given the tool is pan
  Phrase[iOS]: the app is open on a new account
    Given the user is signed in   # a start that iOS can reach

Section: Drawing
  On: desktop                     # also under a Feature, Expectation or Story

The rules

• Platform: <a> | <b> | … declares the platforms. The first is the default.
• Say[P]:, Set[P]: and Shows[P]: replace the plain line for platform P.
• Phrase[P]: replaces the plain phrase of the same name for P.
• On: desktop, iOS says that the section (the whole directory), feature, expectation or story is only about those platforms. Use it for what a platform doesn’t have. An exception to an expectation that isn’t on a platform is dropped there.
• Tag only the platforms that differ from the default. Tagging the default is an error (default-platform), and so is a name that isn’t declared (unknown-platform).

Checking, running and docs

expectation check checks the suite as each platform reads it, and every one must check clean.

A driver reads the suite for its platform and runs its tests:

#![allow(unused)]
fn main() {
let acceptance = Acceptance::load_for("expectations", "iOS")?;
}

Its results are its own (results/app@iOS.tsv), apart from those of runners that are the same everywhere (api.tsv). Acceptance::publish(dir, runner, &report) saves a run’s results and merges them with what other runners saved.

expectation docs writes one manual per platform: the default at the root and the others under platforms/<name>/. Every HTML page has a Platform menu that opens the same page in the other manual.

Writing specs that read well

The generated manual is only as good as the words in the specs. A suite that tests well but reads badly is half done: the same lines are the user’s documentation. This is a loop that works.

1. Start from what the user does

Write a handful of high-level expectations in the user’s words, before any detail. Name states for what the user sees (“the kettle is heating”), not for internals (“the relay is closed”). Name events for what the user means (“the user starts the kettle”), not for the control (“the user presses START”). The control goes in a Say:.

2. Let the checker shape the model

expectation check expectations
expectation graph expectations > flow.mmd

• A conflict is a question the spec hasn’t answered. Decide which behaviour wins, and say so with Overrides:, or narrow a Given so they can’t both apply.
• An unreachable expectation means nothing gets the system into its Given. Add the expectation that does, or fix the Background.
• No way in means a section’s setup can’t be reached from its parent’s. The docs need that route to say “Before you begin”.

3. Write stories as tutorials

One goal per story, in the order a new user meets things. The narrative says who and why. The introduction is the tutorial’s first paragraph. A story’s Thens are what the reader should notice, not everything that’s true; the expectations already check the rest.

expectation plan expectations

shows each story as the test it compiles to, and errors when a story leans on behaviour the expectations don’t promise.

4. Give the docs words

Add Say: to events and phrases, Set: to states the reader changes, and Shows: to states the reader sees. Turn long setups into phrases (“the kettle is plugged in and full”), so “Before you begin” is one line, not a list of states. Define jargon with Term:.

5. Generate and read as a newcomer

expectation docs expectations docs

Read docs/README.md and the site as someone who has never seen the system. Better: have someone else do it, or ask an AI agent to audit the pages as a reader and list what’s vague, missing or confusing.

6. Fix the specs, then loop

For each finding, change the spec, not the generated page:

Then check, regenerate and read again, until the docs read well.

Style

Keep reader-facing text short and plain. Simplified Technical English (ASD-STE100) is a good target: one instruction per sentence, active voice, present tense, common words, no more than about 20 words a sentence. The docs generator already writes instructions this way, so specs in the same style blend in.

How expectations are compared

A condition (a Given, a Then, a trigger) stands for a set of states: every combination of values that satisfies it. Everything below comes down to three questions about those sets:

• do two overlap?
• does one imply the other?
• what’s left after an action overwrites some states?

Named values are bit sets and numbers are intervals, so the answers are exact. They are never guesses from sampling.

Outcome conflicts

Two expectations conflict when the same action can set off both, their Givens overlap, and their Thens contradict. In the overlap, the system can’t do both.

The error names the overlap (“both apply when the kettle is off and the child lock is on, and the dial is above 55 %”) and suggests which is the exception: the more specific one.

Invariant conflicts

An expectation with no When is an invariant: it holds whenever its Given does. It conflicts with an expectation that leaves the system where the invariant’s Given holds but its Then can’t, and with a Background that does the same.

Exceptions

An Overrides: between the two, directly or through a chain, settles a conflict. Where both apply, the overriding one’s Then wins.

• Overrides must form a DAG. A cycle is an error (override-cycle).
• An override between two expectations that never disagree gets a warning (unneeded-override).
• Settled conflicts are listed in the docs as the suite’s exceptions, and the overridden expectation’s page links to each one: “Except when the child lock is on: see …”.

Relative Thens

Two different relative Thens (“goes up by 1” and “goes up by 2”) always disagree. A relative one and an absolute one disagree unless the shared situation pins down the one value where they meet.

The flow DAG

Starting from each section’s setup, and from each scenario’s arranged start, a breadth-first search applies every transition that can fire, and sees what state it leaves.

• It only uses expectations from the same top-level section.
• An expectation’s level is the fewest steps before it applies. Its parents are the expectations that take the last step on those shortest routes.
• Levels only grow along edges, so the result is a DAG even though the behaviour loops (start, run dry, reset, start again).
• Each expectation gets a shortest route from the start, which is a ready-made test for it. That’s what generated coverage tests follow.
• An expectation the search never reaches gets an unreachable warning: nothing can test it.

The search has limits on depth and the number of states it visits. A counter that can keep going up (“the coin box goes up by 1”) makes new states forever, so the search stops at its limit and says “search truncated”. By then it has usually found every expectation.

expectation graph prints the DAG as Mermaid (or DOT with --dot), and the manual’s map page draws it, coloured by verdict.

Ways into a section

For each section, a goal-directed search finds the chain of expectations that gets from its parent’s setup to its own: “press PEN”, or “sign in, open a project, press PEN” for something deeper. That chain, without any step it works without, is the “Before you begin” of the section’s page.

If there is no such chain, it’s a no-way-in warning, because then nothing explains how to get there.

From stories to acceptance tests

A story describes a journey. It doesn’t have to say everything that happens along the way, because the expectations already do. Planning walks each scenario through the expectations, and turns it into a test plan: what to arrange, what to do at each step, and what to check after it.

What fires

An expectation fires at a step when the current state implies its Given, and the step’s action implies its trigger.

• In a compound action (“When the dial is 0 % and the user presses reset”), the other changes happen first, then the event.
• When an exception and the expectation it overrides both fire, only the exception’s Then applies.
• After a step, the state is the old state with the Thens of what fired written over it.

What’s checked

Each step’s checks are:

• the Thens of the expectations that fired,
• the Thens of invariants now in force,
• and the story’s own Thens.

The story never has to restate what the expectations promise. The tests check it anyway, so a story that only says “When the dial is 80 %” still checks that the kettle is heating.

What’s rejected

Planning is strict, because a story that relies on unspecified behaviour would test something nobody agreed on.

Where a test starts

A test starts from its section’s setup (the Backgrounds, from the root down), with its scenario’s Givens written over it. The runner arranges that state before the first step, choosing a value inside each constraint: the first allowed name, or the middle of a range.

Coverage

An expectation that no story exercises gets an untested warning, with a route that would reach it:

warning[untested]: no story exercises «Running dry stops the kettle»
   = help: a story that would: When the dial is above 55 % / When the water runs out

Acceptance::with_coverage() (and expectation plan --coverage) adds a generated test for each one, following that route from the expectation’s own section setup. Generated tests are named “Reaching «…»”. They appear in the test report, but not as tutorials.

With coverage, every expectation gets a verdict:

Drivers

The runner talks to the system under test through a driver, in the vocabulary’s terms: set this state, make this event happen, what is that state now. You can implement the Driver trait, or build one from closures with Bindings.

Bindings

Bindings wraps your system (any value, called the world) and one closure per vocabulary term:

#![allow(unused)]
fn main() {
use expectation_harness::{Bindings, Progress, Screenshot};

let mut driver = Bindings::new(Kettle::default())
    .observe("the kettle", |k| k.phase_name())                  // read a state
    .observe("the dial", |k| k.dial)
    .set("the dial", |k, v| k.dial = v.as_f64().unwrap())       // change a state, as an action
    .arrange("the kettle", |k, v| k.force(v.as_str().unwrap()))  // put a state in place before a test
    .on("the user presses reset", Kettle::reset)                 // make an event happen
    .on("the water runs out", Kettle::run_out)
    .on_reset(|k| *k = Kettle::default())                        // before every test
    .on_settle(|k| k.run_for(1.0))                               // after each action
    .on_advance(|k| { k.run_for(0.1); Progress::Advanced })      // between looks
    .on_screenshot(|k| Screenshot::svg(k.svg()));                // after each step
}

Terms are matched the way steps are: ignoring case, extra spaces and a leading “the”.

The values passed to set and arrange are Values: v.as_str() for a named value, v.as_f64() for a number.

Time: settling and polling

Real systems take time to react. For each step the runner:

1. acts: sets the states and makes the event happen;
2. lets the system settle;
3. observes every check. If they don’t all hold, it advances and observes again, up to RunOptions::patience times (100 by default). The step passes as soon as every check holds at once.

Polling is what lets “Then the kettle is heating” wait for the kettle to warm up. Settling is what makes “Then the kettle is off” mean that it stays off. Without it, the check could pass in the instant before the kettle switches on. Make the settle long enough for whatever should happen to happen, and for whatever shouldn’t to show.

A system with no time in it needs neither: the default advance returns Progress::Static, and the runner stops looking after the first observation.

Not supported yet

Before a test starts, the runner checks that the driver can do everything the plan needs: arrange each starting state, set each changed state, perform each event, observe each check. If it can’t, the test is not supported yet:

The driver can't arrange the child lock, arrange the dial, arrange the kettle and 6 more things yet.

That isn’t a failure. The system, or its driver, just doesn’t do it yet. The test’s steps get a placeholder instead of a screenshot, and the expectations only it would check are “not supported yet” in the docs. This lets a suite describe a whole product while its drivers catch up, and keeps the gap visible.

A driver that can arrange some values of a state and not others overrides Driver::can_arrange.

The Driver trait

For full control, implement Driver yourself. Only set, perform and observe are required:

#![allow(unused)]
fn main() {
pub trait Driver {
    fn needs(&self) -> Vec<String> { vec![] }
    fn fresh(&self) -> Vec<String> { vec![] }
    fn reset(&mut self) -> Result<(), String> { Ok(()) }
    fn arrange(&mut self, var: &Variable, value: &Value) -> Result<(), String> { self.set(var, value) }
    fn set(&mut self, var: &Variable, value: &Value) -> Result<(), String>;
    fn perform(&mut self, event: &Event) -> Result<(), String>;
    fn observe(&mut self, var: &Variable) -> Result<Value, String>;
    fn can_arrange(&self, var: &Variable, value: &Value) -> bool { true }
    fn settle(&mut self) {}
    fn advance(&mut self) -> Progress { Progress::Static }
    fn screenshot(&mut self) -> Result<Option<Screenshot>, String> { Ok(None) }
    fn supports(&self, need: Need<'_>) -> bool { true }
}
}

Variable::key and Event’s key are the normalised names, for matching. Calls may block, so a driver can talk to a browser, a device or another thread. Any Err fails the step with that message.

Running

#![allow(unused)]
fn main() {
let acceptance = Acceptance::load("expectations")?.with_coverage();

// Run, and panic with every failure if anything failed:
acceptance.assert_passes(&mut driver, &RunOptions::default());

// Or keep the report:
let opts = RunOptions { docs_dir: Some("docs".into()), ..Default::default() };
let report = acceptance.run(&mut driver, &opts);
println!("{}", report.failures(&acceptance.suite, &acceptance.plans));
acceptance.docs(Some(&report)).write("docs")?;
}

With a docs_dir, screenshots go to <docs_dir>/screenshots/<test>/<step>.<ext>.

Several runners

Acceptance::publish(dir, runner, &report) saves a run’s results as dir/results/<runner>.tsv, merges them with what other runners saved there, and writes the docs from the merged report. An API runner and a UI runner can each run their own part of one suite, and the manual shows both. expectation docs rebuilds the docs from saved results without running anything.

Results are keyed by each test’s name and a fingerprint of its plan, so results saved for an older version of a test are ignored, not misapplied.

Embedded drivers

Some systems can’t be driven from inside a Rust test: a phone app in a simulator, a desktop app in its own process, a React Native bundle. For those, the driver goes inside the app, and the runner talks to it over a small protocol.

 runner (Rust)                                   app (any stack)
┌────────────────────────┐   POST /hello        ┌─────────────────────────────┐
│ Acceptance + RemoteDriver│ ◀──────────────────── │ embedded Expectation driver │
│ listens on :4477       │   GET /next (long poll) │  observe / set / arrange /  │
│                        │ ────────────────────▶ │  on(event) / reset /        │
│                        │   POST /reply         │  screenshot                 │
└────────────────────────┘ ◀──────────────────── └─────────────────────────────┘

The app connects out to the runner, so it never hosts a server: that suits phones, sandboxes and simulators. The embedded driver says hello, long-polls for commands, runs each one on the app’s own thread, and posts the answer. When no runner is listening it waits quietly and retries, so it can stay in development builds.

In the app

Each client binds the vocabulary to the app the same way Bindings does in Rust: one closure per term.

Swift (iOS, macOS, SwiftUI, UIKit)

// Package: https://github.com/ptescher/expectation, path clients/swift
import ExpectationDriver

let driver = ExpectationDriver(platform: "SwiftUI", app: "Kettle")
    .observe("the kettle") { .name(kettle.phase.name) }
    .observe("the dial") { .number(kettle.dial) }
    .set("the dial") { kettle.dial = $0.number ?? 0 }
    .arrange("the kettle") { kettle.force($0.string ?? "off") }
    .on("the user presses lock") { kettle.pressLock() }
    .onReset { kettle.restore() }
driver.start()   // connects only when EXPECTATION_URL is set

Handlers run on the main actor. On UIKit platforms, screenshots of the key window are taken for you. The app needs NSAllowsLocalNetworking under NSAppTransportSecurity in its Info.plist to reach the runner over plain HTTP. In the simulator, pass the runner’s address with SIMCTL_CHILD_EXPECTATION_URL=http://127.0.0.1:4477 xcrun simctl launch ….

JavaScript (React Native, web, Node)

// npm: clients/js in the repository (expectation-driver)
import { ExpectationDriver } from 'expectation-driver';

const driver = new ExpectationDriver({ platform: 'React Native', app: 'Kettle' })
  .observe('the kettle', () => kettle.phase)
  .set('the dial', v => kettle.setDial(v))
  .on('the user presses lock', () => kettle.pressLock())
  .onReset(() => kettle.restore());
if (__DEV__) driver.start('http://127.0.0.1:4477');

Handlers may be async. There’s no built-in screenshot in JavaScript: pass onScreenshot (returning a base64 PNG), or let the runner take them from outside.

Rust (GPUI, egui, Bevy, anything with a main loop)

#![allow(unused)]
fn main() {
use expectation_remote::client::{Client, Handlers};

let handlers = Handlers::new()
    .observe("the kettle", |v: &KettleView| v.kettle.phase_name())
    .set("the dial", |v, x| v.kettle.dial = x.as_f64().unwrap_or(0.0))
    .on("the user presses lock", |v| v.kettle.press_lock());
let client = Client::connect_env(handlers.hello("GPUI", "Kettle"));

// On the UI thread, every frame or tick:
while let Some(request) = client.try_recv() {
    handlers.handle(&mut view, request);
}
}

The client runs on its own thread; try_recv hands each command to the thread that owns the app.

In the runner

#![allow(unused)]
fn main() {
use expectation_remote::RemoteDriver;

let mut driver = RemoteDriver::listen("127.0.0.1:4477")?
    .settle_for(Duration::from_millis(1000))
    .screenshot_with(|| simulator_screenshot());   // optional: from outside the app
let app = driver.wait_for_app(Duration::from_secs(60))?;
let acceptance = Acceptance::load_for("expectations", &app.platform)?.with_coverage();
let report = acceptance.run(&mut driver, &RunOptions::default());
}

RemoteDriver knows what the app can do from its hello, so anything the app didn’t bind is “not supported yet”, not a failure. Time is real: it sleeps to settle after each action and between looks. If no app connects, every test is “not supported yet”.

Or skip writing a runner and use expectation-run:

cargo install --git https://github.com/ptescher/expectation expectation_remote
expectation-run expectations --platform SwiftUI --docs docs --runner app

It listens, waits for the app, runs the suite for that platform, prints the results, and with --docs saves them and writes the manual. --screenshot-cmd 'xcrun simctl io booted screenshot "$1"' takes screenshots from outside, for apps that don’t take their own.

The protocol

JSON over HTTP/1.1. Anything that can make an HTTP request can be a client.

A command is one of:

{"id": 1, "op": "reset"}
{"id": 2, "op": "arrange", "subject": "the kettle", "value": "off"}
{"id": 3, "op": "set", "subject": "the dial", "value": 80}
{"id": 4, "op": "perform", "event": "the user presses lock"}
{"id": 5, "op": "observe", "subject": "the kettle"}
{"id": 6, "op": "screenshot"}

Terms are the app’s own spelling from its hello. A named value is a JSON string and a number is a JSON number. png is base64.

Events: real taps or the app’s actions?

An embedded driver makes an event happen by calling the same action the button calls, not by synthesising a tap. That’s fast, works the same on every stack, and tests the behaviour. It doesn’t test that the button is wired to the action, or that it’s visible. If that matters, pair the suite with a UI-automation smoke test (XCUITest, Espresso, Detox), or bind the event to a handler that drives the UI.

External services

Tests sometimes need something that isn’t part of the system under test: a database, a server, a container. The harness treats that as a service, so no test, driver or script has to start it.

#![allow(unused)]
fn main() {
use expectation_harness::service::{Command, Docker, Ready, Services};

let mut services = Services::new()
    .with(Command::new("server", "./target/debug/server").ready(Ready::Tcp(8080)))
    .with(Docker::new("database", "postgres:16").port(5432, 55432)   // a database is a container
        .env("POSTGRES_PASSWORD", "test").ready(Ready::Tcp(55432)));

let mut driver = Bindings::new(world).needs(&["server", "database"]).fresh(&["database"]);
let report = acceptance.run_with(&mut driver, &RunOptions::default(), &mut services);
}

Needs

A driver says what it needs (Driver::needs, or Bindings::needs). Before each test, the runner makes sure each one is up: one that is up stays up, and one that died is started again. They all stop when services is dropped, the last started first.

A test the driver can’t run anyway starts nothing.

Fresh

What a test leaves behind (rows the last test wrote, a user it signed in) is cleared by naming the service in Driver::fresh. It’s started afresh before each test that runs.

Not supported yet, not failed

A service that can’t run on this machine (no Docker, no build) makes the tests that need it “not supported yet”, and the reason says how to get it:

database: install PostgreSQL with `brew install postgresql`

A test suite on a laptop without Docker still runs everything else, and the gap is visible in the docs.

Providers

Ready::Tcp(port) waits until the port answers. A port that something else already holds is an error, not a ready service: whatever answers there is probably left over from an earlier run.

Anything else implements the Service trait: a name, whether it can run here (unavailable), and start, which returns a Running that stops when it’s dropped.

The generated manual

Docs::write produces a manual, both as an HTML site in site/ and as Markdown that GitHub renders. It reads like a project’s user documentation, not a test report. The test evidence is all there, folded away.

#![allow(unused)]
fn main() {
acceptance.docs(Some(&report)).write("docs")?;
}

or, from results that runs have saved in docs/results/:

expectation docs expectations docs

See the kettle manual for a whole one.

Pages

• Tutorials, one per story scenario: the story’s introduction, Before you begin (the setup, in the specs’ own words), then numbered steps. Each step says what to do (from the Say:s and Set:s), what you’ll see (the story’s Thens, or what the expectations promise) and shows the screenshot the test took there.
• Topic pages, one per feature: its description, then each expectation as a sentence (“If the kettle is off, press START. The light comes on and the water starts to heat.”), its exceptions as notes, a screenshot of it in action, and the tutorials that show it.
• Section pages: the description, Before you begin (the setup, or the steps that get there from the parent section), the topics in it, and its tutorials as cards.
• Reference: the glossary of Term:s, the map of the flow DAG, the theme, and the test report: every test and its result, and a page per test with each step, check and screenshot.
• Test vocabulary: each state, event and phrase, and how the docs word it.

On every page, the test details of each expectation (its Gherkin, what it follows and leads to, its verdict and the tests that cover it) are collapsed under “How this is tested”. Generated coverage tests appear only in the test report. A step without a screenshot shows a “Not supported yet” or “No screenshot” placeholder.

Markdown

The Markdown has the same pages:

README.md        the overview and contents
tutorials/       one per scenario
features/        one per feature
glossary.md
tests.md         the test report, with a Mermaid map of the flow DAG
tests/           one per test
vocabulary.md
screenshots/
site/            the HTML version

Pages read in order: each section, then its tutorials, its topics and its subsections. Each page’s footer names the spec file to edit to change it.

The outline

By default the manual follows the sections. A suite can have an outline.txt beside its sections to shape it like a book:

Part: Start here
  Page: installing          manual/installing.md
  Page: the-installer [desktop]
Part: Brewing
  Section: The kettle

• Parts group the top-level sections. They’re the navigation’s labels and the README’s contents.
• The order is the order the reader meets them in, and so the previous and next links.
• Pages are hand-written Markdown files in manual/ (install, reference, troubleshooting), placed among the generated ones. A page’s title is its first # line. They can use [[links]] into the generated pages. [desktop] keeps a page out of another platform’s manual.

The sections themselves stay where they are, because their setups inherit from their parents. A top-level section that no part names goes into a last part, “More”, with a warning.

The look

The site is plain HTML and CSS, with no scripts. It follows the reader’s light or dark preference. The colours, type and corners come from the expectation_theme crate, and the theme page in each manual shows them.

The expectation command

expectation checks a suite, shows its graph and plans, and writes its docs, without running anything. Running needs the system under test, so that happens through expectation_harness in your own tests or program, which saves the results for the docs.

cargo install --git https://github.com/ptescher/expectation expectation_cli

Or, from a checkout: cargo run -p expectation_cli -- <command>.

check

expectation check <dir>

Loads the suite, checks the vocabulary, conflicts, overrides and stories, and prints every diagnostic with where it is and what to do. A suite with platforms is checked as each platform reads it. Ends with a one-line summary:

Kettle: 11 expectations in 3 features, 3 stories (3 tests); 2 exceptions; flow DAG 12 edges, 3 levels deep

Exits 1 if there are errors, so it can run in CI.

graph

expectation graph <dir>          # Mermaid
expectation graph <dir> --dot    # Graphviz DOT

Prints the flow DAG: each feature as a group, each expectation as a node (invariants as hexagons), arrows from the start and between expectations, dotted arrows for overrides, and red ones for conflicts nobody has settled.

plan

expectation plan <dir>
expectation plan <dir> --coverage

Prints the acceptance test each story compiles to: what’s arranged, each step, which expectations fire and what’s checked. --coverage adds the generated tests for expectations no story reaches.

docs

expectation docs <dir> <out-dir>

Writes the manual (Markdown and the HTML site) to <out-dir>, using the results that runs have saved in <out-dir>/results/. Without results, every test shows as “not run”.

In your own project

The CLI is also a library: expectation_cli::main() runs the command with the process’s arguments. A workspace can ship the command as its own package, so cargo run -p <yours> -- check expectations builds it at the version the workspace locks:

// tools/expectation/src/main.rs
fn main() {
    expectation_cli::main();
}

Exit codes

Diagnostics

Every diagnostic has a code, a place and usually a hint. Errors stop the suite loading. Warnings don’t, but most of them mean something won’t be tested or documented.

error[bad-value]: `boiling` isn't a value of the kettle
  --> heating.expect:9
   |     Then the kettle is boiling
   = help: the kettle is one of: off, starting, heating, dry

The language

Platforms

Conflicts and the graph

Stories

Crates

Expectation is a workspace of small crates with no dependencies outside the standard library. Most projects only need expectation_harness (and expectation_story for the report types).

Each crate depends only on the ones before it: language, graph and story build on model; docs on those and theme; harness on everything, plus service; and cli on everything but harness.

The API reference has every public type.