1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
<!--
Author: cirakg & claude ai
Summary: Test strategy, how to run every check in Docker, current results, and the real-watch checklist.
-->
# Testing
The full test plan is in [README section 17](../README.md#17-test-plan). This document tracks what exists, how to run it, and the latest results. Every check runs in Docker; `make ci` runs them all in order and stops at the first failure.
## Test layers
| Layer | Command | What it proves |
| --- | --- | --- |
| Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources |
| Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 |
| Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings |
| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.5 % of pixels may differ by more than 24 levels) |
| Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) |
| Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings |
| Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders |
Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED).
## Results
### M0, 2026-10-18 (version 0.1.0)
`make ci` passed in full:
- Python tool tests: 44 passed, with warnings treated as errors.
- Monkey C unit tests: the smoke test passed on all four test devices.
- Lint: 0 policy problems; ruff and mypy clean; 72 of 72 products meet the target rule; strict builds without warnings on all four test devices.
- Screenshots: all four devices match their approved images (0.00 % of pixels differ).
- Performance and memory: within budget on all four devices (numbers in [PERFORMANCE.md](PERFORMANCE.md)).
- Package: `otp-wallet-0.1.0.iq` built for all 72 products without warnings.
- Docs: 0 broken links, 0 lint issues, all diagrams render.
M0 has only the smoke test on the watch side; the real Monkey C suites start with M2 (crypto and OTP test vectors).
## Fake data only
Tests never use real secrets or personal data. Fixtures are generated or hand-written and labelled as fake.
## Real-watch checklist
Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1.
- [ ] M1: install the settings spike from the Store beta; record field limits, accepted characters and write-back behaviour in `docs/SETTINGS_SPIKE.md`.
|
