diff options
Diffstat (limited to 'docs/TESTING.md')
| -rw-r--r-- | docs/TESTING.md | 47 |
1 files changed, 47 insertions, 0 deletions
diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..b3b5a7b --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,47 @@ +<!-- + Author: cirakg & claude ai + Summary: Test strategy, how to run every check in Docker, current results, and the real-watch checklist. +--> +# Testing + +The full test plan is in [README section 17](../README.md#17-test-plan). This document tracks what exists, how to run it, and the latest results. Every check runs in Docker; `make ci` runs them all in order and stops at the first failure. + +## Test layers + +| Layer | Command | What it proves | +| --- | --- | --- | +| Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources | +| Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 | +| Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings | +| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.5 % of pixels may differ by more than 24 levels) | +| Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) | +| Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings | +| Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders | + +Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED). + +## Results + +### M0, 2026-10-18 (version 0.1.0) + +`make ci` passed in full: + +- Python tool tests: 44 passed, with warnings treated as errors. +- Monkey C unit tests: the smoke test passed on all four test devices. +- Lint: 0 policy problems; ruff and mypy clean; 72 of 72 products meet the target rule; strict builds without warnings on all four test devices. +- Screenshots: all four devices match their approved images (0.00 % of pixels differ). +- Performance and memory: within budget on all four devices (numbers in [PERFORMANCE.md](PERFORMANCE.md)). +- Package: `otp-wallet-0.1.0.iq` built for all 72 products without warnings. +- Docs: 0 broken links, 0 lint issues, all diagrams render. + +M0 has only the smoke test on the watch side; the real Monkey C suites start with M2 (crypto and OTP test vectors). + +## Fake data only + +Tests never use real secrets or personal data. Fixtures are generated or hand-written and labelled as fake. + +## Real-watch checklist + +Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1. + +- [ ] M1: install the settings spike from the Store beta; record field limits, accepted characters and write-back behaviour in `docs/SETTINGS_SPIKE.md`. |