From b70cde45b67e1f6bb65d598d25c1a5184bdab520 Mon Sep 17 00:00:00 2001 From: cirakg Date: Sun, 18 Oct 2026 21:10:37 +0200 Subject: docs: add developer, testing and performance guides; record M0 results - DEVELOPER_GUIDE: workflow, how the tooling works (verified simulator behaviour), conventions and how to add devices. - TESTING and PERFORMANCE: test layers, M0 results and measured numbers. - FEASIBILITY: what M0 verified (watchdog 120k/240k, 10 MB storage in device profiles, unreliable monkeydo exit code, icon sizes). - README status, new make targets and layout; CHANGELOG 0.1.0. - Author/Summary headers on every document, including the original brief. Co-Authored-By: Claude Opus 5.5 --- docs/TESTING.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 docs/TESTING.md (limited to 'docs/TESTING.md') diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..b3b5a7b --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,47 @@ + +# Testing + +The full test plan is in [README section 17](../README.md#17-test-plan). This document tracks what exists, how to run it, and the latest results. Every check runs in Docker; `make ci` runs them all in order and stops at the first failure. + +## Test layers + +| Layer | Command | What it proves | +| --- | --- | --- | +| Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources | +| Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 | +| Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings | +| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.5 % of pixels may differ by more than 24 levels) | +| Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) | +| Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings | +| Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders | + +Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED). + +## Results + +### M0, 2026-10-18 (version 0.1.0) + +`make ci` passed in full: + +- Python tool tests: 44 passed, with warnings treated as errors. +- Monkey C unit tests: the smoke test passed on all four test devices. +- Lint: 0 policy problems; ruff and mypy clean; 72 of 72 products meet the target rule; strict builds without warnings on all four test devices. +- Screenshots: all four devices match their approved images (0.00 % of pixels differ). +- Performance and memory: within budget on all four devices (numbers in [PERFORMANCE.md](PERFORMANCE.md)). +- Package: `otp-wallet-0.1.0.iq` built for all 72 products without warnings. +- Docs: 0 broken links, 0 lint issues, all diagrams render. + +M0 has only the smoke test on the watch side; the real Monkey C suites start with M2 (crypto and OTP test vectors). + +## Fake data only + +Tests never use real secrets or personal data. Fixtures are generated or hand-written and labelled as fake. + +## Real-watch checklist + +Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1. + +- [ ] M1: install the settings spike from the Store beta; record field limits, accepted characters and write-back behaviour in `docs/SETTINGS_SPIKE.md`. -- cgit v1.2.3