diff options
Diffstat (limited to 'docs/TESTING.md')
| -rw-r--r-- | docs/TESTING.md | 17 |
1 files changed, 15 insertions, 2 deletions
diff --git a/docs/TESTING.md b/docs/TESTING.md index b3b5a7b..98d984c 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -13,15 +13,28 @@ The full test plan is in [README section 17](../README.md#17-test-plan). This do | Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources | | Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 | | Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings | -| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.5 % of pixels may differ by more than 24 levels) | +| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.01 % of pixels may differ by more than 24 levels, less than one changed character) | | Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) | | Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings | | Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders | Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED). +## Test vectors + +`make vectors` regenerates `test/data/Vectors.mc` with `tools/gen_vectors.py`. Published vectors (FIPS 180-4, RFC 2202, 4226, 4231, 4648, 5869, 6238, 7914, NIST SP 800-38A) are copied from the documents and recomputed with Python's standard library, or with openssl for AES, before they are written; the generator stops on any disagreement. Seeded extra cases cover SHA padding boundaries (55 to 256 bytes), every digit count, odd periods, times around 2038 and 2100, and Steam codes. The Monkey C code under test never generates its own expected values. + ## Results +### M2, 2026-10-18 (version 0.2.0) + +`make ci` passed in full. Monkey C unit tests: 31 of 31 on each of the four test devices, covering 57 hash vectors, 21 HMAC vectors, 5 PBKDF2, 3 HKDF, 2 AES-CBC (encrypt and decrypt), 12 Base32, 100 HOTP/TOTP and 12 Steam vectors, plus tamper detection on every byte of a sealed record, PKCS#7 edge cases, Base32 error positions, period boundaries, the 2038 widening and the job runner. Python tool tests: 44 passed. + +Two things the simulator can't prove, kept for the real watch: + +- **Watchdog.** The simulator didn't stop a 300,000-iteration loop in a test on the Forerunner 255S profile, whose budget is 120,000 bytecodes. Long work is split into jobs by design (`source/util/Jobs.mc`); the real limits are checked on your watch. +- **Speed.** See [PERFORMANCE.md](PERFORMANCE.md). + ### M0, 2026-10-18 (version 0.1.0) `make ci` passed in full: @@ -44,4 +57,4 @@ Tests never use real secrets or personal data. Fixtures are generated or hand-wr Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1. -- [ ] M1: install the settings spike from the Store beta; record field limits, accepted characters and write-back behaviour in `docs/SETTINGS_SPIKE.md`. +- [ ] M1: install the settings spike from the Store beta and run tests A to H in [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md); record the results there. |