1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
|
<!--
Author: cirakg & claude ai
Summary: Test strategy, how to run every check in Docker, current results, and the real-watch checklist.
-->
# Testing
The full test plan is in [README section 17](../README.md#17-test-plan). This document tracks what exists, how to run it, and the latest results. Every check runs in Docker; `make ci` runs them all in order and stops at the first failure.
## Test layers
| Layer | Command | What it proves |
| --- | --- | --- |
| Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources |
| Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 |
| Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings |
| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.01 % of pixels may differ by more than 24 levels, less than one changed character) |
| Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) |
| Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings |
| Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders |
Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED).
## Test vectors
`make vectors` regenerates `test/data/Vectors.mc` with `tools/gen_vectors.py`. Published vectors (FIPS 180-4, RFC 2202, 4226, 4231, 4648, 5869, 6238, 7914, NIST SP 800-38A) are copied from the documents and recomputed with Python's standard library, or with openssl for AES, before they are written; the generator stops on any disagreement. Seeded extra cases cover SHA padding boundaries (55 to 256 bytes), every digit count, odd periods, times around 2038 and 2100, and Steam codes. The Monkey C code under test never generates its own expected values.
## Results
### M3, 2026-10-19 (version 0.3.0)
`make ci` passed in full. Monkey C unit tests: 65 of 65 on each of the four test devices. The 34 new tests in `test/storage/` and `test/util/` cover:
- **Codec:** round trips of every TLV type, and rejection of malformed input (truncation, unknown types, wrong lengths, duplicate tags, invalid UTF-8).
- **Records:** round trips of TOTP, HOTP (64-bit counter), Steam and code entries, with and without encrypted names. Every single-byte change of a record is detected, as are wrong IDs, wrong keys and truncation. Entry validation and wiping are covered, and record sizes are measured.
- **Vault:** persistence across a restart, update, move, remove, the HOTP counter, damaged and swapped records, locked entries with the session locked, the pending queue, the 150-entry cap, and a full store at 67 fill levels with no half-written state.
- **PIN:** create, unlock and wrong PINs; malformed PINs; the delay schedule; lockout across an app restart on the trusted clock; uptime fallback with restart and timer reset; damaged lockout state; erase after 10 failures; PIN change (entries untouched) and reset; a tampered wrap; auto-lock with zeroed keys; wiped PBKDF2 buffers.
- **Migration:** fresh metadata, the v0 fixture to v1 (idempotent), and refusal of newer or damaged metadata without changing it.
- **UTF-8 validation:** overlong forms, surrogates, code points above U+10FFFF, truncation, NUL.
Python tool tests: 44 passed.
### M2, 2026-10-18 (version 0.2.0)
`make ci` passed in full. Monkey C unit tests: 31 of 31 on each of the four test devices, covering 57 hash vectors, 21 HMAC vectors, 5 PBKDF2, 3 HKDF, 2 AES-CBC (encrypt and decrypt), 12 Base32, 100 HOTP/TOTP and 12 Steam vectors, plus tamper detection on every byte of a sealed record, PKCS#7 edge cases, Base32 error positions, period boundaries, the 2038 widening and the job runner. Python tool tests: 44 passed.
Two things the simulator can't prove, kept for the real watch:
- **Watchdog.** The simulator didn't stop a 300,000-iteration loop in a test on the Forerunner 255S profile, whose budget is 120,000 bytecodes. Long work is split into jobs by design (`source/util/Jobs.mc`); the real limits are checked on your watch.
- **Speed.** See [PERFORMANCE.md](PERFORMANCE.md).
### M0, 2026-10-18 (version 0.1.0)
`make ci` passed in full:
- Python tool tests: 44 passed, with warnings treated as errors.
- Monkey C unit tests: the smoke test passed on all four test devices.
- Lint: 0 policy problems; ruff and mypy clean; 72 of 72 products meet the target rule; strict builds without warnings on all four test devices.
- Screenshots: all four devices match their approved images (0.00 % of pixels differ).
- Performance and memory: within budget on all four devices (numbers in [PERFORMANCE.md](PERFORMANCE.md)).
- Package: `otp-wallet-0.1.0.iq` built for all 72 products without warnings.
- Docs: 0 broken links, 0 lint issues, all diagrams render.
M0 has only the smoke test on the watch side; the real Monkey C suites start with M2 (crypto and OTP test vectors).
## Fake data only
Tests never use real secrets or personal data. Fixtures are generated or hand-written and labelled as fake.
## Real-watch checklist
Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1.
- [ ] M1: install the settings spike from the Store beta and run tests A to H in [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md); record the results there.
|
