# Testing The full test plan is in [README section 17](../README.md#17-test-plan). This document tracks what exists, how to run it, and the latest results. Every check runs in Docker; `make ci` runs them all in order and stops at the first failure. ## Test layers | Layer | Command | What it proves | | --- | --- | --- | | Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources | | Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 | | Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings | | Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.01 % of pixels may differ by more than 24 levels, less than one changed character) | | Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) | | Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings | | Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders | Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED). ## Test vectors `make vectors` regenerates `test/data/Vectors.mc` with `tools/gen_vectors.py`. Published vectors (FIPS 180-4, RFC 2202, 4226, 4231, 4648, 5869, 6238, 7914, NIST SP 800-38A) are copied from the documents and recomputed with Python's standard library, or with openssl for AES, before they are written; the generator stops on any disagreement. Seeded extra cases cover SHA padding boundaries (55 to 256 bytes), every digit count, odd periods, times around 2038 and 2100, and Steam codes. The Monkey C code under test never generates its own expected values. ## Results ### M5, 2026-10-20 (version 0.5.0) Monkey C unit tests: 101 per device (29 new in `test/import/`): - **JSON tokenizer:** every token kind; escapes including surrogate pairs; integers versus other numbers; 16 kinds of invalid document; incomplete input (8 cases); the depth limit; long strings (icons, including escaped quotes) skipped unread; long keys refused. - **`otpauth://`:** URIs pasted run together; percent-encoded labels; a missing issuer; an issuer mismatch; `;` separators; `+` as a space; unknown parameters; HOTP with and without a counter; both Steam forms; nine error cases, each with its message. - **Aegis:** - Database versions 1, 2 and 3, with groups listed after the entries. - totp, hotp and steam imported; motp and yandex skipped. - SHA-1, SHA-256 and SHA-512; 6, 7 and 8 digits; periods 20, 30, 50 and 60. - Favourites, icons, notes and Unicode names. - Encrypted exports refused, in both forms. - Bad entries (9 digits, a duplicated `secret` field, invalid Base32, a non-integer digit count), and non-Aegis, newer-version and too-deeply-nested documents. - An incomplete export. - **Importer:** - The report text. - Duplicates of an earlier import, of a locked entry while the watch is locked, and within one import. - Possible replacements; "Require PIN" entries going pending. - Replace mode keeping manual entries. - A two-part JSON import, sealed while it waits, and the expiry of a waiting part. - Unusable text: empty, unknown, Google Authenticator links, encrypted, invalid. - A store that fills up mid-import, keeping only whole entries. - Fingerprints. - 100 entries. - **Fuzz corpus** (`tools/gen_fuzz_corpus.py`, seed 4242, `make corpus`): 50 mutated Aegis documents and 50 mutated URIs. - The tokenizer always finishes. - The Aegis reader always ends in a definite state, with every item either valid or explained. - The full importer never stores an invalid entry: 21 entries were stored from the inputs that survived mutation. Python tool tests: 55. The new ones cover the corpus generator, including that the committed corpus matches it. Scripted UI flow: the UI-test build now puts an Aegis export of 100 entries into the Import form, as the phone would, and the flow starts with the results screen that opens after the import. 11 screens per watch (10 on the touch-only ones). Every watch reports the import's duration and peak memory, and the flow fails above 70 % of the watch-app memory. ### M4, 2026-10-19 (version 0.4.0) `make ci` passed in full. - **Monkey C unit tests:** 72 of 72 on each of the four test devices. The 7 new ones cover the code cache (same codes as the OTP module, expiry at the period end, the HOTP counter, invalidation, refusal while locked), code grouping, text helpers and the option defaults. One also opens the UI-test fixture's PIN wrap, which was computed in Python by `tools/gen_fixture_wrap.py`. That cross-checks the wrap format between two implementations. - **Python tool tests:** 50 passed. They now include the UI-flow driver's parsing and the fixture-wrap generator, checked against RFC 5869 and against the constant in the app. - **Scripted UI flow** (`make screenshots`, `tools/ui_flow.py`) on seven simulated watches: the Forerunner 255S, Venu Sq 2, Forerunner 70 and Forerunner 965, plus the spot checks fēnix 7, Instinct 3 AMOLED 45 mm and Venu X1. Each run does this: 1. Opens the app from its glance. 2. Opens a TOTP entry and a HOTP entry, and gets the next HOTP code. 3. Opens the locked entry: PIN screen, a wrong PIN, the right PIN, then auto-lock closing it. 4. Opens the menu and About (watches with buttons). 5. Enters five wrong PINs until the 30 s wait appears. Ten screens per watch (nine on the touch-only Venu Sq 2 and Venu X1) match their approved images, and two runs give identical pixels. - **Codes on screen:** the TOTP, HOTP and next codes in the screenshots match an independent Python computation for the frozen time 1800000026. GitHub 768 147, next 050 219; Google 2624 9947; Bank (SHA-512) 849 419; HOTP counters 5 and 6: 254 676 and 287 922. Screen shapes: of the 72 target products, 69 have round screens and 3 rectangular ones (Venu Sq 2, Venu Sq 2 Music, Venu X1); both kinds are in the flow. None has a semi-round or semi-octagon screen. The code treats semi-round like round and other shapes like rectangles, untested on a real profile. What the flow found and what changed because of it: - **The watchdog is enforced in normal simulator runs** (unlike unit-test runs). Two things tripped it on the Forerunner 255S profile (120,000 bytecodes per event): - 200 PBKDF2 iterations per step. The inner loop now XORs 32-bit words, which makes it 2.3× faster, and a step is 100 iterations. - Two SHA-512 codes in one draw. Screens now compute at most one code per timer event, separately from drawing. - **Opening the app needs care in the simulator.** It starts in the glance, and START alone doesn't open the app from there. - **Taps:** on touch screens a tap first arrives as "select", so the keypad and the list didn't see where the finger was. - **Venu X1:** this watch has no MENU button. The list now opens the menu on a long press. - **Missing glyph:** the backspace glyph doesn't exist in the watch fonts, so the delete key is drawn. ### M3, 2026-10-19 (version 0.3.0) `make ci` passed in full. Monkey C unit tests: 65 of 65 on each of the four test devices. The 34 new tests in `test/storage/` and `test/util/` cover: - **Codec:** round trips of every TLV type, and rejection of malformed input (truncation, unknown types, wrong lengths, duplicate tags, invalid UTF-8). - **Records:** round trips of TOTP, HOTP (64-bit counter), Steam and code entries, with and without encrypted names. Every single-byte change of a record is detected, as are wrong IDs, wrong keys and truncation. Entry validation and wiping are covered, and record sizes are measured. - **Vault:** persistence across a restart, update, move, remove, the HOTP counter, damaged and swapped records, locked entries with the session locked, the pending queue, the 150-entry cap, and a full store at 67 fill levels with no half-written state. - **PIN:** create, unlock and wrong PINs; malformed PINs; the delay schedule; lockout across an app restart on the trusted clock; uptime fallback with restart and timer reset; damaged lockout state; erase after 10 failures; PIN change (entries untouched) and reset; a tampered wrap; auto-lock with zeroed keys; wiped PBKDF2 buffers. - **Migration:** fresh metadata, the v0 fixture to v1 (idempotent), and refusal of newer or damaged metadata without changing it. - **UTF-8 validation:** overlong forms, surrogates, code points above U+10FFFF, truncation, NUL. Python tool tests: 44 passed. ### M2, 2026-10-18 (version 0.2.0) `make ci` passed in full. Monkey C unit tests: 31 of 31 on each of the four test devices, covering 57 hash vectors, 21 HMAC vectors, 5 PBKDF2, 3 HKDF, 2 AES-CBC (encrypt and decrypt), 12 Base32, 100 HOTP/TOTP and 12 Steam vectors, plus tamper detection on every byte of a sealed record, PKCS#7 edge cases, Base32 error positions, period boundaries, the 2038 widening and the job runner. Python tool tests: 44 passed. Two things the simulator can't prove, kept for the real watch: - **Watchdog.** The simulator didn't stop a 300,000-iteration loop in a test on the Forerunner 255S profile, whose budget is 120,000 bytecodes. Long work is split into jobs by design (`source/util/Jobs.mc`); the real limits are checked on your watch. - **Speed.** See [PERFORMANCE.md](PERFORMANCE.md). ### M0, 2026-10-18 (version 0.1.0) `make ci` passed in full: - Python tool tests: 44 passed, with warnings treated as errors. - Monkey C unit tests: the smoke test passed on all four test devices. - Lint: 0 policy problems; ruff and mypy clean; 72 of 72 products meet the target rule; strict builds without warnings on all four test devices. - Screenshots: all four devices match their approved images (0.00 % of pixels differ). - Performance and memory: within budget on all four devices (numbers in [PERFORMANCE.md](PERFORMANCE.md)). - Package: `otp-wallet-0.1.0.iq` built for all 72 products without warnings. - Docs: 0 broken links, 0 lint issues, all diagrams render. M0 has only the smoke test on the watch side; the real Monkey C suites start with M2 (crypto and OTP test vectors). ## Fake data only Tests never use real secrets or personal data. Fixtures are generated or hand-written and labelled as fake. ## Real-watch checklist Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1. - [ ] M1: install the settings spike from the Store beta and run tests A to H in [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md); record the results there. - [ ] M4: sideload a debug build (`make build DEVICE=fr70`, then copy `build/debug/fr70.prg` to `GARMIN/APPS` over USB). Open the menu, choose "Debug: benchmark", and send me the numbers. They set the real PBKDF2 iteration count and the SHA-512 cost on your watch. - [ ] M4: with "Debug: demo entries", check the list, a code, the countdown ring, HOTP "next", and the PIN keypad with your finger. Also check a wrong PIN (vibration) and the 30 s wait after five wrong PINs. - [ ] M4: check that leaving the app and coming back asks for the PIN again for the locked demo entry. Check that the glance in the glance loop shows the number of entries. - [ ] M5: in Garmin Connect, paste a few `otpauth://` links of test accounts (not your real ones) into the Import fields, turn "Import now" on and save. Check the result on the watch and in the Result field. Also check that the text fields are emptied afterwards: this answers part of the M1 question.