diff options
Diffstat (limited to 'docs/TESTING.md')
| -rw-r--r-- | docs/TESTING.md | 31 |
1 files changed, 31 insertions, 0 deletions
diff --git a/docs/TESTING.md b/docs/TESTING.md index a38fd76..54a331e 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -26,6 +26,34 @@ Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, butto ## Results +### M4, 2026-10-19 (version 0.4.0) + +`make ci` passed in full. + +- **Monkey C unit tests:** 72 of 72 on each of the four test devices. The 7 new ones cover the code cache (same codes as the OTP module, expiry at the period end, the HOTP counter, invalidation, refusal while locked), code grouping, text helpers and the option defaults. One also opens the UI-test fixture's PIN wrap, which was computed in Python by `tools/gen_fixture_wrap.py`. That cross-checks the wrap format between two implementations. +- **Python tool tests:** 50 passed. They now include the UI-flow driver's parsing and the fixture-wrap generator, checked against RFC 5869 and against the constant in the app. +- **Scripted UI flow** (`make screenshots`, `tools/ui_flow.py`) on seven simulated watches: the Forerunner 255S, Venu Sq 2, Forerunner 70 and Forerunner 965, plus the spot checks fēnix 7, Instinct 3 AMOLED 45 mm and Venu X1. Each run does this: + 1. Opens the app from its glance. + 2. Opens a TOTP entry and a HOTP entry, and gets the next HOTP code. + 3. Opens the locked entry: PIN screen, a wrong PIN, the right PIN, then auto-lock closing it. + 4. Opens the menu and About (watches with buttons). + 5. Enters five wrong PINs until the 30 s wait appears. + + Ten screens per watch (nine on the touch-only Venu Sq 2 and Venu X1) match their approved images, and two runs give identical pixels. +- **Codes on screen:** the TOTP, HOTP and next codes in the screenshots match an independent Python computation for the frozen time 1800000026. GitHub 768 147, next 050 219; Google 2624 9947; Bank (SHA-512) 849 419; HOTP counters 5 and 6: 254 676 and 287 922. + +Screen shapes: of the 72 target products, 69 have round screens and 3 rectangular ones (Venu Sq 2, Venu Sq 2 Music, Venu X1); both kinds are in the flow. None has a semi-round or semi-octagon screen. The code treats semi-round like round and other shapes like rectangles, untested on a real profile. + +What the flow found and what changed because of it: + +- **The watchdog is enforced in normal simulator runs** (unlike unit-test runs). Two things tripped it on the Forerunner 255S profile (120,000 bytecodes per event): + - 200 PBKDF2 iterations per step. The inner loop now XORs 32-bit words, which makes it 2.3× faster, and a step is 100 iterations. + - Two SHA-512 codes in one draw. Screens now compute at most one code per timer event, separately from drawing. +- **Opening the app needs care in the simulator.** It starts in the glance, and START alone doesn't open the app from there. +- **Taps:** on touch screens a tap first arrives as "select", so the keypad and the list didn't see where the finger was. +- **Venu X1:** this watch has no MENU button. The list now opens the menu on a long press. +- **Missing glyph:** the backspace glyph doesn't exist in the watch fonts, so the delete key is drawn. + ### M3, 2026-10-19 (version 0.3.0) `make ci` passed in full. Monkey C unit tests: 65 of 65 on each of the four test devices. The 34 new tests in `test/storage/` and `test/util/` cover: @@ -71,3 +99,6 @@ Tests never use real secrets or personal data. Fixtures are generated or hand-wr Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1. - [ ] M1: install the settings spike from the Store beta and run tests A to H in [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md); record the results there. +- [ ] M4: sideload a debug build (`make build DEVICE=fr70`, then copy `build/debug/fr70.prg` to `GARMIN/APPS` over USB). Open the menu, choose "Debug: benchmark", and send me the numbers. They set the real PBKDF2 iteration count and the SHA-512 cost on your watch. +- [ ] M4: with "Debug: demo entries", check the list, a code, the countdown ring, HOTP "next", and the PIN keypad with your finger. Also check a wrong PIN (vibration) and the 30 s wait after five wrong PINs. +- [ ] M4: check that leaving the app and coming back asks for the PIN again for the locked demo entry. Check that the glance in the glance loop shows the number of entries. |
