diff options
Diffstat (limited to 'docs/TESTING.md')
| -rw-r--r-- | docs/TESTING.md | 29 |
1 files changed, 28 insertions, 1 deletions
diff --git a/docs/TESTING.md b/docs/TESTING.md index d8302af..b77ab5d 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -12,7 +12,7 @@ The full test plan is in [README section 17](../README.md#17-test-plan). This do | --- | --- | --- | | Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources | | Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 | -| Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings | +| Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, the resource XML against the SDK's `resources.xsd` with the settings cross-check (`tools/check_resources.py`), and strict debug and release builds with zero compiler warnings | | Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.01 % of pixels may differ by more than 24 levels, less than one changed character) | | Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) | | Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings | @@ -26,6 +26,25 @@ Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, butto ## Results +### M7, 2026-10-20 (phone settings forms, version 0.7.0) + +Monkey C unit tests: 154 per device (34 new, in `test/settings/`), all passing on the four test devices. + +- **Add OTP:** every type (TOTP, HOTP with its counter, Steam ignoring algorithm, digits and period), and each validation message of README 10.4: Base32 character and position, `=` padding, missing secret, digits, period, algorithm, type, counter, control characters, length in bytes, a missing name, the warning below 80 bits. After a problem only the secret is cleared; after success the whole form, with the switch off. +- **Add code:** a valid payload for each of the 12 symbologies (2D codes come out cached and decodable) and an invalid one for each, with the encoder's message (check digit, length, character and position, capacity). Options and "Fix if wrong" are stored as sent, and codes too large for the screen are added with the README 12.2 warning. +- **Manage entry:** every action on unlocked entries; locked entries' changes queued even while unlocked, then applied by confirmation; "Require PIN" applied at once or queued; unknown, malformed, pending and damaged IDs; a queued change whose entry vanished; changes arriving while the queue is applied; tampered queue items dropped; the 16-item limit. +- **Idempotency:** the same Add OTP, Manage and Import form twice is applied once; any change makes a new request. +- **The whole run:** all forms at once in README order; the About catalog, its 800-character cut and the storage line; "Encrypt names" on and off, with locked entries converted after the PIN; work detected only when needed. +- **The contract with `properties.xml`:** every key the forms read exists and every form default equals its property's default (`settingsDefaultsMatchProperties`). The phone's side, all 13 resource files against the SDK's `resources.xsd` plus the settings, property and string cross-check, runs in `make lint` (`tools/check_resources.py`, 4 Python tests of its own). + +Scripted UI flow (end-to-end): the UI-test build fills in the Add OTP, Add code and Manage entry forms next to the import, as the phone would, so they take the real path: properties, `onSettingsChanged`-style processing after the first frame, the timer-driven job and the results screen. 20 screens per watch (17 on the touch-only ones), on seven watches: + +- `otp_form`: the TOTP added through the form shows 309 848 and next 489 290, the RFC 6238 values for its fake key at the frozen time, computed in Python. +- `code_form`: the Code 128 card added through the form decodes from the screenshot on every watch. +- `queue`, `queue_results` (watches with buttons): the Manage form's edit of the locked Bank entry waits under "Phone changes"; "Confirm all" asks for the PIN and then applies it. + +The flow caught one watchdog trip on the Forerunner 255S that the unit tests can't: hashing the 100-entry import form by copying it into one buffer and zeroing it. The fields now go to the native HMAC one by one. + ### M6d, 2026-10-20 (Aztec; M6 complete, version 0.6.0) Monkey C unit tests: 120 per device. @@ -197,3 +216,11 @@ Things the simulator can't verify, done by you on the Forerunner 70. The checkli - [ ] M6b: open "Member QR" (a demo entry) and scan it with your phone's camera. The first opening encodes it; note roughly how long that takes. The second opening should be instant (cache). - [ ] M6c: scan the "Parcel" (Data Matrix) and "Ticket" (PDF417) demo cards with a phone app that reads them, for example the zxing-based "Barcode Scanner". - [ ] M6d: scan the "Rail pass" (Aztec) demo card with a phone app that reads Aztec. +- [ ] M7, with the Store beta, using a test account (never a real one): + - **Add OTP:** fill in the form with the account's Base32 secret, turn on "Add this entry" and save. The watch shows the results; the code must match the one in your phone's authenticator app. Check the form's Result field and whether the form is emptied on the phone. + - **Add code:** add a test EAN-13 (for example `5901234123457`) and a short QR text; scan both from the watch. + - **Manage entry:** with the IDs from the About group on the phone (or "Entry IDs" on the watch), rename one entry, move it to position 1 and delete it. + - **Locked entries:** set "Require PIN" on an entry, then send "Don't require PIN" for it. The watch must not apply it until you open the menu, choose "Phone changes", "Confirm all" and enter the PIN. + - **Repeats:** send the same Add OTP form twice (turn the switch on again without changing anything). The second time the result is "Already added". + - **Options:** change Theme and Code size; the watch changes as soon as the phone saves. + - **Limits:** does the About catalog (up to 800 characters) show completely on the phone? |
