diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/DEVELOPER_GUIDE.md | 10 | ||||
| -rw-r--r-- | docs/FEASIBILITY.md | 7 | ||||
| -rw-r--r-- | docs/PERFORMANCE.md | 18 | ||||
| -rw-r--r-- | docs/TESTING.md | 17 |
4 files changed, 50 insertions, 2 deletions
diff --git a/docs/DEVELOPER_GUIDE.md b/docs/DEVELOPER_GUIDE.md index e6f53a9..eacea84 100644 --- a/docs/DEVELOPER_GUIDE.md +++ b/docs/DEVELOPER_GUIDE.md @@ -25,6 +25,8 @@ Everything runs in Docker through the `Makefile`; the host needs only Docker and | Accept changed screens after reviewing build/screenshots/ | `make screenshots-approve` | | Start-up time and memory budgets | `make perf`, `make memtest` | | Signed package for the Store beta | `make package` | +| Regenerate the crypto/OTP test vectors | `make vectors` | +| Build the M1 settings spike | `make spike` (instructions: [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md)) | | Everything, in order | `make ci` | | A shell in the container | `make shell` | @@ -66,6 +68,14 @@ The simulator is a GTK program. The scripts start it on a virtual display (Xvfb - `monkeydo` exits with 1 even when every unit test passed, so `tools/ciq_test_report.py` judges the console output instead. - For a normal (non-test) run, `monkeydo` blocks while the app runs and streams the app's `System.println` output. Quotes in that output arrive escaped, which is why the diagnostics use plain `key=value` pairs. +### Monkey C facts that matter here (verified with SDK 9.2.0) + +- `Long` is 64-bit and wraps on overflow; `>>` is an arithmetic shift, so logical shifts and rotations mask off the copied sign bits (see `source/crypto/Sha512.mc`). +- Assigning a value above 255 to a ByteArray element throws; mask with `& 0xFF`. +- Native HMAC supports only SHA-256; HMAC-SHA1 and HMAC-SHA512 go through `Hmac.generic`. +- Function calls are expensive in the VM; hot loops inline small helpers (SHA-512 went 37 % faster that way). +- The simulator doesn't enforce the watchdog in test runs. Anything long runs as a `Jobs.Job` with a bounded step. + ### Screenshots `tools/sim_screenshot.py` doesn't assume where the watch is drawn in the simulator window. It searches for the device drawing (named in the device's `simulator.json`) in the window capture, comparing only the bezel, because the app draws inside the display area. The match is near-exact (mean difference below 0.3 out of 255); the drawing sits at offset (1, 26) because of the window border and the menu bar. The display rectangle from `simulator.json` is then cut out, round screens get their corners blacked out, and the result is compared with `test/golden/<device>.png`. diff --git a/docs/FEASIBILITY.md b/docs/FEASIBILITY.md index cb3431a..8665139 100644 --- a/docs/FEASIBILITY.md +++ b/docs/FEASIBILITY.md @@ -254,6 +254,13 @@ With SDK 9.2.0 and the device files of all 72 products installed: - **Launcher icons:** the 72 products use eight icon sizes (38 to 70 px). Each gets an exact-size icon, because the compiler warns when it scales one, and warnings fail the build. - **SDK login (11):** the CLI's browser login can't reach the container (it waits on a random localhost port), so `make sdk` uses email and password; that worked for your account. +## Verified during M2 (2026-10-18) + +- **Native HMAC-SHA1 doesn't exist (2): confirmed.** The constructor rejects HASH_SHA1 ("Invalid :algorithm option value"). HMAC-SHA1 is built on the native SHA-1 hash per RFC 2104 and passes every RFC 2202 vector. +- **AES-256-CBC works (2).** Records use AES-256; the NIST vectors pass on all four test devices. +- **64-bit arithmetic:** `Long` overflow wraps like unsigned arithmetic, `>>` copies the sign bit, and hex literals above 2^63 parse correctly, which is what the SHA-512 implementation relies on. ByteArrays throw on values above 255 rather than wrapping. +- **Watchdog in the simulator:** not enforced in test runs (a 300,000-iteration loop passed on a 120,000-bytecode profile), so the job design is the protection and the real limits are checked on the watch. + ## Decisions Answered on 2026-10-18: diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 2e668c2..daab3c2 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -25,6 +25,24 @@ The targets are in [README section 18](../README.md#18-performance-budgets). Thi ## Results +### M2, 2026-10-18 (version 0.2.0, crypto and OTP primitives) + +`test/BenchTest.mc`, simulator in Docker, average per call (identical within noise on all four test devices): + +| Operation | Time | Notes | +| --- | --- | --- | +| TOTP, SHA-1 | 0.35 ms | HMAC built on the native SHA-1 | +| TOTP, SHA-256 | 0.05 ms | Native HMAC-SHA256 | +| TOTP, SHA-512 | 17.5 ms | SHA-512 in Monkey C; was 27.9 ms before inlining the rotations | +| PBKDF2, 1000 iterations | 52 ms | Was 89 ms before removing the per-iteration wipe; native HMAC alone is about 3 ms | +| Seal + open a record | 0.25 ms | AES-256-CBC + HMAC-SHA256, two HKDF key derivations cached | +| Start-up self-test | 6.7 ms | All known-answer tests | +| PBKDF2 calibration for 1 s | about 12,000 iterations | In the simulator; the watch will choose far fewer | + +`make perf` and `make memtest` with the larger app: in-app start-up 1 to 7 ms; launch to first frame rose from 6.5 s to 10.5 s, because `monkeydo` now sends a larger debug program (crypto code and debug information), still under its 15 s guard. Memory after the first frame: 2.3 % (768 KB watches) to 3.4 % (Forerunner 255S). The signed package for 72 products is 1.5 MB. + +What this means for the watch, to be confirmed with the on-watch benchmark in M4: the simulator runs on a desktop CPU, so absolute times will be many times higher on the Forerunner 70. The ratios hold: SHA-512 entries cost about 50 times as much as SHA-1 entries, and Monkey C loop overhead, not the native HMAC, dominates PBKDF2. If the watch manages fewer than 2,000 PBKDF2 iterations per second, unlocking takes longer than the 1 s target (the lower bound is kept for security); M8 can move more of the XOR work into native calls. + ### M0, 2026-10-18 (version 0.1.0, placeholder app) Simulator in Docker on your PC (12 cores), debug builds: diff --git a/docs/TESTING.md b/docs/TESTING.md index b3b5a7b..98d984c 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -13,15 +13,28 @@ The full test plan is in [README section 17](../README.md#17-test-plan). This do | Python tool tests | `make test` (first part) | The tools that judge everything else are correct: test-log parser, screenshot checker, budget checker, product rule, link checker, linter, icon generator. Includes Hypothesis fuzzing of the log parser and integration checks against the real docs and sources | | Monkey C unit tests | `make test` (second part) | Garmin's Run No Evil tests, run in the headless simulator on each test device; judged from the console output, because `monkeydo` always exits with 1 | | Static checks | `make lint` | Project policy (headers, no logging, no network, no secret-like literals), ruff and strict mypy on the tools, the product rule for all 72 products, and strict debug and release builds with zero compiler warnings | -| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.5 % of pixels may differ by more than 24 levels) | +| Screenshots | `make screenshots` | Each test device's screen matches its approved image in `test/golden/` (at most 0.01 % of pixels may differ by more than 24 levels, less than one changed character) | | Performance and memory | `make perf`, `make memtest` | First-frame time and memory use within `tools/budgets.json` ([PERFORMANCE.md](PERFORMANCE.md)) | | Packaging | `make package` | The signed `.iq` builds for all 72 products with zero warnings | | Docs | `make docs` | No broken links or anchors, no Markdown lint issues, every Mermaid diagram renders | Test devices (`TEST_DEVICES`): Forerunner 255S (weakest: 218 px round MIP, buttons only, 512 KB), Venu Sq 2 (rectangular AMOLED), Forerunner 70 (your watch) and Forerunner 965 (454 px round AMOLED). +## Test vectors + +`make vectors` regenerates `test/data/Vectors.mc` with `tools/gen_vectors.py`. Published vectors (FIPS 180-4, RFC 2202, 4226, 4231, 4648, 5869, 6238, 7914, NIST SP 800-38A) are copied from the documents and recomputed with Python's standard library, or with openssl for AES, before they are written; the generator stops on any disagreement. Seeded extra cases cover SHA padding boundaries (55 to 256 bytes), every digit count, odd periods, times around 2038 and 2100, and Steam codes. The Monkey C code under test never generates its own expected values. + ## Results +### M2, 2026-10-18 (version 0.2.0) + +`make ci` passed in full. Monkey C unit tests: 31 of 31 on each of the four test devices, covering 57 hash vectors, 21 HMAC vectors, 5 PBKDF2, 3 HKDF, 2 AES-CBC (encrypt and decrypt), 12 Base32, 100 HOTP/TOTP and 12 Steam vectors, plus tamper detection on every byte of a sealed record, PKCS#7 edge cases, Base32 error positions, period boundaries, the 2038 widening and the job runner. Python tool tests: 44 passed. + +Two things the simulator can't prove, kept for the real watch: + +- **Watchdog.** The simulator didn't stop a 300,000-iteration loop in a test on the Forerunner 255S profile, whose budget is 120,000 bytecodes. Long work is split into jobs by design (`source/util/Jobs.mc`); the real limits are checked on your watch. +- **Speed.** See [PERFORMANCE.md](PERFORMANCE.md). + ### M0, 2026-10-18 (version 0.1.0) `make ci` passed in full: @@ -44,4 +57,4 @@ Tests never use real secrets or personal data. Fixtures are generated or hand-wr Things the simulator can't verify, done by you on the Forerunner 70. The checklist grows with each milestone; it starts with M1. -- [ ] M1: install the settings spike from the Store beta; record field limits, accepted characters and write-back behaviour in `docs/SETTINGS_SPIKE.md`. +- [ ] M1: install the settings spike from the Store beta and run tests A to H in [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md); record the results there. |