path: root/docs/DEVELOPER_GUIDE.md

<!--
  Author:  cirakg & claude ai
  Summary: How to build, test and debug OTP Wallet in Docker, how the tooling works, and the coding conventions.
-->
# Developer guide

Everything runs in Docker through the `Makefile`; the host needs only Docker and make. This guide covers the daily workflow, how the tooling works underneath, and the conventions every change follows. The design and the build plan are in the [README](../README.md).

## First-time setup

1. Docker access: your user must be in the `docker` group (log out and in after joining, or run `newgrp docker` in a terminal).
2. `make image` builds the development image (about 1.1 GB; it needs the network).
3. `make sdk` is interactive: you accept Garmin's agreements, log in, and the SDK (9.2.0) plus the device files of all 72 products download into `.cache/home` (about 1.1 GB). The login token is deleted at the end.
4. `make key` creates the developer key in `.cache/keys/`. Back that folder up; see [README 15.6](../README.md#156-developer-key).

## Daily workflow

| Task | Command |
| --- | --- |
| Build for one device | `make build DEVICE=fr70` |
| Run the app and watch it in your browser | `make sim DEVICE=fr70`, then open http://127.0.0.1:6080/vnc.html |
| Unit tests (Python tools and Monkey C, 4 devices) | `make test` |
| All static checks | `make lint` |
| Screenshots against the approved images | `make screenshots` |
| Accept changed screens after reviewing build/screenshots/ | `make screenshots-approve` |
| Start-up time and memory budgets (app and glance) | `make perf`, `make memtest` |
| Largest 2D codes at twice the steps per event (watchdog) | `make watchdog` |
| Signed package for the Store beta | `make package` |
| Regenerate the crypto/OTP test vectors | `make vectors` |
| Build the M1 settings spike | `make spike` (instructions: [SETTINGS_SPIKE.md](SETTINGS_SPIKE.md)) |
| Everything, in order | `make ci` |
| A shell in the container | `make shell` |

Use other devices with `TEST_DEVICES="fr70 venu3"` on any target that takes a device list. Every target runs serially, even under `make -j`.

## Repository layout

| Path | Content |
| --- | --- |
| `source/` | The watch app (Monkey C), one folder per module (README section 5) |
| `test/` | Monkey C unit tests (`(:test)` functions), compiled only into test builds |
| `test/golden/` | Approved simulator screenshots: one folder per device, one image per screen of the UI flow |
| `resources/` | Strings, the default launcher icon, the option properties and, from M7, the settings form |
| `resources-icons/`, `icons.jungle` | Launcher icons at each size the products need, generated by `make icons` |
| `manifest.xml`, `monkey.jungle`, `test.jungle`, `uitest.jungle`, `stress.jungle` | App manifest and build configuration |
| `docker/` | Images (`Dockerfile`, `docs.Dockerfile`), hash-locked dependencies, and the container scripts |
| `tools/` | Python tools (test-log judge, screenshot checker, linters, icon generator) and their tests |
| `docs/` | This guide and the other documents |
| `.cache/`, `build/` | Git-ignored: SDK, device files, key; build outputs |

## How the tooling works

### Builds

`docker/scripts/build-app.sh` compiles one device in one mode:

- `debug`: what `make build`, `make sim` and `make perf` use. It contains the `(:debug)` diagnostics, the debug menu items (demo entries, performance overlay, on-watch benchmark) and the fixtures.
- `uitest`: a debug build plus `uitest.jungle`, for `make screenshots`. It keeps the `(:uitest)` half of `source/util/TestMode.mc`, so the app starts with a wiped store, fake entries, the PIN 1234, a frozen clock and a 10 s auto-lock.
- `uitest-extra`: like `uitest`, but with `uitest-extra.jungle`, which sets `TestMode.SCENARIO` to `SCENARIO_EXTRA`, so `Fixtures.loadExtra` loads a second scenario: a legacy 6-digit PIN without the `wl` marker, a queued Require PIN, UPC-A, Code 39, a broken EAN-13, a large QR and the light theme. `screenshots.sh` runs it after the main flow with `ui_flow.py --scenario extra`.
- `stress`: a debug build plus `stress.jungle`, for `make watchdog`. It keeps the `(:stress)` half of `source/util/StressTest.mc`: the app starts straight into the watchdog stress test instead of the list.
- `release`: `monkeyc -r` removes every `(:debug)` declaration; used by `make package`.
- `test`: adds `test.jungle` and `--unit-test`.

Every build uses strict type checking (`-l 3`) and fails on any compiler warning. The launcher icon is the classic source of warnings: the compiler warns when it has to scale an icon, so every product gets an icon of exactly its size through `icons.jungle`. After adding a product to the manifest, run `make sdk` (device files) and then `make icons`.

### The headless simulator

The simulator is a GTK program. The scripts start it on a virtual display (Xvfb on `:99`) with the openbox window manager, so its window can be found for screenshots. Behaviour verified with SDK 9.2.0:

- It accepts `monkeydo` on TCP port 1234; the scripts wait until that port listens (read from `/proc/net/tcp`).
- It keeps a single-instance lock file, `~/Sim-dev`, holding its process ID. That file survives in `.cache/home`, and a later container often reuses the process ID for something else, so the simulator would believe it's already running. `ciq_start_simulator` deletes the stale lock first.
- `monkeydo` exits with 1 even when every unit test passed, so `tools/ciq_test_report.py` judges the console output instead.
- For a normal (non-test) run, `monkeydo` blocks while the app runs and streams the app's `System.println` output. Quotes in that output arrive escaped, which is why the diagnostics use plain `key=value` pairs.
- **It starts apps that have a glance in the glance.** `ciq_launch_app` waits for the glance's `DIAG shown glance` line, then opens the app the way the simulator accepts. On touch screens that's a tap on the glance; on button-only watches it's DOWN (which selects the glance) and then START. START alone does nothing in the simulator. The press is retried if the app doesn't open.
- **A second simulator in the same container opens a second window**, and input may go to the wrong one. `ciq_start_simulator` reuses one that's already listening.
- Both `monkeydo` and the harness append to the app log; with `>` one would overwrite the other's lines.

### Monkey C facts that matter here (verified with SDK 9.2.0)

- `Long` is 64-bit and wraps on overflow; `>>` is an arithmetic shift, so logical shifts and rotations mask off the copied sign bits (see `source/crypto/Sha512.mc`).
- Assigning a value above 255 to a ByteArray element throws; mask with `& 0xFF`.
- Native HMAC supports only SHA-256; HMAC-SHA1 and HMAC-SHA512 go through `Hmac.generic`.
- Function calls are expensive in the VM; hot loops inline small helpers (SHA-512 went 37 % faster that way).
- **Long `&&` chains with `instanceof`** can make the compiler's type checker run out of memory: one test condition needed more than 6 GB until it was split into separate `if` statements.
- **JSON resources over 8 KB** produce a compiler warning, so large tables are split; the PDF417 patterns are three resources, one per cluster.
- **The watchdog** ("Code Executed Too Long") is not enforced in unit-test runs, but it is in normal simulator runs, with the device's `watchdogCount` per event: 120,000 bytecodes on the Forerunner 255/255S, 240,000 on the others. M4 measured what it means. 200 byte-wise PBKDF2 iterations in one event tripped it on the 255S, and so did two SHA-512 codes in one draw. So:
  - Anything long runs as a `Jobs.Job` with a bounded step (`Kdf.ITERATIONS_PER_STEP` = 100).
  - Screens compute at most one SHA-512 code per event, in a timer event separate from drawing.
  - The scripted UI flow runs on the 255S profile, so a regression trips there.
  - **M8 found that step sizes guessed from "bytecodes per loop" were 2 to 3 times too large**: a simple loop iteration costs 50 to 70 counted bytecodes, not 10 to 20. Every step of the 2D encoders, the cache packer and the run finder is now sized by measurement in `make watchdog`, which runs the largest input of each with *two* steps per event: a pass means every step uses at most half the budget. Do the same for any new job: add its worst input to `StressTest.start()`.
  - `Jobs.JobRunner` runs one step per 50 ms tick on the 512 KB watches and two on the others (`Jobs.stepsPerTick`, by app memory). `tools/gen_products.py --check` (part of `make lint`) fails if a product's `watchdogCount` doesn't match that rule.
  - Drawing counts too: one `fillRectangle` per run in `onUpdate` is fine up to about 1,500 runs per 120,000 bytecodes. `RunsPainter` paints larger codes into a buffered bitmap in steps first (the Venu X1 fits a QR with 8,000 runs).
- `StringUtil.convertEncodedString` stops the app with an uncatchable "System Error" on invalid UTF-8; `try` doesn't help. Decode untrusted bytes with `Bytes.toUtf8`, which validates first.
- `Application.Storage` stores ByteArrays (its `ValueType` includes them, unlike `Application.PropertyValueType`), so records are stored as single ByteArrays of known size. Dictionaries passed to it must be typed `Dictionary<Application.Storage.KeyType, Application.Storage.ValueType>`; `Dictionary<String, ...>` doesn't type-check.
- `Array.slice` returns `Array<Object or Null>`; cast it back (`as Array<String>`).
- `Application.Properties.getValue` on a property that isn't in `properties.xml` stops the app with an uncatchable "Unexpected Type Error". Every option is therefore declared in `resources/settings/properties.xml`.
- **On touch screens, a tap is delivered as the `onSelect` behaviour first**, and `onTap` only runs if `onSelect` returns false. Delegates that need the tap position (the list, the PIN keypad) return false from `onSelect` on touch screens, and handle the START button in `onKey`.
- **Glances:** because `getGlanceView` is `(:glance)`, the compiler checks the whole app class against the glance scope. Methods that never run in glance mode opt out with `(:typecheck(disableGlanceCheck))`.
  - `AppBase.initialize` runs before non-glance code is loaded, so `Diag` (which it calls) is `(:glance)`.
  - `onStop` also runs after the glance, so it only touches `AppState` once the full app has started.
- `Attention.vibrate` makes the simulator shake the device drawing, which spoils screenshots; the UI-test build doesn't vibrate.
- A `ConfirmationDelegate` closes its dialog after `onResponse`, so a view switch from `onResponse` must wait for a short timer.
- `Attention has :vibrate` inside a class that has a private `vibrate` method warns (the symbol lookup finds the method); the helper is called `buzz`.
- `has` and `hidden` are reserved words; they can't name a function or a variable.
- Under strict typing, a property value (`Application.Properties.ValueType`) can't be compared with `==` to a Boolean or Number: narrow it with `instanceof` first (`Forms.flag`, `Forms.number`).
- **Memory of objects** (measured on the Forerunner 255S): an `Entry` takes about 490 bytes even when empty, an import item with its entry about 760, and an 80-byte ByteArray about 110. Hold many small records packed in ByteArrays (`ImportItem.pack`) rather than as objects, and grids in ByteArrays rather than `Array<Number>` (a 132 x 132 grid of Numbers took about 90 KB).
- The app's code and constant data are loaded whole at start: about 156 KB of the 255S's 512 KB before anything runs (release and debug builds alike). Only resources (`WatchUi.loadResource`) load on demand.
- `module` is a reserved word too.
- **The glance** loads every `(:glance)` module; the crypto it needs for the favourite code (`Hkdf`, `Hmac`, `Hashes`, `Sha512`, `Aes`, `SealedBox`, `Otp`, `Bytes`, `Clock`, `TestMode`) took it from 19 % to 37 % of its 64 KB. A glance module can't name app-only classes (the compiler checks the glance scope), hence `GlanceCode` (glance) and `GlanceStore` (app) in one file. In the simulator, an app that exits does not return to its glance; a new launch starts there (`screenshots.sh` launches again for `glance_code.png`).
- Zeroing a ByteArray is a byte loop in the VM: zeroing a 10 KB import text in one event tripped the watchdog on the Forerunner 255S. The native HMAC takes data in pieces (`update`), so large content is hashed without copying it into one buffer.

### Screenshots and the UI flow

`make screenshots` builds both UI-test variants and runs `tools/ui_flow.py` on every device in `TEST_DEVICES` and `SPOT_DEVICES`: the main flow, the glance with its code, the extra scenario, then the glance without a code. All seven watches take about 65 minutes; the Forerunner 70 alone (`make screenshots TEST_DEVICES=fr70 SPOT_DEVICES=`) about 8.

1. **Input.** The flow clicks the buttons where the device's `simulator.json` draws them. It taps and swipes on touch screens, and long-presses for the menu where there's no MENU button.
2. **Waiting.** After each input it waits for the screen's `DIAG shown <screen>` line. The list and the OTP screen print it only once every visible code is drawn.
3. **Touch targets.** It reads where to tap from `DIAG list_focus` (the focused row) and `DIAG pin_keys` (the keypad).
4. **Comparison.** Each screen is cut out of the window capture by `tools/sim_screenshot.py` and compared with `test/golden/<device>/<screen>.png`. That tool finds the device drawing by its bezel, so it doesn't assume a window layout. Differences go to `build/screenshots/<device>/<screen>-diff.png`.

Because the entry IDs, the clock and the codes are fixed in the UI-test build, every run is pixel-identical. Menu items are opened by label from the `DIAG menu items=` line; touch-only watches show menus as a freely scrolling list without focus, so their menu geometry is in `TOUCH_MENU_ROWS` in `tools/ui_flow.py` (a new touch-only test device needs an entry). PIN steps wait for `DIAG pin_state`, the encoding progress is checked from `DIAG code_progress`, and `shot(..., review=True)` saves screens that change from run to run (the benchmark, the progress) without comparing them. A state that needs other fixture data goes into the extra scenario, or a new `SCENARIO_*` with its own jungle and build mode.

### Performance and memory

Debug builds print `DIAG first_frame startup_ms=... mem_used=... mem_total=...` after their first frame, and the glance `DIAG glance mem_used=... mem_total=...` (`source/util/Diag.mc`). `docker/scripts/app-diag.sh` adds a `HARNESS first_frame wall_ms=...` line, the wall-clock time from launching `monkeydo` until that first frame. `tools/check_budgets.py` compares both with `tools/budgets.json`. See [PERFORMANCE.md](PERFORMANCE.md) for the numbers and their limits.

## Conventions

- **Headers**: every file starts with `Author: cirakg & claude ai` and a one-line `Summary:` in its comment syntax. JSON files use `author`/`_author` and `description`/`_summary` keys, and PNG images use Author and Description text chunks. Only `docker/docs/package-lock.json`, which npm rewrites, is exempt. `make lint` enforces this (rule HDR001).
- **Comments**: every module, class and non-trivial function explains its purpose, inputs, outputs, errors and security implications.
- **No logging**: `System.println` is allowed only in `source/util/Diag.mc`, and only in `(:debug)` code. Secrets, codes and entry names never go to a log.
- **No network**: `manifest.xml` requests no permissions and no code uses `Toybox.Communications`. `make lint` enforces both.
- **Secrets**: no real secrets anywhere in the repository; test data is fake and says so.
- **Small functions, named constants**, strict typing in Monkey C (`-l 3`) and Python (`mypy --strict`), and warnings treated as errors in the compiler, pytest and the linters.
- **Commits**: [Conventional Commits](https://www.conventionalcommits.org/) on `main`, one finished and verified unit per commit, with `CHANGELOG.md` updated.

## Adding things

- **A device**: add `<iq:product id="..."/>` to `manifest.xml`, then `make sdk` (downloads its files), `make icons` and `make products` (checks it meets the target rule: a watch, API 5.0 or newer, at least 512 KB, and the watchdog the job runner assumes for its memory).
- **A test device**: add it to `TEST_DEVICES` (unit tests, perf and screenshots) or `SPOT_DEVICES` (screenshots only) in the `Makefile`. Then run `make screenshots`, review `build/screenshots/<device>/`, and run `make screenshots-approve`.
- **A screen in the UI flow**: print `Diag.shown("<name>")` after its first complete draw, add the steps to `run_flow` in `tools/ui_flow.py`, and approve the new goldens after reviewing them.
- **Import inputs**: new parser cases go into `test/import/` (fixtures are built in `ImportSupport.mc`, all fake). To widen the fuzz corpus, change the seeds or mutations in `tools/gen_fuzz_corpus.py` and run `make corpus`. The Python tests check that the committed corpus matches the generator.
- **A settings form or field**:
  1. Declare each property in `resources/settings/properties.xml`. Reading an undeclared property crashes the app in a way that can't be caught.
  2. Add the setting to `resources/settings/settings.xml` and its strings to `resources/strings/strings.xml`. `make lint` validates both against the SDK schema and checks that they agree (`tools/check_resources.py`).
  3. Read the fields through a `Forms.Store` in a reader like `source/settings/OtpForm.mc`, so tests can use `Forms.MemoryStore`. Put every field in the form's hash content and its default in the form's `defaults()`; the test `settingsDefaultsMatchProperties` checks those against `properties.xml`.
  4. Add a phase to `SettingsJob` in README 10.3 order. Heavy work (encoding, many records) goes into its own steps: one step is one timer event under the watchdog.
- **On-watch numbers**: sideload a debug build (`make build DEVICE=fr70`, copy `build/debug/fr70.prg` to the watch's `GARMIN/APPS` folder over USB). Then open the menu and choose "Debug: benchmark". The watch installs the app and removes the `.prg` from the folder; to remove the app, use the watch's own app menu, then delete the `fr70.*` storage files in `GARMIN/APPS/DATA` and `GARMIN/APPS/SETTINGS` ([YOUR_STEPS.md](YOUR_STEPS.md#45-remove-the-debug-build)).
- **A barcode symbology**: add an encoder module under `source/codes/` and dispatch to it in `Codes.encode`. Then:
  - Add reference rows in `tools/gen_code_vectors.py` (zxing-cpp's writer, or segno for QR) and run `make codes`.
  - Add `CODEDUMP` lines for seeded payloads in its test, so `tools/check_codes.py` decodes them in every test run.
  - Add its zxing-cpp format to `tools/barcode_ref.py`.

  When decoding with zxing-cpp, restrict the format: by default it reads Code 39 as full-ASCII, which changes pairs such as "+S".