Feature/streamed install prototype #27

2026-06-11T08:52:09+02:00

ddidderr commented

2026-06-11 08:52:09 +02:00

No description provided.

ddidderr added 15 commits 2026-06-11 08:52:09 +02:00

feat(peer): prototype streamed installs 373def6d44

Add a streamed-install prototype that can receive archive-derived install bytes
straight into local/ without first storing the peer-owned root archive payload.
This is intended for low-disk clients that want to install a game but opt out of
becoming a downloadable peer source for that game.

The protocol gains a current-version-only StreamInstall request and framed
StreamInstallFrame responses. The peer core owns the generic transport,
transaction, path validation, size checks, CRC32 verification, and lifecycle
state. The archive-specific work is hidden behind StreamInstallProvider so the
prototype can use unrar while the final implementation can swap in a better
provider without rewriting the peer command path.

The receiver writes into .local.installing and only promotes to local/ after the
full stream verifies. It deliberately does not write the root version.ini or
archive files, so the settled local state is installed=true, downloaded=false,
and availability=LocalOnly. That preserves the existing rule that local/ is not
served to peers and makes streamed receivers non-sources by construction.

The CLI is the only caller for now. It exposes stream-install and provides the
prototype unrar implementation with unrar lt for entry metadata and unrar p for
file bytes. This is simple and good enough to prove non-solid archive streaming,
but it is not the production provider shape for solid archives because per-file
unrar p would repeatedly decompress prefixes. The Tauri app explicitly passes
stream_install_provider: None, so the GUI behavior stays unchanged until a real
product path is designed.

Document the production-readiness work in NEXT_STEPS.md. The main follow-up is
to make the provider abstraction final-ish and replace the per-file CLI unrar
provider with a one-pass archive provider, then wire a deliberate GUI low-disk
mode, retry semantics, and broader failure scenarios.

Test Plan:
- just fmt
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= just test
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py \
  S39 S40 --build-image
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= just clippy
- git diff --check
- git diff --cached --check

Follow-up: NEXT_STEPS.md

Claude's Review notes cc147def73

fix(stream-install)!: stream archive payloads as raw frames 5dd356eca8

Streamed installs were sending FileChunk payloads through the shared JSON
Message impl. serde_json serializes bytes as arrays of integers, which
bloats wire traffic and burns CPU on large archives. Replace
StreamInstallFrame encoding with tagged frames: JSON control frames keep
their shape under tag 0, while file chunks carry raw bytes under tag 1.

The stream install metadata now carries unpacked archive size and mandatory
CRC32. The CLI unrar provider validates CRCs up front, runs one
archive-wide unrar p stream, splits stdout by listed file sizes, and
refuses trailing or missing bytes. That avoids solid archive
re-decompression and sidesteps unrar wildcard masks for path arguments.

Receivers now sample existing download progress events for streamed
installs, report staging-relative chunk paths, and retry trusted peers with
a fresh streamed-install transaction after a failed attempt. The current
protocol policy does not preserve compatibility with older stream-install
builds.

Test Plan:
- just fmt
- just test
- just clippy
- git diff --check
- git diff --cached --check

BREAKING CHANGE: StreamInstallFrame now uses tagged frames with raw chunk
payloads and requires current peers on both sides of streamed installs.

Refs: NEXT_STEPS_CLAUDES_REVIEW.md

remove NEXT_STEPS_CLAUDES_REVIEW.md, it has been applied 389511f620

feat(tauri): add low-disk streamed install action 40697a73e5

NEXT_STEPS item 1 called out that streamed install was still CLI-only
because the Tauri app started the peer with no stream provider. Users can now
choose an explicit "Low disk install" action from the game detail modal for
remote-only games instead of taking the default archive-preserving download
path.

The GUI command queues a normal peer detail fetch first so the peer database
has the file metadata needed for source validation. A small pending handoff in
Tauri routes the resulting GotGameFiles event into StreamInstallGame instead
of DownloadGameFiles, and clears that pending state on no-peer or download
failure events. This keeps the existing download continuation untouched for
the default action.

The external unrar stream provider moved from the CLI harness into
lanspread-peer so CLI and Tauri use the same implementation. Tauri resolves
the bundled unrar sidecar path and injects that provider at peer startup;
falling back to the noop provider keeps peer startup alive if the sidecar
cannot be resolved, while the streamed install operation still fails safely.

Test Plan:
- just fmt
- just test
- just frontend-test
- just clippy
- just build
- git diff --check

Refs: NEXT_STEPS.md item 1

docs(stream-install): mark one-pass provider complete c313f7c9ae

NEXT_STEPS item 2 still described the old per-file unrar provider shape even
though the current shared provider now performs one technical listing pass and
one sequential unrar payload pass per archive. Update the roadmap so the next
implementation slice starts at the remaining solid-archive policy work instead
of chasing an already-replaced extraction loop.

The item 3 wording now keeps the solid/non-solid archive fork explicit without
suggesting the current provider still needs to be swapped merely to avoid
per-file extraction.

Test Plan:
- git diff --check

Refs: NEXT_STEPS.md item 2

test(peer-cli): cover solid streamed installs 0e970dcec7

NEXT_STEPS item 3 needed solid archive handling to be a deliberate
contract instead of an incidental RAR header attribute. Add a tiny real
solid RAR fixture and S41 to the extended peer-cli scenarios so the
Docker harness proves this path end to end.

The scenario verifies the source archive with container-bundled
`unrar lt`, streams the install with the injected provider, and then
asserts the receiver is installed local-only without a root archive or
root `version.ini`. It also compares local payload SHA-256 hashes against
`unrar p` output and checks the streamed byte count matches the extracted
entries. This keeps the existing one metadata pass plus one sequential
payload pass contract covered for solid archives.

Test Plan:
- just fmt
- just test
- python3 -m py_compile crates/lanspread-peer-cli/scripts/run_extended_scenarios.py
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S41 --build-image
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S41
- git diff --check
- git diff --cached --check

Refs: NEXT_STEPS.md item 3

refactor(peer): name streamed integrity boundary bb7497c0ff

NEXT_STEPS item 4 needed the streamed-install integrity model to be a
conscious decision. Keep the current runtime behavior, but name it as
sender archive integrity: the receiver verifies streamed file size and
RAR CRC32 from the sender's archive metadata before committing the
install transaction.

This protects against truncation, transport corruption, and stream
provider bugs. It deliberately does not claim malicious-peer protection,
because the sender controls both the streamed bytes and the RAR metadata.
The docs now say that trusted content requires a future catalog schema
with catalog-owned archive or extracted-file SHA-256 hashes.

Test Plan:
- just fmt
- just test
- just clippy
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S41 --build-image
- git diff --check
- git diff --cached --check

Refs: NEXT_STEPS.md item 4

test(peer-cli): cover streamed retry fallback 88bfaeb04a

NEXT_STEPS item 5 needs streamed installs to have an explicit retry
policy. The handler already retries whole-stream attempts across the
majority-validated peer set, so add S42 to prove that behavior with the
Docker harness instead of leaving it implicit.

S42 starts two catalog-version-matching `cnctw` sources. The first source
sorts first in retry order but has `--unrar /missing-unrar`, so its stream
attempt fails before sending chunks. The second source then completes a
fresh whole-stream attempt. The scenario asserts local-only installed
state, no root archive or sentinel, no `.local.installing` staging
leftover, chunk events only from the good source, matching streamed byte
count, and SHA-256 payload equality against the good source's `unrar p`.

This pins the current policy: retry the entire stream from another
validated peer, do not preserve partial files across attempts, and do not
promise byte-offset resume.

Test Plan:
- python3 -m py_compile crates/lanspread-peer-cli/scripts/run_extended_scenarios.py
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S42
- git diff --check
- git diff --cached --check

Refs: NEXT_STEPS.md item 5

test(peer-cli): expand streamed install edge coverage 9288fda037

NEXT_STEPS item 6 called for the remaining streamed-install edge cases to
be covered in the peer-cli matrix. Add S43-S47 for already-installed
rejection, corrupt archive rollback, sender disconnect, receiver cancel,
and sorted multi-archive streaming.

The receiver-cancel scenario needs the harness to drive the same runtime
path as the GUI, so `lanspread-peer-cli` now accepts a narrow
`cancel-download` command that forwards to `PeerCommand::CancelDownload`.
A parser test covers the new JSONL command shape.

Add `fixture-multi/cnctw`, a tiny two-archive RAR fixture. S47 uses it to
prove streamed installs process root `.eti` archives in sorted order and
commit only extracted `local/` payloads, not the root archives or
`version.ini` sentinel.

Test Plan:
- just fmt
- python3 -m py_compile crates/lanspread-peer-cli/scripts/run_extended_scenarios.py
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S43 S44 S45 S46 S47 --build-image
- just test
- just clippy
- git diff --check
- git diff --cached --check

Refs: NEXT_STEPS.md item 6

feat(ui): label streamed installs as not shareable f62515451b

NEXT_STEPS item 7 needed the installed-but-not-downloaded state to be
clear to users. Keep streamed installs in the installed visual state so
sorting, filters, and the primary Play action stay unchanged, but make the
sharing limitation visible in the UI.

Cards now label that state as `Not shareable`, while the detail modal
status says `Installed, not shareable`. Downloaded-and-installed games
keep the normal `Installed` wording.

Test Plan:
- just frontend-test
- just build
- git diff --check
- git diff --cached --check

Refs: NEXT_STEPS.md item 7

test(peer-cli): align scenarios with catalog versions 47ef87748f

Remote aggregation now filters to catalog-version roots, but the checked-in
peer-cli fixtures and skew scenarios still stamped synthetic future versions.
That hid fixture rows in S3 and left scenario docs asserting latest-version
behavior.

Teach the harness the catalog versions for fixture game IDs, stamp generated
fixtures with catalog versions by default, and update skew, mesh, propagation,
and throughput scenarios to expect only catalog-version peers. Also wire S38
into the executable matrix so the documented first-play launch-setting scenario
is covered by the same full run as S1-S47.

This keeps stale peers as negative coverage: they are absent from list-games and
cannot provide descriptors, votes, or chunks. The fixture version.ini updates
are checked in so alpha, bravo, charlie, and persona roots advertise
downloadable catalog games again.

Test Plan:
- python3 -m py_compile
  crates/lanspread-peer-cli/scripts/run_extended_scenarios.py
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py \
  S3 S8 S14 S15 S16 S17 S21 S22 S23 S24 S29 S30 S31 S34 S36 S37 \
  S39 S40 S41 S42 S43 S44 S45 S46 S47 --build-image
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S38
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py
- git diff --check
- git diff --cached --check

Docs: PEER_CLI_SCENARIOS.md

[deps] cargo update 9c765aba9c

Updating http                       v1.4.1   -> v1.4.2
Updating js-sys                     v0.3.99  -> v0.3.100
Updating regex-syntax               v0.8.10  -> v0.8.11
Updating regex                      v1.12.3  -> v1.12.4
Updating s2n-codec                  v0.81.0  -> v0.82.0
Updating s2n-quic-core              v0.81.0  -> v0.82.0
Updating s2n-quic-crypto            v0.81.0  -> v0.82.0
Updating s2n-quic-platform          v0.81.0  -> v0.82.0
Updating s2n-quic-rustls            v0.81.0  -> v0.82.0
Updating s2n-quic-tls-default       v0.81.0  -> v0.82.0
Updating s2n-quic-tls               v0.81.0  -> v0.82.0
Updating s2n-quic-transport         v0.81.0  -> v0.82.0
Updating s2n-quic                   v1.81.0  -> v1.82.0
Updating uuid                       v1.23.2  -> v1.23.3
Updating wasm-bindgen-futures       v0.4.72  -> v0.4.73
Updating wasm-bindgen-macro-support v0.2.122 -> v0.2.123
Updating wasm-bindgen-macro         v0.2.122 -> v0.2.123
Updating wasm-bindgen-shared        v0.2.122 -> v0.2.123
Updating wasm-bindgen               v0.2.122 -> v0.2.123
Updating web-sys                    v0.3.99  -> v0.3.100
Updating zerocopy-derive            v0.8.50  -> v0.8.52
Updating zerocopy                   v0.8.50  -> v0.8.52

fix(peer): harden streamed install lifecycle 66c7d5912b

Claude Fable 5's branch review found that receiver cancellation or a QUIC
send failure could leave the sender-side archive producer blocked on the
bounded frame channel. That kept the outbound transfer guard alive and could
block later installs or updates of the same game.

Route archive frames through a cancellable StreamInstallFrameSink instead of
exposing the raw channel sender to providers. The QUIC forwarder now cancels
and closes the receive side before awaiting the producer, so a blocked send
wakes and the transfer guard can drop normally.

Make PeerCommand::StreamInstallGame own its peer metadata preflight inside the
peer core. The Tauri layer now sends the command directly, and the peer runtime
fetches file details from catalog-version peers before running the existing
majority validation and retry logic. This removes the UI-only pending streamed
install set and gives PeerEvent::GotGameFiles one meaning again: continue a
normal archive download.

Tighten the receiver transaction edge cases too. Rollback removes a newly
created empty game root, but preserves pre-existing roots. Once streamed
staging has been promoted to local/, intent or launch-settings cleanup failures
are logged for startup recovery instead of reporting a failed install for bytes
that are already committed.

Accept missing RAR CRC32 metadata for zero-byte files as CRC32 00000000 while
still requiring CRC32 metadata for non-empty files. Update the peer README,
scenario docs, and next-steps handoff so the documented ownership and remaining
trust limitation match the implementation.

Test Plan:
- just fmt
- just test
- just frontend-test
- just clippy
- git diff --check
- python3 -m py_compile \
  crates/lanspread-peer-cli/scripts/run_extended_scenarios.py
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py \
  S39 S40 S41 S42 S43 S44 S45 S46 S47 --build-image

Refs: streamed-install review handoff from Claude Fable 5

fix(peer): drain streamed install senders after completion c00e6eae84

A streamed install sender kept the original frame sink alive outside the
producer task. After the producer sent Complete, or an Error for a provider
failure, the forwarding loop still had a live mpsc sender in scope and waited
forever for another frame.

Move the sink into the producer so the channel closes when the producer exits.
That lets the QUIC writer close, the request task return, and the outbound
TransferGuard drop after successful streamed installs and provider-side
failures.

The peer-cli harness now keeps the outbound-transfer map it passes into the
peer runtime and exposes per-game counts in status. S39 asserts that the source
has no active outbound transfer for cnctw after the streamed install finishes,
which catches the sender-side lifecycle leak that receiver-only assertions
missed. The peer-cli README and scenario table document that status field and
expectation.

Test Plan:
- just fmt
- just test
- just clippy
- git diff --check
- git diff --cached --check
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S39 S40 --build-image
- python3 crates/lanspread-peer-cli/scripts/run_extended_scenarios.py S41 S42 S43 S44 S45 S46 S47

Refs: NEXT_STEPS.md streamed install lifecycle hardening

ddidderr merged commit c00e6eae84 into main

2026-06-11 08:52:33 +02:00

ddidderr deleted branch feature/streamed-install-prototype

2026-06-11 08:52:33 +02:00

ddidderr referenced this pull request

2026-06-11 08:54:14 +02:00

option to remove install files for people with low disk space #22

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ddidderr/lanspread#27