Feature/streamed install prototype #27

Merged
ddidderr merged 15 commits from feature/streamed-install-prototype into main 2026-06-11 08:52:33 +02:00
4 changed files with 112 additions and 31 deletions
Showing only changes of commit bb7497c0ff - Show all commits
+7 -5
View File
@@ -29,12 +29,14 @@ product-ready.
peer-cli flow, including local-only final state, absent root archive/sentinel,
byte count, and extracted payload SHA-256 hashes.
4. **Decide the integrity model**
4. **Done — Decide the integrity model**
Current prototype verifies streamed bytes against RAR CRC32 from the
senders archive headers. That catches corruption and provider bugs. It does
not protect against a malicious peer lying. If you care about that, the next
step is catalog-side trusted hashes for archive or extracted files.
Streamed installs intentionally verify against sender archive metadata for
now: each file must match the RAR-advertised size and CRC32. That catches
transport corruption, truncation, and provider bugs, but does not claim
malicious-peer protection. Trusted content remains a separate catalog schema
step: add catalog-owned archive or extracted-file SHA-256 hashes, then verify
those at the receiver before commit.
5. **Upgrade retry/resume semantics**
+4
View File
@@ -122,6 +122,10 @@ Use S39-S41 to pin down low-disk streamed installs:
- Non-solid and solid archives both install into `local/` without committing a
root archive or root `version.ini`, so the receiver is installed but not a
downloadable source.
- Streamed install integrity is currently sender archive integrity: size and
RAR CRC32 must match the sender's archive metadata. The SHA-256 checks in the
scenarios prove the Docker/provider path matches the source fixture; they are
not catalog-owned trust anchors.
- S41 verifies the fixture is actually solid inside the source container, so
solid handling stays covered by the same Docker harness as the existing
streamed-install scenarios.
+12
View File
@@ -166,6 +166,18 @@ Most scans become O(number of game dirs), with full recursion only when needed.
scratch sentinel files. `local/` and install transaction metadata are
preserved, so a cancelled update of an installed game settles as local-only.
### Streamed install integrity
- Low-disk streamed installs request archive-derived file bytes from one peer
and write them directly into the install transaction staging directory.
- The receiver verifies every streamed file against the sender archive's file
size and RAR CRC32 before the transaction may commit. This catches truncated
streams, transport corruption, and provider bugs.
- This is not malicious-peer protection: the peer controls both the archive
metadata and the streamed bytes. A trusted-content model needs catalog-owned
hashes, either for the root archives or for extracted files, and receiver-side
SHA-256 verification against those catalog values before commit.
## Fault tolerance rules
- Every peer is keyed by `peer_id`, not by IP address.
+89 -26
View File
@@ -37,6 +37,44 @@ const FRAME_CHANNEL_DEPTH: usize = 16;
const STREAM_INSTALL_PROGRESS_UPDATE_INTERVAL: Duration = Duration::from_millis(500);
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
/// Integrity metadata advertised by the sender's RAR archive.
///
/// This catches transport corruption, truncation, and provider bugs. It is not
/// a trusted-content guarantee because a malicious peer controls both the bytes
/// and the archive metadata. Trusted content would need catalog-owned hashes.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct SenderArchiveIntegrity {
expected_size: u64,
expected_crc32: u32,
}
impl SenderArchiveIntegrity {
fn new(expected_size: u64, expected_crc32: u32) -> Self {
Self {
expected_size,
expected_crc32,
}
}
fn verify(self, relative_path: &str, received: u64, actual_crc32: u32) -> eyre::Result<()> {
if received != self.expected_size {
eyre::bail!(
"streamed file {relative_path} size mismatch: got {received}, expected {}",
self.expected_size
);
}
if actual_crc32 != self.expected_crc32 {
eyre::bail!(
"streamed file {relative_path} sender RAR CRC32 mismatch: got {actual_crc32:08X}, expected {:08X}",
self.expected_crc32
);
}
Ok(())
}
}
pub type StreamInstallFuture<'a> = Pin<Box<dyn Future<Output = eyre::Result<()>> + Send + 'a>>;
pub trait StreamInstallProvider: Send + Sync {
@@ -685,10 +723,9 @@ fn bytes_per_second(bytes: u64, elapsed: Duration) -> u64 {
struct IncomingFile {
relative_path: String,
path: PathBuf,
expected_size: u64,
expected_crc32: u32,
integrity: SenderArchiveIntegrity,
received: u64,
hasher: Hasher,
crc32: Hasher,
file: File,
}
@@ -703,10 +740,9 @@ impl IncomingFile {
Self {
relative_path,
path,
expected_size,
expected_crc32,
integrity: SenderArchiveIntegrity::new(expected_size, expected_crc32),
received: 0,
hasher: Hasher::new(),
crc32: Hasher::new(),
file,
}
}
@@ -720,15 +756,15 @@ impl IncomingFile {
) -> eyre::Result<u64> {
let offset = self.received;
let length = u64::try_from(bytes.len())?;
if offset.saturating_add(length) > self.expected_size {
if offset.saturating_add(length) > self.integrity.expected_size {
eyre::bail!(
"streamed file {} exceeded expected size {}",
self.relative_path,
self.expected_size
self.integrity.expected_size
);
}
self.file.write_all(&bytes).await?;
self.hasher.update(&bytes);
self.crc32.update(&bytes);
self.received = self.received.saturating_add(length);
let _ = tx_notify_ui.send(PeerEvent::DownloadGameFileChunkFinished {
@@ -750,23 +786,9 @@ impl IncomingFile {
}
self.file.flush().await?;
if self.received != self.expected_size {
eyre::bail!(
"streamed file {} size mismatch: got {}, expected {}",
self.relative_path,
self.received,
self.expected_size
);
}
let actual = self.hasher.finalize();
if actual != self.expected_crc32 {
eyre::bail!(
"streamed file {} CRC32 mismatch: got {actual:08X}, expected {:08X}",
self.relative_path,
self.expected_crc32
);
}
let actual_crc32 = self.crc32.finalize();
self.integrity
.verify(&self.relative_path, self.received, actual_crc32)?;
log::debug!(
"Received streamed file {} -> {}",
@@ -853,4 +875,45 @@ Details: RAR 5
assert!(err.to_string().contains("has no CRC32"));
}
#[test]
fn sender_archive_integrity_accepts_matching_size_and_crc32() {
let bytes = b"payload";
let integrity =
SenderArchiveIntegrity::new(u64::try_from(bytes.len()).unwrap(), crc32_of(bytes));
integrity
.verify(
"bin/payload.bin",
u64::try_from(bytes.len()).unwrap(),
crc32_of(bytes),
)
.expect("matching sender archive metadata should verify");
}
#[test]
fn sender_archive_integrity_rejects_size_mismatch() {
let integrity = SenderArchiveIntegrity::new(7, crc32_of(b"payload"));
let err = integrity
.verify("bin/payload.bin", 6, crc32_of(b"payload"))
.expect_err("truncated file should fail sender archive integrity");
assert!(err.to_string().contains("size mismatch"));
}
#[test]
fn sender_archive_integrity_rejects_crc32_mismatch() {
let integrity = SenderArchiveIntegrity::new(7, crc32_of(b"payload"));
let err = integrity
.verify("bin/payload.bin", 7, crc32_of(b"paylord"))
.expect_err("mutated file should fail sender archive integrity");
assert!(err.to_string().contains("sender RAR CRC32 mismatch"));
}
fn crc32_of(bytes: &[u8]) -> u32 {
let mut hasher = Hasher::new();
hasher.update(bytes);
hasher.finalize()
}
}