Feature/streamed install prototype #27
+7
-5
@@ -29,12 +29,14 @@ product-ready.
|
|||||||
peer-cli flow, including local-only final state, absent root archive/sentinel,
|
peer-cli flow, including local-only final state, absent root archive/sentinel,
|
||||||
byte count, and extracted payload SHA-256 hashes.
|
byte count, and extracted payload SHA-256 hashes.
|
||||||
|
|
||||||
4. **Decide the integrity model**
|
4. **Done — Decide the integrity model**
|
||||||
|
|
||||||
Current prototype verifies streamed bytes against RAR CRC32 from the
|
Streamed installs intentionally verify against sender archive metadata for
|
||||||
sender’s archive headers. That catches corruption and provider bugs. It does
|
now: each file must match the RAR-advertised size and CRC32. That catches
|
||||||
not protect against a malicious peer lying. If you care about that, the next
|
transport corruption, truncation, and provider bugs, but does not claim
|
||||||
step is catalog-side trusted hashes for archive or extracted files.
|
malicious-peer protection. Trusted content remains a separate catalog schema
|
||||||
|
step: add catalog-owned archive or extracted-file SHA-256 hashes, then verify
|
||||||
|
those at the receiver before commit.
|
||||||
|
|
||||||
5. **Upgrade retry/resume semantics**
|
5. **Upgrade retry/resume semantics**
|
||||||
|
|
||||||
|
|||||||
@@ -122,6 +122,10 @@ Use S39-S41 to pin down low-disk streamed installs:
|
|||||||
- Non-solid and solid archives both install into `local/` without committing a
|
- Non-solid and solid archives both install into `local/` without committing a
|
||||||
root archive or root `version.ini`, so the receiver is installed but not a
|
root archive or root `version.ini`, so the receiver is installed but not a
|
||||||
downloadable source.
|
downloadable source.
|
||||||
|
- Streamed install integrity is currently sender archive integrity: size and
|
||||||
|
RAR CRC32 must match the sender's archive metadata. The SHA-256 checks in the
|
||||||
|
scenarios prove the Docker/provider path matches the source fixture; they are
|
||||||
|
not catalog-owned trust anchors.
|
||||||
- S41 verifies the fixture is actually solid inside the source container, so
|
- S41 verifies the fixture is actually solid inside the source container, so
|
||||||
solid handling stays covered by the same Docker harness as the existing
|
solid handling stays covered by the same Docker harness as the existing
|
||||||
streamed-install scenarios.
|
streamed-install scenarios.
|
||||||
|
|||||||
@@ -166,6 +166,18 @@ Most scans become O(number of game dirs), with full recursion only when needed.
|
|||||||
scratch sentinel files. `local/` and install transaction metadata are
|
scratch sentinel files. `local/` and install transaction metadata are
|
||||||
preserved, so a cancelled update of an installed game settles as local-only.
|
preserved, so a cancelled update of an installed game settles as local-only.
|
||||||
|
|
||||||
|
### Streamed install integrity
|
||||||
|
|
||||||
|
- Low-disk streamed installs request archive-derived file bytes from one peer
|
||||||
|
and write them directly into the install transaction staging directory.
|
||||||
|
- The receiver verifies every streamed file against the sender archive's file
|
||||||
|
size and RAR CRC32 before the transaction may commit. This catches truncated
|
||||||
|
streams, transport corruption, and provider bugs.
|
||||||
|
- This is not malicious-peer protection: the peer controls both the archive
|
||||||
|
metadata and the streamed bytes. A trusted-content model needs catalog-owned
|
||||||
|
hashes, either for the root archives or for extracted files, and receiver-side
|
||||||
|
SHA-256 verification against those catalog values before commit.
|
||||||
|
|
||||||
## Fault tolerance rules
|
## Fault tolerance rules
|
||||||
|
|
||||||
- Every peer is keyed by `peer_id`, not by IP address.
|
- Every peer is keyed by `peer_id`, not by IP address.
|
||||||
|
|||||||
@@ -37,6 +37,44 @@ const FRAME_CHANNEL_DEPTH: usize = 16;
|
|||||||
const STREAM_INSTALL_PROGRESS_UPDATE_INTERVAL: Duration = Duration::from_millis(500);
|
const STREAM_INSTALL_PROGRESS_UPDATE_INTERVAL: Duration = Duration::from_millis(500);
|
||||||
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
||||||
|
|
||||||
|
/// Integrity metadata advertised by the sender's RAR archive.
|
||||||
|
///
|
||||||
|
/// This catches transport corruption, truncation, and provider bugs. It is not
|
||||||
|
/// a trusted-content guarantee because a malicious peer controls both the bytes
|
||||||
|
/// and the archive metadata. Trusted content would need catalog-owned hashes.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
struct SenderArchiveIntegrity {
|
||||||
|
expected_size: u64,
|
||||||
|
expected_crc32: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SenderArchiveIntegrity {
|
||||||
|
fn new(expected_size: u64, expected_crc32: u32) -> Self {
|
||||||
|
Self {
|
||||||
|
expected_size,
|
||||||
|
expected_crc32,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn verify(self, relative_path: &str, received: u64, actual_crc32: u32) -> eyre::Result<()> {
|
||||||
|
if received != self.expected_size {
|
||||||
|
eyre::bail!(
|
||||||
|
"streamed file {relative_path} size mismatch: got {received}, expected {}",
|
||||||
|
self.expected_size
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if actual_crc32 != self.expected_crc32 {
|
||||||
|
eyre::bail!(
|
||||||
|
"streamed file {relative_path} sender RAR CRC32 mismatch: got {actual_crc32:08X}, expected {:08X}",
|
||||||
|
self.expected_crc32
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
pub type StreamInstallFuture<'a> = Pin<Box<dyn Future<Output = eyre::Result<()>> + Send + 'a>>;
|
pub type StreamInstallFuture<'a> = Pin<Box<dyn Future<Output = eyre::Result<()>> + Send + 'a>>;
|
||||||
|
|
||||||
pub trait StreamInstallProvider: Send + Sync {
|
pub trait StreamInstallProvider: Send + Sync {
|
||||||
@@ -685,10 +723,9 @@ fn bytes_per_second(bytes: u64, elapsed: Duration) -> u64 {
|
|||||||
struct IncomingFile {
|
struct IncomingFile {
|
||||||
relative_path: String,
|
relative_path: String,
|
||||||
path: PathBuf,
|
path: PathBuf,
|
||||||
expected_size: u64,
|
integrity: SenderArchiveIntegrity,
|
||||||
expected_crc32: u32,
|
|
||||||
received: u64,
|
received: u64,
|
||||||
hasher: Hasher,
|
crc32: Hasher,
|
||||||
file: File,
|
file: File,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -703,10 +740,9 @@ impl IncomingFile {
|
|||||||
Self {
|
Self {
|
||||||
relative_path,
|
relative_path,
|
||||||
path,
|
path,
|
||||||
expected_size,
|
integrity: SenderArchiveIntegrity::new(expected_size, expected_crc32),
|
||||||
expected_crc32,
|
|
||||||
received: 0,
|
received: 0,
|
||||||
hasher: Hasher::new(),
|
crc32: Hasher::new(),
|
||||||
file,
|
file,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -720,15 +756,15 @@ impl IncomingFile {
|
|||||||
) -> eyre::Result<u64> {
|
) -> eyre::Result<u64> {
|
||||||
let offset = self.received;
|
let offset = self.received;
|
||||||
let length = u64::try_from(bytes.len())?;
|
let length = u64::try_from(bytes.len())?;
|
||||||
if offset.saturating_add(length) > self.expected_size {
|
if offset.saturating_add(length) > self.integrity.expected_size {
|
||||||
eyre::bail!(
|
eyre::bail!(
|
||||||
"streamed file {} exceeded expected size {}",
|
"streamed file {} exceeded expected size {}",
|
||||||
self.relative_path,
|
self.relative_path,
|
||||||
self.expected_size
|
self.integrity.expected_size
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
self.file.write_all(&bytes).await?;
|
self.file.write_all(&bytes).await?;
|
||||||
self.hasher.update(&bytes);
|
self.crc32.update(&bytes);
|
||||||
self.received = self.received.saturating_add(length);
|
self.received = self.received.saturating_add(length);
|
||||||
|
|
||||||
let _ = tx_notify_ui.send(PeerEvent::DownloadGameFileChunkFinished {
|
let _ = tx_notify_ui.send(PeerEvent::DownloadGameFileChunkFinished {
|
||||||
@@ -750,23 +786,9 @@ impl IncomingFile {
|
|||||||
}
|
}
|
||||||
self.file.flush().await?;
|
self.file.flush().await?;
|
||||||
|
|
||||||
if self.received != self.expected_size {
|
let actual_crc32 = self.crc32.finalize();
|
||||||
eyre::bail!(
|
self.integrity
|
||||||
"streamed file {} size mismatch: got {}, expected {}",
|
.verify(&self.relative_path, self.received, actual_crc32)?;
|
||||||
self.relative_path,
|
|
||||||
self.received,
|
|
||||||
self.expected_size
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
let actual = self.hasher.finalize();
|
|
||||||
if actual != self.expected_crc32 {
|
|
||||||
eyre::bail!(
|
|
||||||
"streamed file {} CRC32 mismatch: got {actual:08X}, expected {:08X}",
|
|
||||||
self.relative_path,
|
|
||||||
self.expected_crc32
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
log::debug!(
|
log::debug!(
|
||||||
"Received streamed file {} -> {}",
|
"Received streamed file {} -> {}",
|
||||||
@@ -853,4 +875,45 @@ Details: RAR 5
|
|||||||
|
|
||||||
assert!(err.to_string().contains("has no CRC32"));
|
assert!(err.to_string().contains("has no CRC32"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sender_archive_integrity_accepts_matching_size_and_crc32() {
|
||||||
|
let bytes = b"payload";
|
||||||
|
let integrity =
|
||||||
|
SenderArchiveIntegrity::new(u64::try_from(bytes.len()).unwrap(), crc32_of(bytes));
|
||||||
|
|
||||||
|
integrity
|
||||||
|
.verify(
|
||||||
|
"bin/payload.bin",
|
||||||
|
u64::try_from(bytes.len()).unwrap(),
|
||||||
|
crc32_of(bytes),
|
||||||
|
)
|
||||||
|
.expect("matching sender archive metadata should verify");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sender_archive_integrity_rejects_size_mismatch() {
|
||||||
|
let integrity = SenderArchiveIntegrity::new(7, crc32_of(b"payload"));
|
||||||
|
let err = integrity
|
||||||
|
.verify("bin/payload.bin", 6, crc32_of(b"payload"))
|
||||||
|
.expect_err("truncated file should fail sender archive integrity");
|
||||||
|
|
||||||
|
assert!(err.to_string().contains("size mismatch"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sender_archive_integrity_rejects_crc32_mismatch() {
|
||||||
|
let integrity = SenderArchiveIntegrity::new(7, crc32_of(b"payload"));
|
||||||
|
let err = integrity
|
||||||
|
.verify("bin/payload.bin", 7, crc32_of(b"paylord"))
|
||||||
|
.expect_err("mutated file should fail sender archive integrity");
|
||||||
|
|
||||||
|
assert!(err.to_string().contains("sender RAR CRC32 mismatch"));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn crc32_of(bytes: &[u8]) -> u32 {
|
||||||
|
let mut hasher = Hasher::new();
|
||||||
|
hasher.update(bytes);
|
||||||
|
hasher.finalize()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user