feat: write chunks directly to temp upload files

Completed uploads used to copy every staged chunk into a second file before
renaming the result into data/complete. That doubled write volume and required
peak disk space for both the chunk set and the final file.

Write each chunk directly into one private temp upload file at its final offset
instead. After a chunk write succeeds, record a tiny durable completion marker
for progress and resume scans. Completion now verifies the temp file length and
all markers, then renames the temp file into the completed upload directory.

Add UPL_TEMP_DIR and --temp-dir so operators can choose where upload metadata,
markers, and temp files live. The default remains data/staging, and docs call
out that the temp directory must be on the same filesystem as data/complete for
atomic promotion. The nginx example now aliases only the completed upload
directory, and the smoke test verifies that final-file alias.

This keeps the existing length-based validation model; it does not add per-chunk
hashing.

Test Plan:
- just check
- just nginx-smoke
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt --all
- cargo clippy && cargo clippy --benches && cargo clippy --tests

Refs: none
This commit is contained in:
2026-05-30 18:10:05 +02:00
parent 428af52e2f
commit c072b93726
10 changed files with 232 additions and 101 deletions
+9 -6
View File
@@ -10,23 +10,26 @@ Keep this file as the reusable verification checklist while implementing
- Current coverage:
- `GET /` serves the static browser page.
- `GET /healthz` reports `ok`.
- `POST /api/uploads` creates `meta.json` and chunk directories.
- `POST /api/uploads` creates `meta.json`, a temp upload file, and a
completion-marker directory.
- `POST /api/uploads` rejects an empty file name.
- `PUT /api/uploads/:id/chunks/:index` stores validated chunk files.
- `PUT /api/uploads/:id/chunks/:index` writes validated chunks into the
temp upload file and records completion markers.
- `PUT /api/uploads/:id/chunks/:index` rejects wrong-size chunks.
- `PUT /api/uploads/:id/chunks/:index` rejects out-of-range indexes.
- `PUT /api/uploads/:id/chunks/:index` accepts duplicate chunks.
- `GET /api/uploads/:id` reports completed chunks from disk.
- `POST /api/uploads/:id/complete` assembles verified chunks and removes
staging data.
- `GET /api/uploads/:id` reports completed chunks from disk markers.
- `POST /api/uploads/:id/complete` renames the verified temp upload file
and removes staging data.
- `POST /api/uploads/:id/complete` rejects incomplete uploads.
- `POST /api/uploads/:id/complete` rejects corrupt chunk files.
- `POST /api/uploads/:id/complete` rejects tampered temp upload files.
- `static/app.js` passes `node --check`.
- `just nginx-smoke`
- Runs upl behind nginx in Docker.
- Uploads a 17 MiB file through nginx.
- Restarts the Rust backend mid-upload, resumes through nginx, completes, and
compares SHA-256 hashes.
- Serves the completed file through nginx's final-upload alias.
## Manual