ddidderr
c072b93726
Completed uploads used to copy every staged chunk into a second file before renaming the result into data/complete. That doubled write volume and required peak disk space for both the chunk set and the final file. Write each chunk directly into one private temp upload file at its final offset instead. After a chunk write succeeds, record a tiny durable completion marker for progress and resume scans. Completion now verifies the temp file length and all markers, then renames the temp file into the completed upload directory. Add UPL_TEMP_DIR and --temp-dir so operators can choose where upload metadata, markers, and temp files live. The default remains data/staging, and docs call out that the temp directory must be on the same filesystem as data/complete for atomic promotion. The nginx example now aliases only the completed upload directory, and the smoke test verifies that final-file alias. This keeps the existing length-based validation model; it does not add per-chunk hashing. Test Plan: - just check - just nginx-smoke - cargo clippy && cargo clippy --benches && cargo clippy --tests - cargo +nightly fmt --all - cargo clippy && cargo clippy --benches && cargo clippy --tests Refs: none
upl
upl is a small personal resumable upload service. The intended deployment is:
browser -> nginx -> upl Rust server -> local filesystem
The server writes upload chunks directly into an inaccessible temp file at their final offsets. Once every chunk is present, completion atomically renames that temp file into the completed upload directory.
Project Structure
upl
Rust server
src/main.rs binary entrypoint and listener setup
src/app.rs Axum router, shared state, static file service
src/api.rs HTTP handlers and API error responses
src/model.rs JSON request, response, and metadata shapes
src/storage.rs local filesystem layout, offset writes, and final rename
src/lib.rs library surface used by integration tests
Browser UI
static/index.html upload tool markup
static/styles.css responsive tool styling
static/app.js upload scheduler, retries, and browser resume state
Deployment
deploy/nginx/ nginx reverse proxy example
scripts/ reusable local smoke tests
Validation
tests/ integration tests for server behavior
TESTS.md reusable manual and automated test checklist
Configuration
--bindsets the listen address. It overridesUPL_BINDand defaults to127.0.0.1:3000.--static-dirsets the static asset directory. It overridesUPL_STATIC_DIRand defaults tostatic/inside this repository.--data-dirsets the completed upload data root. Completed files land under itscomplete/subdirectory. It overridesUPL_DATA_DIRand defaults todata/inside this repository.--temp-dirsets the directory for upload metadata, completion markers, and inaccessible temp upload files. It overridesUPL_TEMP_DIRand defaults to<data-dir>/staging.upl --helpprints the full argument help text.- The server accepts request bodies up to 64 MiB, which leaves room for the
planned 16 MiB upload chunks and matches the nginx example in
PLAN.md. - Keep
UPL_TEMP_DIRon the same filesystem as<data-dir>/completeso completion can promote files with an atomic rename.
Common Commands
Use the justfile for routine tasks:
just check
just run
just check also syntax-checks the static browser JavaScript with node.
nginx
Run upl on localhost and put nginx in front of it for TLS and access control:
UPL_BIND=127.0.0.1:3000 \
UPL_DATA_DIR=/srv/upl/data \
UPL_TEMP_DIR=/srv/upl/data/staging \
upl
Use deploy/nginx/upl.conf.example as the starting point for the nginx site.
Before exposing the service, replace the certificate paths and add a protection
layer such as HTTP basic auth, an IP allowlist, or VPN-only access. The nginx
example aliases only /srv/upl/data/complete; do not expose UPL_TEMP_DIR.
For a local Docker-based reverse-proxy smoke test:
just nginx-smoke
The smoke test binds the Rust server to 0.0.0.0 so the nginx container can
reach it through Docker's host gateway. The production nginx example keeps the
server bound to localhost.