feat: support parallel multi-file uploads

The browser upload flow was built around one selected file and one global
upload state. That made the existing chunk pool useful for a single file, but
users could not start several selected files at the same time.

Refactor the browser state into per-file upload items. Each selected file now
has its own upload record, completed-chunk set, abort controller, retry state,
progress row, and saved IndexedDB resume record. The picker accepts multiple
files, `Start all` and `Resume all` use a bounded file-level pool, and each file
keeps the existing bounded chunk pool. This keeps parallel uploads useful
without letting one large selection create unbounded request fan-out.

Keep the server API unchanged. Each file still receives a separate server upload
id, and server-side progress remains authoritative before any missing chunks are
scheduled. Terminal conflicts still stop the affected file without overwriting
completed data.

Update the user-facing markup, styles, project docs, and test checklist for the
multi-file scheduler. Add a server regression test that interleaves two uploads
and verifies the completed files contain exactly their own bytes.

Test Plan:
- just check
- git diff --check
This commit is contained in:
2026-05-30 18:32:29 +02:00
parent a7b3abd54a
commit 1923ff2a6f
8 changed files with 635 additions and 192 deletions
+10 -8
View File
@@ -26,9 +26,9 @@ The program should stay simple:
The browser owns file selection and chunk scheduling. The browser owns file selection and chunk scheduling.
- Let the user pick one file. - Let the user pick one or more files.
- Slice it into fixed-size chunks with `Blob.slice()`. - Slice it into fixed-size chunks with `Blob.slice()`.
- Upload a few chunks concurrently. - Upload a few files concurrently, with a separate chunk pool per file.
- Retry failed chunks with exponential backoff. - Retry failed chunks with exponential backoff.
- Persist pending upload state in IndexedDB. - Persist pending upload state in IndexedDB.
- Use the File System Access API when available so the same local file can be - Use the File System Access API when available so the same local file can be
@@ -194,11 +194,12 @@ The server should not delete staging data until assembly succeeds.
### First Upload ### First Upload
1. User selects a file. 1. User selects one or more files.
2. Browser calls `POST /api/uploads`. 2. Browser creates one selected upload row per file.
3. Browser stores the returned `upload_id` and file handle in IndexedDB. 3. Browser calls `POST /api/uploads` once for each file being started.
4. Browser uploads missing chunks with a small concurrency pool. 4. Browser stores each returned `upload_id` and file handle in IndexedDB.
5. Browser calls `/complete` when all chunks are uploaded. 5. Browser uploads missing chunks with bounded file and chunk concurrency pools.
6. Browser calls `/complete` for each file when all of its chunks are uploaded.
### After Interruption ### After Interruption
@@ -239,7 +240,8 @@ Start with these defaults:
```text ```text
chunk size: 16 MiB chunk size: 16 MiB
concurrency: 3 file concurrency: 3
chunk concurrency per file: 3
max retries per chunk: 5 max retries per chunk: 5
``` ```
+8 -1
View File
@@ -24,7 +24,7 @@ upl
Browser UI Browser UI
static/index.html upload tool markup static/index.html upload tool markup
static/styles.css responsive tool styling static/styles.css responsive tool styling
static/app.js upload scheduler, retries, and browser resume state static/app.js multi-file scheduler, retries, and browser resume state
Deployment Deployment
deploy/nginx/ nginx reverse proxy example deploy/nginx/ nginx reverse proxy example
scripts/ reusable local smoke tests scripts/ reusable local smoke tests
@@ -62,6 +62,13 @@ just run
`just check` also syntax-checks the static browser JavaScript with `node`. `just check` also syntax-checks the static browser JavaScript with `node`.
## Browser Uploads
The browser UI accepts multiple selected files. `Start all` runs up to three
file uploads at the same time, and each file still uploads up to three chunks
concurrently. Every selected file keeps its own upload id, progress markers,
abort controller, retry state, and saved IndexedDB resume record.
## nginx ## nginx
Run `upl` on localhost and put nginx in front of it for TLS and access control: Run `upl` on localhost and put nginx in front of it for TLS and access control:
+4
View File
@@ -21,6 +21,8 @@ Keep this file as the reusable verification checklist while implementing
- `GET /api/uploads/:id` reports completed chunks from disk markers. - `GET /api/uploads/:id` reports completed chunks from disk markers.
- `POST /api/uploads/:id/complete` renames the verified temp upload file - `POST /api/uploads/:id/complete` renames the verified temp upload file
and removes staging data. and removes staging data.
- Parallel upload requests for separate files complete without crossing
bytes between temp upload files.
- `POST /api/uploads/:id/complete` rejects incomplete uploads. - `POST /api/uploads/:id/complete` rejects incomplete uploads.
- `POST /api/uploads/:id/complete` rejects tampered temp upload files. - `POST /api/uploads/:id/complete` rejects tampered temp upload files.
- `static/app.js` passes `node --check`. - `static/app.js` passes `node --check`.
@@ -38,6 +40,8 @@ deployment retests.
- Upload a small file in one pass. - Upload a small file in one pass.
- Upload a file larger than one chunk. - Upload a file larger than one chunk.
- Select multiple files and confirm several upload rows advance at the same
time.
- Kill the browser tab mid-upload and resume. - Kill the browser tab mid-upload and resume.
- Restart the Rust server mid-upload and resume. - Restart the Rust server mid-upload and resume.
- Interrupt the network and resume. - Interrupt the network and resume.
+471 -157
View File
@@ -1,37 +1,33 @@
const DB_NAME = "upl"; const DB_NAME = "upl";
const DB_VERSION = 1; const DB_VERSION = 1;
const STORE_NAME = "uploads"; const STORE_NAME = "uploads";
const CONCURRENCY = 3; const CHUNK_CONCURRENCY = 3;
const FILE_CONCURRENCY = 3;
const MAX_RETRIES = 5; const MAX_RETRIES = 5;
const BASE_RETRY_DELAY_MS = 500; const BASE_RETRY_DELAY_MS = 500;
const COMPLETE_EXISTS_MESSAGE = "complete file already exists"; const COMPLETE_EXISTS_MESSAGE = "complete file already exists";
const fileInput = document.querySelector("#file-input"); const fileInput = document.querySelector("#file-input");
const pickButton = document.querySelector("#pick-button"); const pickButton = document.querySelector("#pick-button");
const fileSummary = document.querySelector("#file-summary"); const uploadSection = document.querySelector("#upload-section");
const fileName = document.querySelector("#file-name"); const uploadList = document.querySelector("#upload-list");
const fileSize = document.querySelector("#file-size");
const startButton = document.querySelector("#start-button"); const startButton = document.querySelector("#start-button");
const pauseButton = document.querySelector("#pause-button"); const pauseButton = document.querySelector("#pause-button");
const resumeButton = document.querySelector("#resume-button"); const resumeButton = document.querySelector("#resume-button");
const eventLog = document.querySelector("#event-log"); const eventLog = document.querySelector("#event-log");
const progressBar = document.querySelector("#progress-bar");
const progressMeta = document.querySelector("#progress-meta");
const pendingSection = document.querySelector("#pending-section"); const pendingSection = document.querySelector("#pending-section");
const pendingList = document.querySelector("#pending-list"); const pendingList = document.querySelector("#pending-list");
const state = { const state = {
abortController: null,
completedChunks: new Set(),
file: null,
fileHandle: null,
pendingRecords: [], pendingRecords: [],
record: null,
resumeAfterReselect: null, resumeAfterReselect: null,
running: false, schedulerAbortController: null,
schedulerRunning: false,
uploadItems: [],
}; };
const dbReady = "indexedDB" in window ? openDatabase() : Promise.resolve(null); const dbReady = "indexedDB" in window ? openDatabase() : Promise.resolve(null);
let nextUploadItemId = 1;
let saveChain = Promise.resolve(); let saveChain = Promise.resolve();
function formatBytes(bytes) { function formatBytes(bytes) {
@@ -101,7 +97,9 @@ async function withStore(mode, callback) {
async function loadRecords() { async function loadRecords() {
const records = (await withStore("readonly", (store) => store.getAll())) ?? []; const records = (await withStore("readonly", (store) => store.getAll())) ?? [];
records.sort((left, right) => right.updated_at.localeCompare(left.updated_at)); records.sort((left, right) =>
(right.updated_at ?? "").localeCompare(left.updated_at ?? ""),
);
state.pendingRecords = records; state.pendingRecords = records;
renderPendingRecords(); renderPendingRecords();
} }
@@ -109,7 +107,7 @@ async function loadRecords() {
async function saveRecord(record) { async function saveRecord(record) {
const nextSave = saveChain.catch(() => null).then(() => writeRecord(record)); const nextSave = saveChain.catch(() => null).then(() => writeRecord(record));
saveChain = nextSave; saveChain = nextSave;
await nextSave; return nextSave;
} }
async function writeRecord(record) { async function writeRecord(record) {
@@ -125,23 +123,31 @@ async function writeRecord(record) {
await withStore("readwrite", (store) => store.put(storedRecord)); await withStore("readwrite", (store) => store.put(storedRecord));
log("Saved resume state without a reusable file handle."); log("Saved resume state without a reusable file handle.");
} }
state.record = storedRecord;
await loadRecords(); await loadRecords();
return storedRecord;
} }
async function deleteRecord(uploadId) { async function deleteRecord(uploadId) {
await withStore("readwrite", (store) => store.delete(uploadId)); await withStore("readwrite", (store) => store.delete(uploadId));
if (state.record?.upload_id === uploadId) {
state.record = null;
}
await loadRecords(); await loadRecords();
} }
function renderPendingRecords() { function renderPendingRecords() {
pendingList.replaceChildren(); pendingList.replaceChildren();
pendingSection.hidden = state.pendingRecords.length === 0;
for (const record of state.pendingRecords) { const activeUploadIds = new Set(
state.uploadItems
.map((item) => item.record?.upload_id)
.filter((uploadId) => Boolean(uploadId)),
);
const visibleRecords = state.pendingRecords.filter(
(record) => !activeUploadIds.has(record.upload_id),
);
pendingSection.hidden = visibleRecords.length === 0;
for (const record of visibleRecords) {
const item = document.createElement("li"); const item = document.createElement("li");
item.className = "pending-item"; item.className = "pending-item";
@@ -158,6 +164,7 @@ function renderPendingRecords() {
resume.type = "button"; resume.type = "button";
resume.className = "secondary"; resume.className = "secondary";
resume.textContent = isReadyToFinish(record) ? "Finish" : "Resume"; resume.textContent = isReadyToFinish(record) ? "Finish" : "Resume";
resume.disabled = state.schedulerRunning || !hasAvailableFileSlot();
resume.addEventListener("click", () => { resume.addEventListener("click", () => {
void resumePendingRecord(record); void resumePendingRecord(record);
}); });
@@ -166,6 +173,7 @@ function renderPendingRecords() {
remove.type = "button"; remove.type = "button";
remove.className = "danger"; remove.className = "danger";
remove.textContent = "Clear"; remove.textContent = "Clear";
remove.disabled = state.schedulerRunning;
remove.addEventListener("click", () => { remove.addEventListener("click", () => {
void deleteRecord(record.upload_id); void deleteRecord(record.upload_id);
}); });
@@ -176,31 +184,111 @@ function renderPendingRecords() {
} }
} }
function renderFile(file) { function renderUploadItems() {
if (!file) { uploadList.replaceChildren();
fileSummary.hidden = true; uploadSection.hidden = state.uploadItems.length === 0;
fileName.textContent = "";
fileSize.textContent = "";
return;
}
fileName.textContent = file.name; for (const item of state.uploadItems) {
fileSize.textContent = formatBytes(file.size); const row = document.createElement("li");
fileSummary.hidden = false; row.className = "upload-item";
const header = document.createElement("div");
header.className = "upload-item-header";
const meta = document.createElement("div");
meta.className = "upload-meta";
const title = document.createElement("strong");
title.textContent = item.file.name;
const detail = document.createElement("span");
detail.textContent = uploadItemDetail(item);
meta.append(title, detail);
const actions = document.createElement("div");
actions.className = "upload-item-actions";
const start = document.createElement("button");
start.type = "button";
start.textContent = uploadActionLabel(item);
start.disabled =
!canRunItem(item) || state.schedulerRunning || !hasAvailableFileSlot();
start.addEventListener("click", () => {
void runUploadItem(item);
});
const pause = document.createElement("button");
pause.type = "button";
pause.className = "secondary";
pause.textContent = "Pause";
pause.disabled = !item.running;
pause.addEventListener("click", () => {
item.abortController?.abort();
});
const remove = document.createElement("button");
remove.type = "button";
remove.className = "secondary";
remove.textContent = "Remove";
remove.disabled = item.running || item.queued;
remove.addEventListener("click", () => {
removeUploadItem(item);
});
actions.append(start, pause, remove);
header.append(meta, actions);
const progress = document.createElement("div");
progress.className = "upload-progress";
const progressWrap = document.createElement("div");
progressWrap.className = "progress-wrap";
progressWrap.setAttribute("aria-label", `${item.file.name} upload progress`);
const progressBar = document.createElement("div");
progressBar.className = "progress-bar";
progressBar.style.width = `${progressPercentage(
item.completedCount,
item.totalChunks,
)}%`;
const progressMeta = document.createElement("div");
progressMeta.className = "progress-meta";
progressMeta.textContent = `${item.completedCount} of ${item.totalChunks} chunks`;
progressWrap.append(progressBar);
progress.append(progressWrap, progressMeta);
row.append(header, progress);
uploadList.append(row);
}
} }
function renderButtons() { function renderButtons() {
startButton.disabled = !state.file || state.running || Boolean(state.record); const hasRunnable = state.uploadItems.some((item) => canRunItem(item));
pauseButton.disabled = !state.running; const hasRunnableResume = state.uploadItems.some(
resumeButton.disabled = !state.file || state.running || !state.record; (item) => item.record && canRunItem(item),
resumeButton.textContent = );
state.record && isReadyToFinish(state.record) ? "Finish" : "Resume"; const hasRunningOrQueued = state.uploadItems.some((item) => item.running || item.queued);
const hasFileSlot = hasAvailableFileSlot();
startButton.disabled = !hasRunnable || state.schedulerRunning || !hasFileSlot;
pauseButton.disabled = !hasRunningOrQueued;
resumeButton.disabled = !hasRunnableResume || state.schedulerRunning || !hasFileSlot;
} }
function updateProgress(completedCount, totalChunks) { function renderAll() {
const percentage = totalChunks === 0 ? 0 : (completedCount / totalChunks) * 100; renderUploadItems();
progressBar.style.width = `${percentage}%`; renderPendingRecords();
progressMeta.textContent = `${completedCount} of ${totalChunks} chunks`; renderButtons();
}
function progressPercentage(completedCount, totalChunks) {
if (totalChunks <= 0) {
return 0;
}
return Math.min(100, Math.max(0, (completedCount / totalChunks) * 100));
} }
function sameFile(record, file) { function sameFile(record, file) {
@@ -211,10 +299,34 @@ function sameFile(record, file) {
); );
} }
function sameUploadItemFile(item, file) {
return (
item.file.name === file.name &&
item.file.size === file.size &&
item.file.lastModified === file.lastModified
);
}
function findPendingRecord(file) { function findPendingRecord(file) {
return state.pendingRecords.find((record) => sameFile(record, file)) ?? null; return state.pendingRecords.find((record) => sameFile(record, file)) ?? null;
} }
function findUploadItem(file, record = null) {
return (
state.uploadItems.find((item) => {
if (item.finished || item.terminal) {
return false;
}
if (record?.upload_id && item.record?.upload_id === record.upload_id) {
return true;
}
return sameUploadItemFile(item, file);
}) ?? null
);
}
function completedChunkCount(record) { function completedChunkCount(record) {
return Math.min(record.completed_chunks ?? 0, record.total_chunks ?? 0); return Math.min(record.completed_chunks ?? 0, record.total_chunks ?? 0);
} }
@@ -224,6 +336,13 @@ function isReadyToFinish(record) {
return totalChunks === 0 || completedChunkCount(record) >= totalChunks; return totalChunks === 0 || completedChunkCount(record) >= totalChunks;
} }
function isUploadItemReadyToFinish(item) {
return (
Boolean(item.record) &&
(item.totalChunks === 0 || item.completedCount >= item.totalChunks)
);
}
function savedUploadDetail(record) { function savedUploadDetail(record) {
const totalChunks = record.total_chunks ?? 0; const totalChunks = record.total_chunks ?? 0;
const completedChunks = completedChunkCount(record); const completedChunks = completedChunkCount(record);
@@ -239,50 +358,149 @@ function savedUploadDetail(record) {
return `${formatBytes(record.size)} - ${completedChunks} of ${totalChunks} chunks uploaded`; return `${formatBytes(record.size)} - ${completedChunks} of ${totalChunks} chunks uploaded`;
} }
function clearSelection({ resetProgress = true } = {}) { function uploadItemDetail(item) {
state.file = null; return `${formatBytes(item.file.size)} - ${item.statusText}`;
state.fileHandle = null;
state.record = null;
state.completedChunks = new Set();
state.resumeAfterReselect = null;
fileInput.value = "";
renderFile(null);
if (resetProgress) {
updateProgress(0, 0);
}
renderButtons();
} }
async function selectFile(file, fileHandle = null, record = null) { function uploadActionLabel(item) {
if (record && !sameFile(record, file)) { if (item.finished) {
log("Selected file does not match the pending upload."); return "Done";
}
if (!item.record) {
return "Start";
}
return isUploadItemReadyToFinish(item) ? "Finish" : "Resume";
}
function initialUploadStatus(record) {
if (!record) {
return "Ready to create an upload record.";
}
if (isReadyToFinish(record)) {
return "Ready to finish saved upload.";
}
return "Ready to resume upload.";
}
function canRunItem(item) {
return (
Boolean(item.file) &&
!item.running &&
!item.queued &&
!item.finished &&
!item.terminal
);
}
function runningFileCount() {
return state.uploadItems.filter((item) => item.running).length;
}
function hasAvailableFileSlot() {
return runningFileCount() < FILE_CONCURRENCY;
}
function setItemProgress(item, completedCount, totalChunks) {
item.totalChunks = Math.max(0, totalChunks);
item.completedCount =
item.totalChunks === 0
? 0
: Math.min(Math.max(0, completedCount), item.totalChunks);
}
async function selectFiles(files, fileHandles = []) {
const selectedFiles = Array.from(files);
if (selectedFiles.length === 0) {
log("Choose files to begin.");
renderButtons();
return; return;
} }
state.file = file; const resumeRecord = state.resumeAfterReselect;
state.fileHandle = fileHandle; let matchedResumeRecord = false;
state.record = record ?? findPendingRecord(file); let addedCount = 0;
state.completedChunks = new Set();
renderFile(file); for (const [index, file] of selectedFiles.entries()) {
updateProgress(state.record?.completed_chunks ?? 0, state.record?.total_chunks ?? 0); let record = findPendingRecord(file);
renderButtons(); if (resumeRecord && sameFile(resumeRecord, file)) {
record = resumeRecord;
if (!state.record) { matchedResumeRecord = true;
log("Ready to create an upload record.");
} else if (isReadyToFinish(state.record)) {
log("Ready to finish saved upload.");
} else {
log("Ready to resume upload.");
} }
const previousCount = state.uploadItems.length;
addUploadItem(file, fileHandles[index] ?? null, record);
if (state.uploadItems.length > previousCount) {
addedCount += 1;
}
}
if (resumeRecord && !matchedResumeRecord) {
log("Selected files did not include the pending upload file.");
}
state.resumeAfterReselect = null;
fileInput.value = "";
renderAll();
if (addedCount === 1) {
log("Ready to upload 1 file.");
} else if (addedCount > 1) {
log(`Ready to upload ${addedCount} files.`);
}
}
function addUploadItem(file, fileHandle = null, record = null) {
if (record && !sameFile(record, file)) {
log("Selected file does not match the pending upload.");
return null;
}
const existingItem = findUploadItem(file, record);
if (existingItem) {
log(`${file.name} is already selected.`);
return existingItem;
}
const item = {
abortController: null,
completedChunks: new Set(),
completedCount: record ? completedChunkCount(record) : 0,
file,
fileHandle,
finished: false,
id: nextUploadItemId,
queued: false,
record,
running: false,
statusText: initialUploadStatus(record),
terminal: false,
totalChunks: record?.total_chunks ?? 0,
};
nextUploadItemId += 1;
state.uploadItems.push(item);
return item;
}
function removeUploadItem(item) {
if (item.running || item.queued) {
return;
}
state.uploadItems = state.uploadItems.filter((candidate) => candidate.id !== item.id);
renderAll();
} }
async function pickFile() { async function pickFile() {
if ("showOpenFilePicker" in window) { if ("showOpenFilePicker" in window) {
try { try {
const [handle] = await window.showOpenFilePicker({ multiple: false }); const handles = await window.showOpenFilePicker({ multiple: true });
const file = await handle.getFile(); const files = await Promise.all(handles.map((handle) => handle.getFile()));
await selectFile(file, handle); await selectFiles(files, handles);
return; return;
} catch (error) { } catch (error) {
if (isAbortError(error)) { if (isAbortError(error)) {
@@ -296,18 +514,7 @@ async function pickFile() {
} }
fileInput.addEventListener("change", () => { fileInput.addEventListener("change", () => {
const [file] = fileInput.files; void selectFiles(fileInput.files);
if (!file) {
renderFile(null);
renderButtons();
log("Choose a file to begin.");
return;
}
const record = state.resumeAfterReselect ?? findPendingRecord(file);
state.resumeAfterReselect = null;
void selectFile(file, null, record);
}); });
pickButton.addEventListener("click", () => { pickButton.addEventListener("click", () => {
@@ -315,21 +522,19 @@ pickButton.addEventListener("click", () => {
}); });
startButton.addEventListener("click", () => { startButton.addEventListener("click", () => {
void runUpload(); void runUploadItems(state.uploadItems);
}); });
pauseButton.addEventListener("click", () => { pauseButton.addEventListener("click", () => {
if (state.abortController) { pauseUploads();
state.abortController.abort();
}
}); });
resumeButton.addEventListener("click", () => { resumeButton.addEventListener("click", () => {
void runUpload(); void runUploadItems(state.uploadItems.filter((item) => item.record));
}); });
async function resumePendingRecord(record) { async function resumePendingRecord(record) {
if (state.running) { if (state.schedulerRunning) {
return; return;
} }
@@ -337,8 +542,11 @@ async function resumePendingRecord(record) {
const granted = await requestFileHandlePermission(record.file_handle); const granted = await requestFileHandlePermission(record.file_handle);
if (granted) { if (granted) {
const file = await record.file_handle.getFile(); const file = await record.file_handle.getFile();
await selectFile(file, record.file_handle, record); const item = addUploadItem(file, record.file_handle, record);
await runUpload(); renderAll();
if (item) {
await runUploadItem(item);
}
return; return;
} }
} }
@@ -365,127 +573,224 @@ async function requestFileHandlePermission(handle) {
return (await handle.requestPermission(options)) === "granted"; return (await handle.requestPermission(options)) === "granted";
} }
async function runUpload() { async function runUploadItems(items) {
if (!state.file || state.running) { if (state.schedulerRunning) {
return;
}
const runnableItems = items.filter((item) => canRunItem(item));
const availableSlots = FILE_CONCURRENCY - runningFileCount();
if (runnableItems.length === 0 || availableSlots <= 0) {
return; return;
} }
const controller = new AbortController(); const controller = new AbortController();
state.abortController = controller; state.schedulerAbortController = controller;
state.running = true; state.schedulerRunning = true;
renderButtons();
for (const item of runnableItems) {
item.queued = true;
item.statusText = "Queued.";
}
renderAll();
try { try {
if (!state.record) { await runPool(
await createUploadRecord(); runnableItems,
async (item) => {
throwIfAborted(controller.signal);
if (!item.queued) {
return;
}
await runUploadItem(item);
},
availableSlots,
controller.signal,
);
} catch (error) {
if (!isAbortError(error)) {
log(`Upload scheduler failed: ${error.message}`);
}
} finally {
for (const item of runnableItems) {
if (item.queued) {
item.queued = false;
item.statusText = "Paused.";
}
} }
const progress = await fetchJson(`/api/uploads/${state.record.upload_id}`, { if (state.schedulerAbortController === controller) {
state.schedulerAbortController = null;
}
state.schedulerRunning = false;
renderAll();
}
}
function pauseUploads() {
state.schedulerAbortController?.abort();
for (const item of state.uploadItems) {
if (item.running) {
item.abortController?.abort();
} else if (item.queued) {
item.queued = false;
item.statusText = "Paused.";
}
}
renderAll();
}
async function runUploadItem(item) {
if (
item.running ||
item.finished ||
item.terminal ||
(!item.queued && !hasAvailableFileSlot()) ||
(!item.queued && !canRunItem(item))
) {
return;
}
const controller = new AbortController();
item.abortController = controller;
item.queued = false;
item.running = true;
item.statusText = item.record
? "Checking saved progress."
: "Creating upload record.";
renderAll();
try {
if (!item.record) {
await createUploadRecord(item, controller.signal);
}
const progress = await fetchJson(`/api/uploads/${item.record.upload_id}`, {
signal: controller.signal, signal: controller.signal,
}); });
state.completedChunks = new Set(progress.completed_chunks); item.completedChunks = new Set(progress.completed_chunks);
await saveRecord({ setItemProgress(item, item.completedChunks.size, progress.total_chunks);
...state.record, item.record = await saveRecord({
completed_chunks: state.completedChunks.size, ...item.record,
completed_chunks: item.completedCount,
chunk_size: progress.chunk_size, chunk_size: progress.chunk_size,
total_chunks: progress.total_chunks, total_chunks: progress.total_chunks,
}); });
updateProgress(state.completedChunks.size, progress.total_chunks); renderAll();
const missingChunks = buildMissingChunkList(progress.total_chunks, state.completedChunks); const missingChunks = buildMissingChunkList(progress.total_chunks, item.completedChunks);
log( item.statusText =
missingChunks.length === 0 missingChunks.length === 0
? "All chunks already uploaded." ? "All chunks already uploaded."
: `Uploading ${missingChunks.length} missing chunks.`, : `Uploading ${missingChunks.length} missing chunks.`;
); renderAll();
await runPool(missingChunks, (index) => await runPool(
uploadChunkWithRetry(index, progress.chunk_size, progress.total_chunks, controller.signal), missingChunks,
(index) =>
uploadChunkWithRetry(
item,
index,
progress.chunk_size,
progress.total_chunks,
controller.signal,
),
CHUNK_CONCURRENCY,
controller.signal,
); );
if (controller.signal.aborted) { if (controller.signal.aborted) {
return; return;
} }
log("Completing upload."); item.statusText = "Completing upload.";
const uploadId = state.record.upload_id; renderAll();
const uploadId = item.record.upload_id;
const complete = await fetchJson(`/api/uploads/${uploadId}/complete`, { const complete = await fetchJson(`/api/uploads/${uploadId}/complete`, {
method: "POST", method: "POST",
signal: controller.signal, signal: controller.signal,
}); });
updateProgress(progress.total_chunks, progress.total_chunks); setItemProgress(item, progress.total_chunks, progress.total_chunks);
log(`Complete: ${complete.file_path}`); item.finished = true;
item.statusText = `Complete: ${complete.file_path}`;
await deleteRecord(uploadId); await deleteRecord(uploadId);
clearSelection({ resetProgress: false });
} catch (error) { } catch (error) {
if (controller.signal.aborted || isAbortError(error)) { if (controller.signal.aborted || isAbortError(error)) {
log("Upload paused."); item.statusText = "Paused.";
} else if (await handleTerminalUploadError(error)) { } else if (await handleTerminalUploadError(item, error)) {
controller.abort(); controller.abort();
} else { } else {
controller.abort(); controller.abort();
log(`Upload failed: ${error.message}`); item.statusText = `Upload failed: ${error.message}`;
} }
} finally { } finally {
if (state.abortController === controller) { if (item.abortController === controller) {
state.abortController = null; item.abortController = null;
} }
state.running = false; item.running = false;
renderButtons(); renderAll();
} }
} }
async function handleTerminalUploadError(error) { async function handleTerminalUploadError(item, error) {
if (!state.record || typeof error.status !== "number") { if (!item.record || typeof error.status !== "number") {
return false; return false;
} }
const uploadId = state.record.upload_id; const uploadId = item.record.upload_id;
if (error.status === 404) { if (error.status === 404) {
await deleteRecord(uploadId); await deleteRecord(uploadId);
clearSelection(); item.record = null;
log("Saved upload progress no longer exists on the server."); item.completedChunks = new Set();
setItemProgress(item, 0, 0);
item.statusText = "Saved upload progress no longer exists. Start again.";
log(`${item.file.name}: saved upload progress no longer exists on the server.`);
return true; return true;
} }
if (error.status === 409 && error.message === COMPLETE_EXISTS_MESSAGE) { if (error.status === 409 && error.message === COMPLETE_EXISTS_MESSAGE) {
await deleteRecord(uploadId); await deleteRecord(uploadId);
clearSelection(); item.terminal = true;
log("The completed file already exists on the server. Cleared saved upload progress."); item.statusText = "Complete file already exists on the server.";
log(`${item.file.name}: complete file already exists on the server.`);
return true; return true;
} }
return false; return false;
} }
async function createUploadRecord() { async function createUploadRecord(item, signal) {
const response = await fetchJson("/api/uploads", { const response = await fetchJson("/api/uploads", {
method: "POST", method: "POST",
headers: { "Content-Type": "application/json" }, headers: { "Content-Type": "application/json" },
body: JSON.stringify({ body: JSON.stringify({
name: state.file.name, name: item.file.name,
size: state.file.size, size: item.file.size,
last_modified: state.file.lastModified, last_modified: item.file.lastModified,
}), }),
signal,
}); });
const record = { const record = {
upload_id: response.upload_id, upload_id: response.upload_id,
name: state.file.name, name: item.file.name,
size: state.file.size, size: item.file.size,
last_modified: state.file.lastModified, last_modified: item.file.lastModified,
chunk_size: response.chunk_size, chunk_size: response.chunk_size,
total_chunks: response.total_chunks, total_chunks: response.total_chunks,
completed_chunks: 0, completed_chunks: 0,
file_handle: state.fileHandle, file_handle: item.fileHandle,
updated_at: new Date().toISOString(), updated_at: new Date().toISOString(),
}; };
await saveRecord(record); item.record = await saveRecord(record);
updateProgress(0, response.total_chunks); setItemProgress(item, 0, response.total_chunks);
log("Upload record created."); item.statusText = "Upload record created.";
renderAll();
} }
function buildMissingChunkList(totalChunks, completedChunks) { function buildMissingChunkList(totalChunks, completedChunks) {
@@ -498,18 +803,19 @@ function buildMissingChunkList(totalChunks, completedChunks) {
return missing; return missing;
} }
async function uploadChunkWithRetry(index, chunkSize, totalChunks, signal) { async function uploadChunkWithRetry(item, index, chunkSize, totalChunks, signal) {
for (let attempt = 0; attempt <= MAX_RETRIES; attempt += 1) { for (let attempt = 0; attempt <= MAX_RETRIES; attempt += 1) {
throwIfAborted(signal); throwIfAborted(signal);
try { try {
await uploadChunk(index, chunkSize, signal); await uploadChunk(item, index, chunkSize, signal);
state.completedChunks.add(index); item.completedChunks.add(index);
updateProgress(state.completedChunks.size, totalChunks); setItemProgress(item, item.completedChunks.size, totalChunks);
await saveRecord({ item.record = await saveRecord({
...state.record, ...item.record,
completed_chunks: state.completedChunks.size, completed_chunks: item.completedCount,
}); });
renderAll();
return; return;
} catch (error) { } catch (error) {
if (isAbortError(error) || attempt === MAX_RETRIES) { if (isAbortError(error) || attempt === MAX_RETRIES) {
@@ -517,34 +823,42 @@ async function uploadChunkWithRetry(index, chunkSize, totalChunks, signal) {
} }
const delayMs = BASE_RETRY_DELAY_MS * 2 ** attempt; const delayMs = BASE_RETRY_DELAY_MS * 2 ** attempt;
log(`Retrying chunk ${index} after ${delayMs} ms.`); item.statusText = `Retrying chunk ${index} after ${delayMs} ms.`;
renderAll();
await delay(delayMs, signal); await delay(delayMs, signal);
} }
} }
} }
async function uploadChunk(index, chunkSize, signal) { async function uploadChunk(item, index, chunkSize, signal) {
const start = index * chunkSize; const start = index * chunkSize;
const end = Math.min(state.file.size, start + chunkSize); const end = Math.min(item.file.size, start + chunkSize);
const body = state.file.slice(start, end); const body = item.file.slice(start, end);
const response = await fetch(`/api/uploads/${state.record.upload_id}/chunks/${index}`, { const response = await fetch(
`/api/uploads/${item.record.upload_id}/chunks/${index}`,
{
method: "PUT", method: "PUT",
headers: { "Content-Type": "application/octet-stream" }, headers: { "Content-Type": "application/octet-stream" },
body, body,
signal, signal,
}); },
);
if (!response.ok) { if (!response.ok) {
throw new Error(await responseError(response)); throw new ApiRequestError(await responseError(response), response.status);
} }
} }
async function runPool(items, worker) { async function runPool(items, worker, concurrency, signal = null) {
let nextIndex = 0; let nextIndex = 0;
const workers = Array.from( const workers = Array.from(
{ length: Math.min(CONCURRENCY, items.length) }, { length: Math.min(concurrency, items.length) },
async () => { async () => {
while (nextIndex < items.length) { while (nextIndex < items.length) {
if (signal) {
throwIfAborted(signal);
}
const item = items[nextIndex]; const item = items[nextIndex];
nextIndex += 1; nextIndex += 1;
await worker(item); await worker(item);
@@ -612,7 +926,7 @@ function isAbortError(error) {
async function initialize() { async function initialize() {
await loadRecords(); await loadRecords();
renderButtons(); renderAll();
} }
void initialize(); void initialize();
+11 -16
View File
@@ -18,33 +18,28 @@
</div> </div>
<div class="file-picker"> <div class="file-picker">
<button id="pick-button" type="button">Choose file</button> <button id="pick-button" type="button">Choose files</button>
<input id="file-input" type="file"> <input id="file-input" type="file" multiple>
</div> </div>
<div class="file-summary" id="file-summary" hidden>
<strong id="file-name"></strong>
<span id="file-size"></span>
</div>
<div class="progress-wrap" aria-label="Upload progress">
<div class="progress-bar" id="progress-bar"></div>
</div>
<div class="progress-meta" id="progress-meta">0 of 0 chunks</div>
<div class="actions"> <div class="actions">
<button id="start-button" type="button" disabled>Start</button> <button id="start-button" type="button" disabled>Start all</button>
<button id="pause-button" type="button" disabled>Pause</button> <button id="pause-button" type="button" disabled>Pause all</button>
<button id="resume-button" type="button" disabled>Resume</button> <button id="resume-button" type="button" disabled>Resume all</button>
</div> </div>
<section class="upload-section" id="upload-section" hidden>
<h2>Selected uploads</h2>
<ul class="upload-list" id="upload-list"></ul>
</section>
<section class="pending-section" id="pending-section" hidden> <section class="pending-section" id="pending-section" hidden>
<h2>Saved upload progress</h2> <h2>Saved upload progress</h2>
<ul class="pending-list" id="pending-list"></ul> <ul class="pending-list" id="pending-list"></ul>
</section> </section>
<ol class="event-log" id="event-log" aria-live="polite"> <ol class="event-log" id="event-log" aria-live="polite">
<li>Choose a file to begin.</li> <li>Choose files to begin.</li>
</ol> </ol>
</section> </section>
</main> </main>
+52 -6
View File
@@ -88,16 +88,48 @@ h1 {
clip: rect(0, 0, 0, 0); clip: rect(0, 0, 0, 0);
} }
.file-summary { .upload-section {
display: grid; display: grid;
gap: 4px; gap: 12px;
padding: 14px; }
.upload-list {
display: grid;
gap: 12px;
margin: 0;
padding: 0;
list-style: none;
}
.upload-item {
display: grid;
gap: 12px;
padding: 12px;
border: 1px solid var(--line); border: 1px solid var(--line);
border-radius: 8px; border-radius: 8px;
} }
.file-summary span { .upload-item-header {
display: grid;
grid-template-columns: minmax(0, 1fr) auto;
align-items: start;
gap: 12px;
}
.upload-meta {
display: grid;
gap: 4px;
min-width: 0;
}
.upload-meta span {
color: var(--muted); color: var(--muted);
font-size: 0.875rem;
}
.upload-progress {
display: grid;
gap: 6px;
} }
.progress-wrap { .progress-wrap {
@@ -115,7 +147,6 @@ h1 {
} }
.progress-meta { .progress-meta {
margin-top: -12px;
color: var(--muted); color: var(--muted);
font-size: 0.875rem; font-size: 0.875rem;
} }
@@ -126,6 +157,13 @@ h1 {
gap: 10px; gap: 10px;
} }
.upload-item-actions {
display: flex;
flex-wrap: wrap;
justify-content: flex-end;
gap: 8px;
}
button { button {
min-width: 96px; min-width: 96px;
min-height: 40px; min-height: 40px;
@@ -184,7 +222,7 @@ h2 {
} }
.pending-item strong, .pending-item strong,
.file-summary strong { .upload-meta strong {
overflow-wrap: anywhere; overflow-wrap: anywhere;
} }
@@ -232,4 +270,12 @@ h2 {
.pending-item { .pending-item {
grid-template-columns: 1fr; grid-template-columns: 1fr;
} }
.upload-item-header {
grid-template-columns: 1fr;
}
.upload-item-actions {
justify-content: flex-start;
}
} }
+74
View File
@@ -66,6 +66,80 @@ async fn assembles_completed_upload() -> Result<(), Box<dyn std::error::Error>>
Ok(()) Ok(())
} }
#[tokio::test]
async fn parallel_uploads_keep_bytes_separate() -> Result<(), Box<dyn std::error::Error>> {
let temp_dir = TempDir::new()?;
let app = test_app(temp_dir.path());
let chunk_size = usize::try_from(CHUNK_SIZE)?;
let left_upload = create_upload(&app, "left.bin", CHUNK_SIZE + 4).await?;
let right_upload = create_upload(&app, "right.bin", CHUNK_SIZE + 5).await?;
let mut expected_left = vec![b'l'; chunk_size];
expected_left.extend_from_slice(b"eft!");
let mut expected_right = vec![b'r'; chunk_size];
expected_right.extend_from_slice(b"ight!");
let left_first = chunk_request(
&left_upload.upload_id,
0,
expected_left[..chunk_size].to_vec(),
)?;
let left_final = chunk_request(
&left_upload.upload_id,
1,
expected_left[chunk_size..].to_vec(),
)?;
let right_first = chunk_request(
&right_upload.upload_id,
0,
expected_right[..chunk_size].to_vec(),
)?;
let right_final = chunk_request(
&right_upload.upload_id,
1,
expected_right[chunk_size..].to_vec(),
)?;
let (left_first, right_first, left_final, right_final) = tokio::join!(
app.clone().oneshot(left_first),
app.clone().oneshot(right_first),
app.clone().oneshot(left_final),
app.clone().oneshot(right_final),
);
assert_eq!(left_first?.status(), StatusCode::NO_CONTENT);
assert_eq!(right_first?.status(), StatusCode::NO_CONTENT);
assert_eq!(left_final?.status(), StatusCode::NO_CONTENT);
assert_eq!(right_final?.status(), StatusCode::NO_CONTENT);
let left_complete = empty_request(
Method::POST,
&format!("/api/uploads/{}/complete", left_upload.upload_id),
)?;
let right_complete = empty_request(
Method::POST,
&format!("/api/uploads/{}/complete", right_upload.upload_id),
)?;
let (left_complete, right_complete) = tokio::join!(
app.clone().oneshot(left_complete),
app.clone().oneshot(right_complete),
);
assert_eq!(left_complete?.status(), StatusCode::OK);
assert_eq!(right_complete?.status(), StatusCode::OK);
assert_eq!(
tokio::fs::read(temp_dir.path().join("complete").join("left.bin")).await?,
expected_left
);
assert_eq!(
tokio::fs::read(temp_dir.path().join("complete").join("right.bin")).await?,
expected_right
);
Ok(())
}
#[tokio::test] #[tokio::test]
async fn rejects_incomplete_upload() -> Result<(), Box<dyn std::error::Error>> { async fn rejects_incomplete_upload() -> Result<(), Box<dyn std::error::Error>> {
let temp_dir = TempDir::new()?; let temp_dir = TempDir::new()?;
+2 -1
View File
@@ -22,7 +22,8 @@ async fn serves_index_page() -> Result<(), Box<dyn std::error::Error>> {
let body = String::from_utf8(body.to_vec())?; let body = String::from_utf8(body.to_vec())?;
assert!(body.contains("<title>upl</title>")); assert!(body.contains("<title>upl</title>"));
assert!(body.contains("Choose file")); assert!(body.contains("Choose files"));
assert!(body.contains("Selected uploads"));
assert!(body.contains("Saved upload progress")); assert!(body.contains("Saved upload progress"));
assert!(!body.contains("Server online")); assert!(!body.contains("Server online"));