Faster first library scan: bounded concurrency and a 32 KB first read

An uncached scan probed one image at a time and read 256 KB of each. On
a NAS that is bound by per-file latency (the server/Synology: ~20 ms per
image, ~100 min for 298k images). Now chapter folders of a book and
their images are probed SCAN_CONCURRENCY (default 8, 1-64) at a time
and assembled in folder order, and imageSize() reads 32 KB first, then
256 KB, then the whole file: a prefix gives either no size or the same
size, so results do not change. Tests: every format and read stage
equals the whole-file answer; concurrency 1 and 8 build the same index
and cache; the limiter keeps its bound and per-task errors. (BL-0043)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed 2026-10-22 20:07:29 +02:00
1 parent d6254a8be5
commit 5e14e54286
7 files changed
+169 -22

No files matched your search

+1 -1
View File
@@ -364,7 +364,7 @@ docker compose up -d --build
| Aspect | Behaviour |
|--------|-----------|
| Library size | The index keeps a small object per page (file name, size, tags), in the order of a few hundred bytes. Hundreds of thousands of pages fit in a modest container. Unchanged chapters cost one `stat` + one small read per scan |
| First scan | Reads ≤ 256 KB per image once. Later scans use the cache |
| First scan | Reads 32 KB per image once (≤ 256 KB, or the whole file for JPEGs with huge EXIF/ICC blocks), `SCAN_CONCURRENCY` (8) at a time. Later scans use the cache |
| User data | `db.json` is rewritten as a whole, at most about twice per second while people read. Comfortable up to a few MB (a household or small group, thousands of chapters read) |
| Concurrency | One process, one writer. No clustering or horizontal scaling by design |
| Images | Served as stored. No resizing or thumbnails, so large covers make the library page heavy |
+4 -2
View File
@@ -148,7 +148,9 @@ Everything is plain data, and the whole index lives in memory.
## 3. Image sizes
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 256 KB** and parses the header:
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 32 KB** and parses the header; only if the size isn't there yet does it read on to 256 KB (and, for JPEG, the whole file). Parsing a prefix either finds nothing or the same size as the whole file, so the small first read never changes a result.
Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on the server against a Synology (NFS) the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.
| Format | Where the size is |
|--------|-------------------|
@@ -156,7 +158,7 @@ The reader lays pages out before they load, so it needs every image's size. Inst
| GIF | logical screen (bytes 6–9) |
| BMP | DIB header (bytes 18–25) |
| WebP | `VP8 ` / `VP8L` / `VP8X` chunk |
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 256 KB, the whole file is read once |
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 32 KB, then 256 KB, the whole file is read once |
| AVIF / HEIF | first `ispe` box |
- If the size can't be read, it is stored as `0×0`. The reader then assumes 1000×1414 (2000×1414 for `double`) and corrects the layout when the image arrives.