Faster first library scan: bounded concurrency and a 32 KB first read
An uncached scan probed one image at a time and read 256 KB of each. On a NAS that is bound by per-file latency (the server/Synology: ~20 ms per image, ~100 min for 298k images). Now chapter folders of a book and their images are probed SCAN_CONCURRENCY (default 8, 1-64) at a time and assembled in folder order, and imageSize() reads 32 KB first, then 256 KB, then the whole file: a prefix gives either no size or the same size, so results do not change. Tests: every format and read stage equals the whole-file answer; concurrency 1 and 8 build the same index and cache; the limiter keeps its bound and per-task errors. (BL-0043) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d6254a8be5
commit
5e14e54286
7 files changed
+169
-22
No files matched your search
+4
-2
@@ -148,7 +148,9 @@ Everything is plain data, and the whole index lives in memory.
|
||||
|
||||
## 3. Image sizes
|
||||
|
||||
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 256 KB** and parses the header:
|
||||
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 32 KB** and parses the header; only if the size isn't there yet does it read on to 256 KB (and, for JPEG, the whole file). Parsing a prefix either finds nothing or the same size as the whole file, so the small first read never changes a result.
|
||||
|
||||
Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on the server against a Synology (NFS) the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.
|
||||
|
||||
| Format | Where the size is |
|
||||
|--------|-------------------|
|
||||
@@ -156,7 +158,7 @@ The reader lays pages out before they load, so it needs every image's size. Inst
|
||||
| GIF | logical screen (bytes 6–9) |
|
||||
| BMP | DIB header (bytes 18–25) |
|
||||
| WebP | `VP8 ` / `VP8L` / `VP8X` chunk |
|
||||
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 256 KB, the whole file is read once |
|
||||
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 32 KB, then 256 KB, the whole file is read once |
|
||||
| AVIF / HEIF | first `ispe` box |
|
||||
|
||||
- If the size can't be read, it is stored as `0×0`. The reader then assumes 1000×1414 (2000×1414 for `double`) and corrects the layout when the image arrives.
|
||||
|
||||
Reference in new issue
Block a user