Pro 2.x legacy. 6 docs (3 sch sheets + 3 PCBs), 0.2 MB plain. The
deprecated 主控板V1 sch/pcb pair is correctly skipped (filter via
ticket.schematics/pcbs keys, see crawler commit c3cac97).
batch-50 success rate is now 50/50 (was 49/50).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run on dev1 (Guangzhou) for the latency advantage. Walltime 3:41 vs
Singapore-estimated 1-2h (~30x speedup, mostly from image.lceda.cn
RTT going from 263ms to 2.6ms).
Pro 25: 24 ok + 1 fail (Super Dial 7f7565ef11 — Pro 2.x legacy
schematic/lists 401, separate cookie-perm issue)
611 docs, 31 MB total
Std 25: 25 ok, 97 docs, 74 MB total
Combined: 49/50 success, 708 docs, 105 MB new disk usage
--max-source-mb 200 cap was not tripped; the 25 Pro candidates are all
under 10 MB, so the 481 MB X86-board outlier from the original sample
was not representative.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three crawler ergonomics for batch operations:
--no-cover Skip cover image download. For scan-only modes (license/meta
scrape) this drops ~1.3s/project and avoids slow-CDN hangs.
--concurrency N ThreadPoolExecutor wrapping the per-project loop. Default
1 = serial (current behavior). Anonymous endpoints tolerate
5+ comfortably; output uses a print lock for readable
interleaved progress. fetch_cover plumbs through crawl_one.
Drop cross-host sleep #1: in crawl_one between detail HTML (oshwhub.com)
and cover image (image.lceda.cn). Different hosts — sleep was unnecessary.
Saves ~1s/project. Sleep #2 (post-cover, before next iteration) stays — it
gates the next oshwhub.com hit.
download_to gains max_seconds wall budget (default 60s, cover uses 15s).
Defends against pathologically slow CDN connections — observed 10 KB/s
on image.lceda.cn for one project, would have hung 6+ min on a 3.6 MB
cover otherwise. httpx default timeout resets per chunk, so streaming
downloads need an external wall-clock guard.
batch-50 Step 1 (license/meta scrape) shipped:
50/50 candidates have metadata.json + license recorded
License distribution: GPL 3.0 32, Public Domain 6, NC variants 8,
CERN-OHL 1, MIT 1, CC BY 3.0 1
Forge-friendly (non-NC): 41/50 (82%)
Declared attachments: 180 files / 2.36 GB (median 18 MB/proj, max 304 MB)
Walltime: 3min 26s for 28 projects at concurrency=5 (server-side
HTML render bound, not sleep-bound)
One orphan partial cover (a670e60a...) cleaned up — leftover from the
first aborted run before the timeout fix landed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
