Knowit
c6fd111d6d
Three crawler ergonomics for batch operations:
--no-cover Skip cover image download. For scan-only modes (license/meta
scrape) this drops ~1.3s/project and avoids slow-CDN hangs.
--concurrency N ThreadPoolExecutor wrapping the per-project loop. Default
1 = serial (current behavior). Anonymous endpoints tolerate
5+ comfortably; output uses a print lock for readable
interleaved progress. fetch_cover plumbs through crawl_one.
Drop cross-host sleep #1: in crawl_one between detail HTML (oshwhub.com)
and cover image (image.lceda.cn). Different hosts — sleep was unnecessary.
Saves ~1s/project. Sleep #2 (post-cover, before next iteration) stays — it
gates the next oshwhub.com hit.
download_to gains max_seconds wall budget (default 60s, cover uses 15s).
Defends against pathologically slow CDN connections — observed 10 KB/s
on image.lceda.cn for one project, would have hung 6+ min on a 3.6 MB
cover otherwise. httpx default timeout resets per chunk, so streaming
downloads need an external wall-clock guard.
batch-50 Step 1 (license/meta scrape) shipped:
50/50 candidates have metadata.json + license recorded
License distribution: GPL 3.0 32, Public Domain 6, NC variants 8,
CERN-OHL 1, MIT 1, CC BY 3.0 1
Forge-friendly (non-NC): 41/50 (82%)
Declared attachments: 180 files / 2.36 GB (median 18 MB/proj, max 304 MB)
Walltime: 3min 26s for 28 projects at concurrency=5 (server-side
HTML render bound, not sleep-bound)
One orphan partial cover (a670e60a...) cleaned up — leftover from the
first aborted run before the timeout fix landed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
61 lines
2.6 KiB
JSON
61 lines
2.6 KiB
JSON
{
|
|
"detail_url": "https://oshwhub.com/jie326513988/SDka-mo-shui-ping-yue-du-qi",
|
|
"cover_url": "https://image.lceda.cn/pullimage/i3Ii5diadqRhKUS92NFiD50e3lLbRdylqXlg0ay3.jpeg",
|
|
"attachments": [
|
|
{
|
|
"name": "SDCardFormatterv5_WinEN.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/2/HZGnA5mXc4LTgZIQwLX3Sc10JZD9dgSf2Hy4Tml2.zip",
|
|
"original_id": "60de0ca8ff0c428ca78e7256f559e7d3"
|
|
},
|
|
{
|
|
"name": "2.9寸墨水屏打板资料-V2.32.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/2/ve3jKgj45OGqqPxDO7dMqBjiwtCdmesombqoX1Qt.zip",
|
|
"original_id": "90e843c39b824090aa4361d892d5460f"
|
|
},
|
|
{
|
|
"name": "studio_video_1646578755868.mp4",
|
|
"url": "https://image.lceda.cn/attachments/2022/3/O7rolhyNjPElI1tkWLb9E9i6TOgnW8xsZBc7aKDF.mp4",
|
|
"original_id": "602ecfaf95774961bc523383a7f97442"
|
|
},
|
|
{
|
|
"name": "V2.41打板资料.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/Vl0kitUSAeYj59TSAdxlOFMTiJX4ehjHOz36UGhy.zip",
|
|
"original_id": "0bfea258a84c41599be29d0d1b98e2b7"
|
|
},
|
|
{
|
|
"name": "SD普壳-3mm厚电池.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/4AAHKHjGwc7TupmQZaR3YpxC0sQma3AZK5szmCYb.zip",
|
|
"original_id": "e35396798c09454eaf39a5ad1c7e107e"
|
|
},
|
|
{
|
|
"name": "SD普壳-5mm厚电池.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/YIdHOTwE2lDuOjqKZucil0pIp70KmUfAXd5fjP8s.zip",
|
|
"original_id": "8a83adcb66a146b2a6d4587f08fcebc2"
|
|
},
|
|
{
|
|
"name": "SD外壳-前置光-3mm厚电池.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/uSNoQFZ8MGsNQzrJ6KZqp8j8hzGCxRkkxUJDnRHy.zip",
|
|
"original_id": "b96b06f0488a40989693dc208e144c63"
|
|
},
|
|
{
|
|
"name": "SD外壳-前置光-5mm厚电池.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/SxSMfnmdUFvR8fchB0dUhrOReaHxk2FCFmp6fKQ3.zip",
|
|
"original_id": "e5e22e591566402085c1714781f8f7c3"
|
|
},
|
|
{
|
|
"name": "SD外壳通用按键.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/T7zXxYFA6P2BmTnZKm1k8k5APQaClyxkczpugbbe.zip",
|
|
"original_id": "e6df5bb5890143e1849961813164ed6a"
|
|
},
|
|
{
|
|
"name": "SD外壳-背透-普通屏-3mm电池.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/4/Fe5SewbFcqzjPGRftzCpPZ2jyk2YWf6mOB11suXH.zip",
|
|
"original_id": "0a20158e54ee42aca5592110d65e7d9a"
|
|
},
|
|
{
|
|
"name": "二次元打印面板_2022-05-14.zip",
|
|
"url": "https://image.lceda.cn/attachments/2022/5/p59xKPNESySN0pDu5b155hFjZIzPYa0ijRATXagA.zip",
|
|
"original_id": "ddde95909d3b4c339f62561141b3ff8f"
|
|
}
|
|
]
|
|
} |