crawler: --no-cover, --concurrency, drop cross-host sleep + batch-50 Step 1 done

Three crawler ergonomics for batch operations: --no-cover Skip cover image download. For scan-only modes (license/meta scrape) this drops ~1.3s/project and avoids slow-CDN hangs. --concurrency N ThreadPoolExecutor wrapping the per-project loop. Default 1 = serial (current behavior). Anonymous endpoints tolerate 5+ comfortably; output uses a print lock for readable interleaved progress. fetch_cover plumbs through crawl_one. Drop cross-host sleep #1: in crawl_one between detail HTML (oshwhub.com) and cover image (image.lceda.cn). Different hosts — sleep was unnecessary. Saves ~1s/project. Sleep #2 (post-cover, before next iteration) stays — it gates the next oshwhub.com hit. download_to gains max_seconds wall budget (default 60s, cover uses 15s). Defends against pathologically slow CDN connections — observed 10 KB/s on image.lceda.cn for one project, would have hung 6+ min on a 3.6 MB cover otherwise. httpx default timeout resets per chunk, so streaming downloads need an external wall-clock guard. batch-50 Step 1 (license/meta scrape) shipped: 50/50 candidates have metadata.json + license recorded License distribution: GPL 3.0 32, Public Domain 6, NC variants 8, CERN-OHL 1, MIT 1, CC BY 3.0 1 Forge-friendly (non-NC): 41/50 (82%) Declared attachments: 180 files / 2.36 GB (median 18 MB/proj, max 304 MB) Walltime: 3min 26s for 28 projects at concurrency=5 (server-side HTML render bound, not sleep-bound) One orphan partial cover (a670e60a...) cleaned up — leftover from the first aborted run before the timeout fix landed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 01:35:11 +08:00
parent fe6971f3f9
commit c6fd111d6d
171 changed files with 5410 additions and 17 deletions
--- a/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/_urls.json
+++ b/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/_urls.json
@@ -0,0 +1,36 @@
+{
+  "detail_url": "https://oshwhub.com/myzhazha/esp32s3_86-kai-fa-ban",
+  "cover_url": "https://image.lceda.cn/pullimage/qbr0oZSUa25qiNIioPT4PUzSCSwWAuzERX8WnuHC.jpeg",
+  "attachments": [
+    {
+      "name": "外壳模型_打印用.zip",
+      "url": "https://image.lceda.cn/attachments/2022/12/kMxCuSpay5aq9JegFyburT7xCJjxgkqF6z7MQYvC.zip",
+      "original_id": "c9b528d9d012433e83306e23ef58bc0b"
+    },
+    {
+      "name": "Video_x264.mp4",
+      "url": "https://image.lceda.cn/attachments/2023/1/WeeCzShnEc6RtwHbsc5jybhJzmbmkbP7uTE6IACN.mp4",
+      "original_id": "90ffcd04132a4304854af83ae0884813"
+    },
+    {
+      "name": "音乐播放demo所需歌曲+歌词文件.rar",
+      "url": "https://image.lceda.cn/attachments/2023/6/wgIxElIEbpUah8IZqJce1Qner6QGvcEky3EbCtcI.rar",
+      "original_id": "0429db0c9b184c7584592c6b8ca78255"
+    },
+    {
+      "name": "所需库文件.rar",
+      "url": "https://image.lceda.cn/attachments/2023/7/yQgcZC2XZJKKREKgCwBxCD3tMyNclbGA3tdPR9nX.rar",
+      "original_id": "02e1f455212e4ea8b1595acc1e435a11"
+    },
+    {
+      "name": "DEMO.rar",
+      "url": "https://image.lceda.cn/attachments/2023/9/aoNveNkArCVxFuoQ583SEzrygtmxFZLwncYjhedx.rar",
+      "original_id": "90dea25d84194ed1bcc8f8c212f4efbb"
+    },
+    {
+      "name": "多合一固件.zip",
+      "url": "https://image.lceda.cn/attachments/2024/1/TqhcSDFeIpphwpxrI1dNn38FkyHbqME6kg2mb3FO.zip",
+      "original_id": "f7dfd2d9814a4ab48fb3d2907c353019"
+    }
+  ]
+}
--- a/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/cover.jpeg
+++ b/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/cover.jpeg
--- a/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/description.md
+++ b/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/description.md
@@ -0,0 +1,9 @@
+# ESP32S3 86盒开发板 带触摸
+
+使用ESP32S3N8R8作为主控的86面板开发板，带触摸带扬声器，可用于86面板开发，LVGL开发
+
+---
+- Source: https://oshwhub.com/myzhazha/esp32s3_86-kai-fa-ban
+- Author: 小渣渣 (myzhazha)
+- License: GPL 3.0
+- Published: 2024-01-02T02:06:18.000Z
--- a/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/metadata.json
+++ b/data/raw/oshwhub/53e6c236b9d14e74944e460ac11c7c1d/metadata.json
@@ -0,0 +1,95 @@
+{
+  "source": "oshwhub",
+  "source_url": "https://oshwhub.com/myzhazha/esp32s3_86-kai-fa-ban",
+  "project_id": "53e6c236b9d14e74944e460ac11c7c1d",
+  "title": "ESP32S3 86盒开发板 带触摸",
+  "description_short": "使用ESP32S3N8R8作为主控的86面板开发板，带触摸带扬声器，可用于86面板开发，LVGL开发",
+  "description_path": "description.md",
+  "author": {
+    "username": "myzhazha",
+    "display_name": "小渣渣",
+    "user_id": "26fa92c7af6f431490e2da8805f1d791"
+  },
+  "license": "GPL 3.0",
+  "tags": [],
+  "created_at": "2022-09-01T09:15:36.000Z",
+  "updated_at": "2025-12-29T06:58:15.000Z",
+  "published_at": "2024-01-02T02:06:18.000Z",
+  "crawled_at": "2026-04-28T17:01:38.044590+00:00",
+  "metrics": {
+    "likes": 666,
+    "stars": 1419,
+    "forks": 0,
+    "views": 141262,
+    "watch": 0,
+    "comments": 325
+  },
+  "cover": {
+    "url": "https://image.lceda.cn/pullimage/qbr0oZSUa25qiNIioPT4PUzSCSwWAuzERX8WnuHC.jpeg",
+    "path": "cover.jpeg"
+  },
+  "files": [
+    {
+      "name": "外壳模型_打印用.zip",
+      "url": "https://image.lceda.cn/attachments/2022/12/kMxCuSpay5aq9JegFyburT7xCJjxgkqF6z7MQYvC.zip",
+      "original_id": "c9b528d9d012433e83306e23ef58bc0b",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 88416,
+      "md5": "39563ddb5b5a4a22ccb4d047a9d7aa5a"
+    },
+    {
+      "name": "Video_x264.mp4",
+      "url": "https://image.lceda.cn/attachments/2023/1/WeeCzShnEc6RtwHbsc5jybhJzmbmkbP7uTE6IACN.mp4",
+      "original_id": "90ffcd04132a4304854af83ae0884813",
+      "ext": "mp4",
+      "mime": "video/mp4",
+      "size": 31433231,
+      "md5": "4935ce6a03a695bef0c7ea4489aa2afa"
+    },
+    {
+      "name": "音乐播放demo所需歌曲+歌词文件.rar",
+      "url": "https://image.lceda.cn/attachments/2023/6/wgIxElIEbpUah8IZqJce1Qner6QGvcEky3EbCtcI.rar",
+      "original_id": "0429db0c9b184c7584592c6b8ca78255",
+      "ext": "rar",
+      "mime": "application/octet-stream",
+      "size": 39480048,
+      "md5": "719df066f48e4dc799e767ef06619b2d"
+    },
+    {
+      "name": "所需库文件.rar",
+      "url": "https://image.lceda.cn/attachments/2023/7/yQgcZC2XZJKKREKgCwBxCD3tMyNclbGA3tdPR9nX.rar",
+      "original_id": "02e1f455212e4ea8b1595acc1e435a11",
+      "ext": "rar",
+      "mime": "application/octet-stream",
+      "size": 48938076,
+      "md5": "4ae8fe7350f4cb282ecb4e60f7e6cd74"
+    },
+    {
+      "name": "DEMO.rar",
+      "url": "https://image.lceda.cn/attachments/2023/9/aoNveNkArCVxFuoQ583SEzrygtmxFZLwncYjhedx.rar",
+      "original_id": "90dea25d84194ed1bcc8f8c212f4efbb",
+      "ext": "rar",
+      "mime": "application/octet-stream",
+      "size": 3217239,
+      "md5": "132e348e4573fd67ccb9a54e9e4d3491"
+    },
+    {
+      "name": "多合一固件.zip",
+      "url": "https://image.lceda.cn/attachments/2024/1/TqhcSDFeIpphwpxrI1dNn38FkyHbqME6kg2mb3FO.zip",
+      "original_id": "f7dfd2d9814a4ab48fb3d2907c353019",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 19132490,
+      "md5": "d94418a0996d2e3da712cde25b21ddff"
+    }
+  ],
+  "raw_fields": {
+    "path": "myzhazha/esp32s3_86-kai-fa-ban",
+    "grade": 4,
+    "origin": "pro",
+    "public": true,
+    "publish": true,
+    "skipped_files": []
+  }
+}