crawler: --no-cover, --concurrency, drop cross-host sleep + batch-50 Step 1 done

Three crawler ergonomics for batch operations: --no-cover Skip cover image download. For scan-only modes (license/meta scrape) this drops ~1.3s/project and avoids slow-CDN hangs. --concurrency N ThreadPoolExecutor wrapping the per-project loop. Default 1 = serial (current behavior). Anonymous endpoints tolerate 5+ comfortably; output uses a print lock for readable interleaved progress. fetch_cover plumbs through crawl_one. Drop cross-host sleep #1: in crawl_one between detail HTML (oshwhub.com) and cover image (image.lceda.cn). Different hosts — sleep was unnecessary. Saves ~1s/project. Sleep #2 (post-cover, before next iteration) stays — it gates the next oshwhub.com hit. download_to gains max_seconds wall budget (default 60s, cover uses 15s). Defends against pathologically slow CDN connections — observed 10 KB/s on image.lceda.cn for one project, would have hung 6+ min on a 3.6 MB cover otherwise. httpx default timeout resets per chunk, so streaming downloads need an external wall-clock guard. batch-50 Step 1 (license/meta scrape) shipped: 50/50 candidates have metadata.json + license recorded License distribution: GPL 3.0 32, Public Domain 6, NC variants 8, CERN-OHL 1, MIT 1, CC BY 3.0 1 Forge-friendly (non-NC): 41/50 (82%) Declared attachments: 180 files / 2.36 GB (median 18 MB/proj, max 304 MB) Walltime: 3min 26s for 28 projects at concurrency=5 (server-side HTML render bound, not sleep-bound) One orphan partial cover (a670e60a...) cleaned up — leftover from the first aborted run before the timeout fix landed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 01:35:11 +08:00
parent fe6971f3f9
commit c6fd111d6d
171 changed files with 5410 additions and 17 deletions
--- a/data/raw/oshwhub/f2a2530d6e1848109aeda5fd5f9e38d9/_urls.json
+++ b/data/raw/oshwhub/f2a2530d6e1848109aeda5fd5f9e38d9/_urls.json
@@ -0,0 +1,46 @@
+{
+  "detail_url": "https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi",
+  "cover_url": "https://image.lceda.cn/pullimage/wxmiS0dRdmrR71POPSkSyTfii6qWXkzHAj9lmicv.png",
+  "attachments": [
+    {
+      "name": "Firmware_Moji_v0.9.9_0x00_2025-01-22.bin",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/d03f6c19d2e94073b97b38443f9864fd.bin",
+      "original_id": "8ce49eadb48441d2ad6edcb5ef1eb67b"
+    },
+    {
+      "name": "Moji_2025-01-21-1.mp4",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/12db74babe1f43bba73bcd2f709402b6.mp4",
+      "original_id": "daf70fe0ceb8455bb294274da412a656"
+    },
+    {
+      "name": "v1.0.1_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/1abeb5dc5d3442d1ab6fde7a96279043.zip",
+      "original_id": "b67a575a4e7740c8b6c677af32706de8"
+    },
+    {
+      "name": "Moji_AI尬聊.mp4",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/4e85e95da82642d78468b88f8a3455c5.mp4",
+      "original_id": "2ace833bb4394ee58f2cd44b7550a29f"
+    },
+    {
+      "name": "v1.1.3_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/994e4e6111f048fbae30353bf6eb0b64.zip",
+      "original_id": "5e7a8a6dc1094966a57a89858e1bf746"
+    },
+    {
+      "name": "v1.2.1_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/81e3bc80a46b4ee48d15d6e27c78e227.zip",
+      "original_id": "57b033eda2be4265a6411dbd26f5f209"
+    },
+    {
+      "name": "v1.4.6_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/48cdc0f6e8c04a909e74d8d11a1e84fa.zip",
+      "original_id": "5d8b3830ba944aae8c026023b216a3e9"
+    },
+    {
+      "name": "v1.5.0_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/c185b3c633644d64b3e95e1f97dfa678.zip",
+      "original_id": "87ce0d0f38434814957c170b002c675e"
+    }
+  ]
+}
--- a/data/raw/oshwhub/f2a2530d6e1848109aeda5fd5f9e38d9/description.md
+++ b/data/raw/oshwhub/f2a2530d6e1848109aeda5fd5f9e38d9/description.md
@@ -0,0 +1,9 @@
+# Moji 小智AI衍生版
+
+Moji，基于 小智AI 的桌面摆件，兼具智能与温暖，既是助手也是伙伴，以智慧解决问题，用温柔陪伴生活，让每一天更美好。
+
+---
+- Source: https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi
+- Author: 阿慕希 (movecall)
+- License: CC BY-NC 4.0
+- Published: 2025-03-22T07:15:23.000Z
--- a/data/raw/oshwhub/f2a2530d6e1848109aeda5fd5f9e38d9/metadata.json
+++ b/data/raw/oshwhub/f2a2530d6e1848109aeda5fd5f9e38d9/metadata.json
@@ -0,0 +1,113 @@
+{
+  "source": "oshwhub",
+  "source_url": "https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi",
+  "project_id": "f2a2530d6e1848109aeda5fd5f9e38d9",
+  "title": "Moji 小智AI衍生版",
+  "description_short": "Moji，基于 小智AI 的桌面摆件，兼具智能与温暖，既是助手也是伙伴，以智慧解决问题，用温柔陪伴生活，让每一天更美好。",
+  "description_path": "description.md",
+  "author": {
+    "username": "movecall",
+    "display_name": "阿慕希",
+    "user_id": "5f08c597e56a41bab8e5ee2500cedc61"
+  },
+  "license": "CC BY-NC 4.0",
+  "tags": [],
+  "created_at": "2025-01-14T12:58:20.977Z",
+  "updated_at": "2025-10-14T08:14:34.000Z",
+  "published_at": "2025-03-22T07:15:23.000Z",
+  "crawled_at": "2026-04-28T17:23:50.434002+00:00",
+  "metrics": {
+    "likes": 389,
+    "stars": 1108,
+    "forks": 0,
+    "views": 174750,
+    "watch": 0,
+    "comments": 530
+  },
+  "cover": {
+    "url": "https://image.lceda.cn/pullimage/wxmiS0dRdmrR71POPSkSyTfii6qWXkzHAj9lmicv.png",
+    "path": null
+  },
+  "files": [
+    {
+      "name": "Firmware_Moji_v0.9.9_0x00_2025-01-22.bin",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/d03f6c19d2e94073b97b38443f9864fd.bin",
+      "original_id": "8ce49eadb48441d2ad6edcb5ef1eb67b",
+      "ext": "bin",
+      "mime": "application/octet-stream",
+      "size": 4939936,
+      "md5": "9c57783c3bfe374a8381d2cca8f456b5"
+    },
+    {
+      "name": "Moji_2025-01-21-1.mp4",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/12db74babe1f43bba73bcd2f709402b6.mp4",
+      "original_id": "daf70fe0ceb8455bb294274da412a656",
+      "ext": "mp4",
+      "mime": "video/mp4",
+      "size": 85413731,
+      "md5": "01e4f6e0cd78f536a4227dc6469ae63a"
+    },
+    {
+      "name": "v1.0.1_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/1abeb5dc5d3442d1ab6fde7a96279043.zip",
+      "original_id": "b67a575a4e7740c8b6c677af32706de8",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 2551964,
+      "md5": "d5880288b620d5914def78c9db36aece"
+    },
+    {
+      "name": "Moji_AI尬聊.mp4",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/4e85e95da82642d78468b88f8a3455c5.mp4",
+      "original_id": "2ace833bb4394ee58f2cd44b7550a29f",
+      "ext": "mp4",
+      "mime": "video/mp4",
+      "size": 31922040,
+      "md5": "24b76437e4bc9f5f54cbed4f98360afc"
+    },
+    {
+      "name": "v1.1.3_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/994e4e6111f048fbae30353bf6eb0b64.zip",
+      "original_id": "5e7a8a6dc1094966a57a89858e1bf746",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 2544034,
+      "md5": "7d2cd0ab527f74c307b13ff069920dfe"
+    },
+    {
+      "name": "v1.2.1_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/81e3bc80a46b4ee48d15d6e27c78e227.zip",
+      "original_id": "57b033eda2be4265a6411dbd26f5f209",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 2552227,
+      "md5": "7259473dca1cb855c3eb51b812e43e66"
+    },
+    {
+      "name": "v1.4.6_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/48cdc0f6e8c04a909e74d8d11a1e84fa.zip",
+      "original_id": "5d8b3830ba944aae8c026023b216a3e9",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 5272612,
+      "md5": "9a98b546398b8cba907871dc58b4cde0"
+    },
+    {
+      "name": "v1.5.0_movecall-moji-esp32s3.zip",
+      "url": "https://image.lceda.cn/oshwhub/project/attachments/c185b3c633644d64b3e95e1f97dfa678.zip",
+      "original_id": "87ce0d0f38434814957c170b002c675e",
+      "ext": "zip",
+      "mime": "application/x-zip-compressed",
+      "size": 2580211,
+      "md5": "0fa874ee510d97fa70cf63f7b5baa99e"
+    }
+  ],
+  "raw_fields": {
+    "path": "movecall/moji-xiaozhi-ai-derivative-editi",
+    "grade": 3,
+    "origin": "pro",
+    "public": true,
+    "publish": true,
+    "skipped_files": []
+  }
+}