crawler: --no-cover, --concurrency, drop cross-host sleep + batch-50 Step 1 done

Three crawler ergonomics for batch operations: --no-cover Skip cover image download. For scan-only modes (license/meta scrape) this drops ~1.3s/project and avoids slow-CDN hangs. --concurrency N ThreadPoolExecutor wrapping the per-project loop. Default 1 = serial (current behavior). Anonymous endpoints tolerate 5+ comfortably; output uses a print lock for readable interleaved progress. fetch_cover plumbs through crawl_one. Drop cross-host sleep #1: in crawl_one between detail HTML (oshwhub.com) and cover image (image.lceda.cn). Different hosts — sleep was unnecessary. Saves ~1s/project. Sleep #2 (post-cover, before next iteration) stays — it gates the next oshwhub.com hit. download_to gains max_seconds wall budget (default 60s, cover uses 15s). Defends against pathologically slow CDN connections — observed 10 KB/s on image.lceda.cn for one project, would have hung 6+ min on a 3.6 MB cover otherwise. httpx default timeout resets per chunk, so streaming downloads need an external wall-clock guard. batch-50 Step 1 (license/meta scrape) shipped: 50/50 candidates have metadata.json + license recorded License distribution: GPL 3.0 32, Public Domain 6, NC variants 8, CERN-OHL 1, MIT 1, CC BY 3.0 1 Forge-friendly (non-NC): 41/50 (82%) Declared attachments: 180 files / 2.36 GB (median 18 MB/proj, max 304 MB) Walltime: 3min 26s for 28 projects at concurrency=5 (server-side HTML render bound, not sleep-bound) One orphan partial cover (a670e60a...) cleaned up — leftover from the first aborted run before the timeout fix landed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 01:35:11 +08:00
parent fe6971f3f9
commit c6fd111d6d
171 changed files with 5410 additions and 17 deletions
--- a/data/raw/oshwhub/ae88c322aff74ad3b1e9d5c403dfc4d4/_urls.json
+++ b/data/raw/oshwhub/ae88c322aff74ad3b1e9d5c403dfc4d4/_urls.json
@@ -0,0 +1,31 @@
+{
+  "detail_url": "https://oshwhub.com/fh001/usb-gong-shuai-ce-shi-yi",
+  "cover_url": "https://image.lceda.cn/pullimage/4XStvdZIf6el4fygj1sXF9ewYI9WA0ms1V8L5inC.jpeg",
+  "attachments": [
+    {
+      "name": "充电测试 红米note10pro 65w.mp4",
+      "url": "https://image.lceda.cn/attachments/2022/3/6ey3aXbh4byYjadLemIT3GXAkM5KPc9GSjyGW2Bc.mp4",
+      "original_id": "eae588fd6f2d40fba5c48f738e86d0b5"
+    },
+    {
+      "name": "快充诱骗与电压精度测试.mp4",
+      "url": "https://image.lceda.cn/attachments/2022/3/Ha7RXs0ty9pKigTWkVzro0L6JMXMLCHOjTyVB8E9.mp4",
+      "original_id": "95e2d19df3bf40429c5c1f1ebd97ec60"
+    },
+    {
+      "name": "电流精度测试.mp4",
+      "url": "https://image.lceda.cn/attachments/2022/3/g0wCi1iyGXVvsCx9k88kkDuVJaaTBoIptnIik4bk.mp4",
+      "original_id": "a29bbad356a7451b8a4f49e133007b5a"
+    },
+    {
+      "name": "代码22-3-20.rar",
+      "url": "https://image.lceda.cn/attachments/2022/3/pxKqrVHGJu7vxl5JhBbLYtAOOUTCvsP499b2N9wW.rar",
+      "original_id": "5eb1153c97564c63b4d9606bc2e94458"
+    },
+    {
+      "name": "外壳.zip",
+      "url": "https://image.lceda.cn/attachments/2022/3/F14uCfjW9exeSH1pNC3QQKB9dU9LmJwFW4HN0xFy.zip",
+      "original_id": "2ae57cd39a564ef382209dd8a4c74dbd"
+    }
+  ]
+}
--- a/data/raw/oshwhub/ae88c322aff74ad3b1e9d5c403dfc4d4/description.md
+++ b/data/raw/oshwhub/ae88c322aff74ad3b1e9d5c403dfc4d4/description.md
@@ -0,0 +1,9 @@
+# 【毕设】USB功率测试仪
+
+USB功率测试器，可测试目前几乎所有协议的快充，短时最高功率160W，长时间最大功率120W
+
+---
+- Source: https://oshwhub.com/fh001/usb-gong-shuai-ce-shi-yi
+- Author: fh001 (fh001)
+- License: GPL 3.0
+- Published: 2022-07-24T09:15:16.000Z
--- a/data/raw/oshwhub/ae88c322aff74ad3b1e9d5c403dfc4d4/metadata.json
+++ b/data/raw/oshwhub/ae88c322aff74ad3b1e9d5c403dfc4d4/metadata.json
@@ -0,0 +1,86 @@
+{
+  "source": "oshwhub",
+  "source_url": "https://oshwhub.com/fh001/usb-gong-shuai-ce-shi-yi",
+  "project_id": "ae88c322aff74ad3b1e9d5c403dfc4d4",
+  "title": "【毕设】USB功率测试仪",
+  "description_short": "USB功率测试器，可测试目前几乎所有协议的快充，短时最高功率160W，长时间最大功率120W",
+  "description_path": "description.md",
+  "author": {
+    "username": "fh001",
+    "display_name": "fh001",
+    "user_id": "a7ac3462734d4c1cb878b165f31e4bcf"
+  },
+  "license": "GPL 3.0",
+  "tags": [],
+  "created_at": "2022-02-25T06:54:27.000Z",
+  "updated_at": "2025-04-25T11:22:35.000Z",
+  "published_at": "2022-07-24T09:15:16.000Z",
+  "crawled_at": "2026-04-28T17:30:23.058192+00:00",
+  "metrics": {
+    "likes": 409,
+    "stars": 806,
+    "forks": 203,
+    "views": 55697,
+    "watch": 0,
+    "comments": 342
+  },
+  "cover": {
+    "url": "https://image.lceda.cn/pullimage/4XStvdZIf6el4fygj1sXF9ewYI9WA0ms1V8L5inC.jpeg",
+    "path": null
+  },
+  "files": [
+    {
+      "name": "充电测试 红米note10pro 65w.mp4",
+      "url": "https://image.lceda.cn/attachments/2022/3/6ey3aXbh4byYjadLemIT3GXAkM5KPc9GSjyGW2Bc.mp4",
+      "original_id": "eae588fd6f2d40fba5c48f738e86d0b5",
+      "ext": "mp4",
+      "mime": "video/mp4",
+      "size": 14939124,
+      "md5": "66f104fae316f4619ebaf83a31f16f0c"
+    },
+    {
+      "name": "快充诱骗与电压精度测试.mp4",
+      "url": "https://image.lceda.cn/attachments/2022/3/Ha7RXs0ty9pKigTWkVzro0L6JMXMLCHOjTyVB8E9.mp4",
+      "original_id": "95e2d19df3bf40429c5c1f1ebd97ec60",
+      "ext": "mp4",
+      "mime": "video/mp4",
+      "size": 38086966,
+      "md5": "b853931fcfa568edae398505b693dafd"
+    },
+    {
+      "name": "电流精度测试.mp4",
+      "url": "https://image.lceda.cn/attachments/2022/3/g0wCi1iyGXVvsCx9k88kkDuVJaaTBoIptnIik4bk.mp4",
+      "original_id": "a29bbad356a7451b8a4f49e133007b5a",
+      "ext": "mp4",
+      "mime": "video/mp4",
+      "size": 17206476,
+      "md5": "485b30da50046c0d29062ff73e7267e8"
+    },
+    {
+      "name": "代码22-3-20.rar",
+      "url": "https://image.lceda.cn/attachments/2022/3/pxKqrVHGJu7vxl5JhBbLYtAOOUTCvsP499b2N9wW.rar",
+      "original_id": "5eb1153c97564c63b4d9606bc2e94458",
+      "ext": "rar",
+      "mime": "application/octet-stream",
+      "size": 8100753,
+      "md5": "cadb3a8f320594750e7a38c396144b85"
+    },
+    {
+      "name": "外壳.zip",
+      "url": "https://image.lceda.cn/attachments/2022/3/F14uCfjW9exeSH1pNC3QQKB9dU9LmJwFW4HN0xFy.zip",
+      "original_id": "2ae57cd39a564ef382209dd8a4c74dbd",
+      "ext": "zip",
+      "mime": "application/zip",
+      "size": 49606,
+      "md5": "559f2cc9ec06f720e915479f91c2dd68"
+    }
+  ],
+  "raw_fields": {
+    "path": "fh001/usb-gong-shuai-ce-shi-yi",
+    "grade": 3,
+    "origin": "std",
+    "public": true,
+    "publish": true,
+    "skipped_files": []
+  }
+}