oshwhub: dump full listing index (33,695 projects) for batch sizing
Probed listing API and learned: total field is exposed (Pro=21,202 / Std=12,493), pageSize accepts >=1000 (full corpus = 35 requests / 71s), sort param is silently ignored. Dump all listings via scripts/dump_listing_index.py to local jsonl so downstream batch-selection no longer hits the API. Why: needed quantitative anchors before scaling Pro batch beyond top-5. License is detail-page only (~19h serial scan), so we want to filter on grade/like *locally* first to shortlist before paying that cost. Quality-tier counts now known: A-tier (grade>=3 & like>=10) = 2,806 across both origins. - scripts/dump_listing_index.py: one-shot scraper, polite QPS, streams to jsonl - docs/sources/oshwhub_listing_full.md: human-readable report with growth trends, quality tiers, owner concentration, and storage-budget anchors Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
144
docs/sources/oshwhub_listing_full.md
Normal file
144
docs/sources/oshwhub_listing_full.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# oshwhub 全量 Listing 索引简报
|
||||
|
||||
**采集时间**：2026-04-28
|
||||
**数据**：`data/state/oshwhub_listing_full.jsonl`（28.4 MB，gitignore，可重抓）
|
||||
**采集脚本**：`scripts/dump_listing_index.py`
|
||||
**接口**：`https://oshwhub.com/api/project?origin={pro,std}&pageSize=1000&page=N`，无鉴权
|
||||
**耗时 / 流量**：35 次请求 × ~1 MB ≈ 35 MB / 71 秒（QPS=1）
|
||||
|
||||
---
|
||||
|
||||
## 1. 总盘
|
||||
|
||||
| origin | 项目数 | 时间跨度 | 唯一作者数 |
|
||||
|---|---:|---|---:|
|
||||
| Pro | **21,202** | 2021-05 ~ 2026-04 | 10,536 |
|
||||
| Std | **12,493** | 2016-07 ~ 2026-04 | 5,531 |
|
||||
| **合计** | **33,695** | — | ~16,000 |
|
||||
|
||||
> "项目"定义：listing API 返回的 `public:true` 公开项目；私有 / 草稿不计入。
|
||||
>
|
||||
> Pro / Std 不重叠（单项目 origin 二选一）。
|
||||
|
||||
---
|
||||
|
||||
## 2. 增长趋势（按 `created_at` 年份）
|
||||
|
||||
| year | Pro | Std |
|
||||
|---:|---:|---:|
|
||||
| 2016 | — | 1 |
|
||||
| 2017 | — | 73 |
|
||||
| 2018 | — | 30 |
|
||||
| 2019 | — | 320 |
|
||||
| 2020 | — | 2,545 |
|
||||
| 2021 | 13 | 3,193 |
|
||||
| 2022 | 1,718 | 3,394 |
|
||||
| 2023 | 4,056 | 1,554 |
|
||||
| 2024 | 6,827 | 905 |
|
||||
| 2025 | 7,436 | 426 |
|
||||
| 2026 (Q1) | 1,152 | 52 |
|
||||
|
||||
**关键观察**：
|
||||
- **Std 在 2021-2022 见顶后断崖下跌**（3394 → 1554 → 905 → 426），用户在迁出。
|
||||
- **Pro 2022 起线性增长**，2024-2025 合计占全 Pro 的 67%。
|
||||
- 2026 仅过 4 个月就有 1,152 Pro 新项目（外推全年 ~3,500，会比 2025 略降，但仍是主战场）。
|
||||
|
||||
→ **抓取策略含义**：Pro corpus 还在快速膨胀，索引建议每月增量重抓一次；Std 已基本静态。
|
||||
|
||||
---
|
||||
|
||||
## 3. 质量分层
|
||||
|
||||
| Tier | 条件 | Pro 命中 | Pro % | Std 命中 | Std % |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| S | grade≥4 & like≥50 | 390 | 1.8% | 193 | 1.5% |
|
||||
| A | grade≥3 & like≥10 | 1,356 | 6.4% | 1,450 | 11.6% |
|
||||
| B | grade≥2 & like≥5 | 2,292 | 10.8% | 3,951 | 31.6% |
|
||||
| 总池 | grade≥1 | 20,028 | 94.5% | 9,510 | 76.1% |
|
||||
| 末档 | grade==0 | 1,174 | 5.5% | 2,983 | 23.9% |
|
||||
|
||||
**两点反直觉**：
|
||||
- **Std 高质量比例 > Pro**（A 档 11.6% vs 6.4%）。Std 平台老，项目有时间累积点赞；Pro 大量是 2024+ 新项目，社交信号还没沉淀。
|
||||
- **Pro 末档比例反而低**（5.5% vs 23.9%）。可能是 grade 内部打分逻辑不同，Std 兜底更宽。
|
||||
|
||||
---
|
||||
|
||||
## 4. 互动指标分布
|
||||
|
||||
### like
|
||||
| | p50 | p90 | p99 | max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Pro | 2 | 22 | 190 | 2,137 |
|
||||
| Std | 6 | 40 | 207 | 1,457 |
|
||||
|
||||
### views
|
||||
| | p50 | p90 | p99 | max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Pro | 1,036 | 6,550 | 39,144 | 1,036,026 |
|
||||
| Std | 3,362 | 13,519 | 53,663 | 348,877 |
|
||||
|
||||
> Std 流量是 Pro 同档位的 ~3 倍（更老更普及）；但 Pro 头部单项 view 能上百万（X86 主板等）。
|
||||
|
||||
---
|
||||
|
||||
## 5. 头部作者（按项目数）
|
||||
|
||||
### Pro top-10
|
||||
| # | username | n |
|
||||
|---:|---|---:|
|
||||
| 1 | course-examples | 79 |
|
||||
| 2 | freakstudio | 79 |
|
||||
| 3 | aknice | 69 |
|
||||
| 4 | eda_kwazdnpkc | 64 |
|
||||
| 5 | windskys | 56 |
|
||||
| 6 | li-chuang-kai-fa-ban | 49 |
|
||||
| 7 | dingcheng | 44 |
|
||||
| 8 | o0v0o | 42 |
|
||||
| 9 | xiaocc22 | 42 |
|
||||
| 10 | eda_ycuzzivoa | 41 |
|
||||
|
||||
### Std top-10
|
||||
| # | username | n |
|
||||
|---:|---|---:|
|
||||
| 1 | li-chuang-zhi-neng-ying-jian-bu | 183 |
|
||||
| 2 | zhqsoft | 144 |
|
||||
| 3 | jixin | 92 |
|
||||
| 4 | Ismartware | 53 |
|
||||
| 5 | FJ956391150 | 52 |
|
||||
| 6 | an_ye | 51 |
|
||||
| 7 | micespring | 48 |
|
||||
| 8 | li-chuang-EDAjing-sai-xiao-zu | 47 |
|
||||
| 9 | 583703056a | 46 |
|
||||
| 10 | akatople | 46 |
|
||||
|
||||
> 立创官方账号占据头部（`course-examples` / `li-chuang-kai-fa-ban` / `li-chuang-zhi-neng-ying-jian-bu` / `li-chuang-EDAjing-sai-xiao-zu`）—— 这些是 **开发板 / 比赛 / 教程** 系列，质量高、license 倾向 GPL/NC-SA，内容受立创内部审核。
|
||||
>
|
||||
> 长尾比较健康：top 1 占比仅 0.4% (Pro) / 1.5% (Std)；不存在被少数账号 dominate 的情况。
|
||||
|
||||
---
|
||||
|
||||
## 6. listing API 注意事项（沉淀给后续爬虫开发）
|
||||
|
||||
- **`origin` 默认值**：不带参数返回 Std；要 Pro 必须显式 `origin=pro`。
|
||||
- **`sort` 参数无效**：`hot/new/newest/latest/created_at/...` 全部静默 fallback 到隐式排序。实测排序 = grade desc → 内部质量分 desc。要"按时间排序"得自己拉全集后本地排。
|
||||
- **`pageSize` 无显式上限**：`pageSize=1000` 工作良好，单次响应 ~1 MB；`pageSize=30`（旧爬虫默认）效率低 30 倍。
|
||||
- **`license` 不在 listing**：要 license 必须挨个抓详情页 (`/<owner>/<path>`)，33,695 个 → QPS≤0.5 → ~19 小时；考虑同步抓索引时**只抓详情页**，不重抓 listing。
|
||||
- **`public:true` 隐含**：listing 只返公开项目；不过我们也只关心公开项目。
|
||||
|
||||
---
|
||||
|
||||
## 7. 后续放量决策的量化锚点
|
||||
|
||||
| 目标 | 候选池大小 | 估算开销（详情页 + 工程源） |
|
||||
|---|---:|---|
|
||||
| **保守**：S 档（grade≥4 & like≥50）双 origin | 583 | 1-2 小时详情，~50 GB Pro 工程源 + ~10 GB Std |
|
||||
| **中度**：A 档（grade≥3 & like≥10）双 origin | 2,806 | ~7 小时详情，~250 GB Pro + ~25 GB Std |
|
||||
| **激进**：B 档双 origin | 6,243 | ~14 小时详情，~500 GB Pro + ~60 GB Std |
|
||||
| 全 Pro | 21,202 | 19h 详情，**~1 TB Pro 源** |
|
||||
| 全 corpus | 33,695 | 19h 详情，~1.1 TB+ |
|
||||
|
||||
> Pro 工程源体积估计基于已落库 5 项均值（最坏单项 481 MB），P50 约 30 MB；中度 + 激进档若加 size cap（如单项 ≤ 200 MB skip）能砍 30-40%。
|
||||
|
||||
**当前下一步建议**：
|
||||
- 先按 **A 档**（2,806 项）在本地索引上 filter，做 license 详情扫描，得到"license 白名单 ∩ A 档"的真实候选清单
|
||||
- 再决定是否动手批量下源
|
||||
41
log.md
41
log.md
@@ -4,6 +4,47 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-04-28 23:30 oshwhub 全量 listing 索引落本地：33,695 项 / 28.4 MB
|
||||
|
||||
**Claude 会话**
|
||||
|
||||
为了在"扩量到 top-30 / top-50 / 全量"前先量化候选池规模 + 质量分布，把 oshwhub listing API 全量扫一遍落地。
|
||||
|
||||
### 关键收获（之前以为是黑箱）
|
||||
|
||||
- **listing API 直接返回 `total` 字段**：Pro 21,202 / Std 12,493，**合计 33,695**。
|
||||
- **`pageSize` 无上限**，实测 1000 工作良好；全量索引 = 35 次请求 / 71 秒 / 35 MB 流量。
|
||||
- **`sort` 参数被服务端静默忽略**——传啥都返回相同顺序（grade desc → 隐式质量分 desc）。"按时间排序"必须先拉全集再本地排。
|
||||
- **`origin` 默认 std**——不带参数永远看不到 Pro 池。
|
||||
- **`license` 不在 listing 响应**，必须挨个抓详情页（QPS=0.5 → ~19 小时全量）。
|
||||
|
||||
### 数据画像（写到 `docs/sources/oshwhub_listing_full.md`）
|
||||
|
||||
- **Pro 长尾极重**：grade=1 占 82%，真正 A 档（grade≥3 & like≥10）只有 1,356 (6.4%)
|
||||
- **Std 高质量比例反而高**：A 档 1,450 (11.6%)，因为平台老 7 年（2016 起 vs Pro 2021 起），项目有时间累积点赞
|
||||
- **Std 已停滞**：2021-2022 见顶（3.4k/年），之后断崖（1.5k → 0.9k → 0.4k → 0.05k 2026Q1）
|
||||
- **Pro 还在快速膨胀**：2023 起线性增长，2025 全年 7.4k，2026Q1 已 1.1k
|
||||
- **作者长尾健康**：Pro 10,536 个 / Std 5,531 个唯一作者；top-1 占比 0.4% / 1.5%
|
||||
- **立创官方账号占据头部**（course-examples / li-chuang-kai-fa-ban / li-chuang-zhi-neng-ying-jian-bu）
|
||||
|
||||
### 实操含义
|
||||
|
||||
放量决策有了量化锚点：S 档 583 项 / A 档 2,806 项 / B 档 6,243 项 / 全量 33,695。Pro 工程源体积外推（基于 5 项实测均值），全 Pro 约 1 TB——超出 Gitea LFS 舒适区，必须配 size cap + license 白名单。
|
||||
|
||||
### 下一步
|
||||
|
||||
1. 在本地 jsonl 上按 A 档过滤，做 license 详情页扫描（一次性 ~7 小时）
|
||||
2. license 白名单 ∩ A 档 → 真候选清单
|
||||
3. 然后再决定批量下载源
|
||||
|
||||
### 文件
|
||||
|
||||
- `scripts/dump_listing_index.py` —— 一次性全量扫描脚本，可重抓
|
||||
- `data/state/oshwhub_listing_full.jsonl` —— 28.4 MB，gitignore（可重建，不入库）
|
||||
- `docs/sources/oshwhub_listing_full.md` —— 给人看的简报
|
||||
|
||||
---
|
||||
|
||||
## 2026-04-28 22:00 Pro 2.x 旧版工程源抓取链路打通，5/5 Pro 项目全部 ✅
|
||||
|
||||
**Claude 会话**
|
||||
|
||||
126
scripts/dump_listing_index.py
Normal file
126
scripts/dump_listing_index.py
Normal file
@@ -0,0 +1,126 @@
|
||||
"""Dump full oshwhub listing index for both origins to a local jsonl.
|
||||
|
||||
只抓 listing API，不抓详情页、不抓附件、不抓工程源。结果落
|
||||
`data/state/oshwhub_listing_full.jsonl`，每行一条 listing 项。
|
||||
|
||||
Usage:
|
||||
uv run python scripts/dump_listing_index.py
|
||||
uv run python scripts/dump_listing_index.py --page-size 500 --sleep 0.5
|
||||
|
||||
API 注意：
|
||||
- `sort` 参数被服务端静默忽略，无论传啥都返回隐式排序（grade desc → 质量 desc）
|
||||
- 默认 `origin` 是 std；要 Pro 必须显式 `origin=pro`
|
||||
- `pageSize` 实测 ≥1000 都接受，单次响应体 ~1 MB
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import statistics as st
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from crawlers.oshwhub.crawler import list_projects, make_client # noqa: E402
|
||||
|
||||
ORIGINS = ("pro", "std")
|
||||
DEFAULT_OUT = Path("data/state/oshwhub_listing_full.jsonl")
|
||||
|
||||
|
||||
def dump_origin(client, origin: str, page_size: int, sleep: float, sink) -> int:
|
||||
# First call to learn `total` / `totalPage`.
|
||||
res = list_projects(client, page=1, page_size=page_size, origin=origin)
|
||||
total = res["total"]
|
||||
n_pages = math.ceil(total / page_size)
|
||||
written = 0
|
||||
for it in res["lists"]:
|
||||
sink.write(json.dumps(it, ensure_ascii=False) + "\n")
|
||||
written += 1
|
||||
print(
|
||||
f"[{origin}] total={total} pages={n_pages} pageSize={page_size}",
|
||||
flush=True,
|
||||
)
|
||||
print(f" page 1/{n_pages}: {len(res['lists'])} items", flush=True)
|
||||
for p in range(2, n_pages + 1):
|
||||
time.sleep(sleep)
|
||||
res = list_projects(client, page=p, page_size=page_size, origin=origin)
|
||||
for it in res["lists"]:
|
||||
sink.write(json.dumps(it, ensure_ascii=False) + "\n")
|
||||
written += 1
|
||||
print(f" page {p:>2}/{n_pages}: {len(res['lists'])} items", flush=True)
|
||||
if written != total:
|
||||
print(
|
||||
f" WARN: wrote {written} but server said total={total} "
|
||||
f"(diff={total - written})",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return written
|
||||
|
||||
|
||||
def summarize(path: Path) -> None:
|
||||
by_origin: dict[str, list[dict]] = {}
|
||||
with path.open() as f:
|
||||
for ln in f:
|
||||
it = json.loads(ln)
|
||||
by_origin.setdefault(it.get("origin") or "?", []).append(it)
|
||||
print("\n===== summary =====")
|
||||
print(f"file: {path} size={path.stat().st_size / 1024 / 1024:.1f} MB")
|
||||
for origin, items in sorted(by_origin.items()):
|
||||
if not items:
|
||||
continue
|
||||
likes = sorted(((i.get("count") or {}).get("like") or 0) for i in items)
|
||||
views = sorted(((i.get("count") or {}).get("views") or 0) for i in items)
|
||||
grades = Counter(i.get("grade") for i in items)
|
||||
# date range from created_at
|
||||
dates = sorted(i.get("created_at") or "" for i in items if i.get("created_at"))
|
||||
date_min = dates[0][:10] if dates else "?"
|
||||
date_max = dates[-1][:10] if dates else "?"
|
||||
print(f"\n[{origin}] n={len(items)}")
|
||||
print(
|
||||
f" likes: median={likes[len(likes) // 2]} "
|
||||
f"p90={likes[int(len(likes) * 0.9)]} "
|
||||
f"p99={likes[int(len(likes) * 0.99)]} max={likes[-1]}"
|
||||
)
|
||||
print(
|
||||
f" views: median={views[len(views) // 2]} "
|
||||
f"p90={views[int(len(views) * 0.9)]} max={views[-1]}"
|
||||
)
|
||||
print(f" grade: {dict(sorted(grades.items(), key=lambda kv: -(kv[0] or 0)))}")
|
||||
print(f" dates: {date_min} .. {date_max}")
|
||||
# Quality tiers (handy for batch sizing)
|
||||
tier_grade3 = sum(1 for i in items if (i.get("grade") or 0) >= 3)
|
||||
tier_like10 = sum(
|
||||
1
|
||||
for i in items
|
||||
if ((i.get("count") or {}).get("like") or 0) >= 10
|
||||
and (i.get("grade") or 0) >= 3
|
||||
)
|
||||
print(f" tier: grade>=3: {tier_grade3} grade>=3 & like>=10: {tier_like10}")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--out", type=Path, default=DEFAULT_OUT)
|
||||
ap.add_argument("--page-size", type=int, default=1000)
|
||||
ap.add_argument("--sleep", type=float, default=1.0, help="seconds between page calls")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
print(f"writing -> {args.out}")
|
||||
t0 = time.time()
|
||||
with args.out.open("w") as sink, make_client() as client:
|
||||
total = 0
|
||||
for origin in ORIGINS:
|
||||
total += dump_origin(client, origin, args.page_size, args.sleep, sink)
|
||||
sink.flush()
|
||||
print(f"\ndone: {total} items in {time.time() - t0:.1f}s")
|
||||
summarize(args.out)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user