Add EasyEDA Std project source ingestion (10 boards backfilled)
打通 oshwhub origin=std 项目的工程源(schematic + PCB dataStr)抓取链路。原
plan.md §1.6 假设需要登录,实测 lceda.cn/api/documents/<doc>?uuid=<doc>&path=<doc>
对公开项目匿名可访问 —— 无需 cookie,无账号封禁风险。
调研:4 轮探测留痕在 data/state/std_probe[1-5]/(gitignored);翻 Std 编辑器
v6.5.51 的 main.min.js bundle 找到 ajaxDetail 端点;按 docType 区分两种
响应 shape(schematic 项目视图 vs PCB 文档视图)。
Crawler:
- make_source_client() 用浏览器 UA + lceda.cn/editor Referer,因为
oshwhub /api/project/<uuid> 端点拒绝 FacereDataset/0.1 UA(CLAUDE.md
UA 例外条款:目标站主动封自定义 UA + 公开静态资源)
- fetch_std_source(): 项目元 → version_documents → 逐文档 dataStr → 落
source/<doc>.json + source/manifest.json
- --with-source(爬新项目时一并抓源)/ --backfill-source(仅扫已有)
- QPS ≤ 0.2 (SLEEP_SOURCE = 5s) 自律
Schema: 加 source_format / source_path / source_documents / editor_version
(前 3 进 enum 锁定,便于后续 Pro / KiCad 源对齐)。
回填结果:10/10 成功,45 个文档,33.2 MB;schema validate 全通。
docTypes 主要是 1 (schematic) 与 3 (pcb);USB 电压电流表只有 PCB 文档(4 个:
主板+盖板+底板+面板,作者未上传原理图源)。
完整调研:docs/sources/easyeda_std_source.md。
This commit is contained in:
@@ -114,5 +114,98 @@
|
||||
"public": true,
|
||||
"publish": true,
|
||||
"skipped_files": []
|
||||
}
|
||||
},
|
||||
"source_format": "easyeda-std",
|
||||
"source_path": "source/",
|
||||
"source_documents": [
|
||||
{
|
||||
"doc_uuid": "581fad31aaa646d2a44d1c71af407e22",
|
||||
"docType": 3,
|
||||
"master": "53fa44b5cd00480ebdab26184f9d3961",
|
||||
"path": "source/581fad31aaa646d2a44d1c71af407e22.json",
|
||||
"size": 42741,
|
||||
"sha256": "b1a80dff30673434e755a6fa8031ca449f7bea657d398a1ceba6233979958513"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "9b589a6563a34589a653af79ce2601fd",
|
||||
"docType": 3,
|
||||
"master": "e7caf824ae9b4017b72afecc878d73be",
|
||||
"path": "source/9b589a6563a34589a653af79ce2601fd.json",
|
||||
"size": 393666,
|
||||
"sha256": "3123da7ca0983915c3c325ac7408ab849a6497653bd614e2e91d91eb44fd1f4e"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "9e6a0767ca0f4388a7d537448900c1d6",
|
||||
"docType": 3,
|
||||
"master": "3c3fc00ebdcc42ffbf4c6de6462773db",
|
||||
"path": "source/9e6a0767ca0f4388a7d537448900c1d6.json",
|
||||
"size": 31536,
|
||||
"sha256": "5f3f78a7454f9b1134ef6e5e62aea03208efa00674928fe2bf7a532370137ce6"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "4ef63d2e30644cda90b1e2614993da59",
|
||||
"docType": 3,
|
||||
"master": "720838a6ed1e4c799a6260e651dea70d",
|
||||
"path": "source/4ef63d2e30644cda90b1e2614993da59.json",
|
||||
"size": 268259,
|
||||
"sha256": "84f6ff4c83bc5ed274c9a5a66643ac06c68ec5126bbb89f9e7b5d80475f30c4b"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "a1ba53dbe7074982908c77f865879936",
|
||||
"docType": 3,
|
||||
"master": "2f060e54b9174d92bf005f5e61368693",
|
||||
"path": "source/a1ba53dbe7074982908c77f865879936.json",
|
||||
"size": 272392,
|
||||
"sha256": "61ea9bbaf84f103a2b52ed8ec8ab62a24e3ab793e6a2fbff403fe39efb99ccdb"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "e997ebc226ba4c23b623d1453b4e83e0",
|
||||
"docType": 3,
|
||||
"master": "20687d8da0964cb68c1ce4ef7224eac8",
|
||||
"path": "source/e997ebc226ba4c23b623d1453b4e83e0.json",
|
||||
"size": 285688,
|
||||
"sha256": "7857ece437ff9b2c1400536956f726f742e476a189b74316845387409fe6602f"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "e5ff1516c9f0487191d462ce9fb624c6",
|
||||
"docType": 3,
|
||||
"master": "bc47fe9314ac411884f1becf81eaefca",
|
||||
"path": "source/e5ff1516c9f0487191d462ce9fb624c6.json",
|
||||
"size": 202162,
|
||||
"sha256": "15433077b44e3a61156a14956b75767f101982a38474c3f7ae69f8d548d36bb5"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "e2ea77b874354c8eb671bf7528e90208",
|
||||
"docType": 1,
|
||||
"master": "3c793a60f73d49f7ad431455a7921f98",
|
||||
"path": "source/e2ea77b874354c8eb671bf7528e90208.json",
|
||||
"size": 1086565,
|
||||
"sha256": "1f643863a52f4c915c15161e6bff0a85311fe2dfd644011f72fee8681efd32a1"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "25931ddab8c74781869e0c450f727692",
|
||||
"docType": 3,
|
||||
"master": "54c6c58e645149b38968c43ab875208b",
|
||||
"path": "source/25931ddab8c74781869e0c450f727692.json",
|
||||
"size": 640884,
|
||||
"sha256": "7a0b98f37bd520688088d849936375dad415a00321c7c5ef8c3bd309d747af30"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "ae09274603bd4aabb8012c6271f9d880",
|
||||
"docType": 3,
|
||||
"master": "37a5b8cea2464a3a897585785d7e9f26",
|
||||
"path": "source/ae09274603bd4aabb8012c6271f9d880.json",
|
||||
"size": 1231006,
|
||||
"sha256": "5ae1ca738a9fe3cbbaad2b6765ff5011dcdf3977dd3782323ec20854c50ee493"
|
||||
},
|
||||
{
|
||||
"doc_uuid": "b70e9127ec9f443db657aedbd85f3c7f",
|
||||
"docType": 3,
|
||||
"master": "02ae98e1877e42e0b936aa6e199c1874",
|
||||
"path": "source/b70e9127ec9f443db657aedbd85f3c7f.json",
|
||||
"size": 1821554,
|
||||
"sha256": "042357f1623fe4919a38c2c871f7da8d5ce3a907c2ec5e2d8e24cf16fb77ade8"
|
||||
}
|
||||
],
|
||||
"editor_version": "6.5.15"
|
||||
}
|
||||
Reference in New Issue
Block a user