Facere/FacereDataset
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
5aefd7c0a7142b190b7e93cb2d0ff29b5d52aec9
FacereDataset/data/raw/oshwhub/7360e73de5dd428e9f29e10573f2d8ac/source/manifest.json
Knowit c6279bff08 Add EasyEDA Pro 2.x legacy source ingestion (5/5 batch closure) 
补齐前一批失败的 2 个 legacy Pro 项目（立创·泰山派 RK3566、立创·梁山派），
打通 Pro 2.x 旧版工程的源抓取链路。结合上一 commit 的 modern Pro 3.x
路径，本仓库 5/5 Pro 项目 EPRO2/dataStr 全部端到端打通。

Pro 2.x 与 Pro 3.x 是两个完全不同的存储模型：
- Pro 3.x：git-style branch + linear history chain，AES-128-GCM 加密的
  EPRO2 增量消息流，按 history 重放（已在前一 commit 打通）
- Pro 2.x：无 branch / 无 history。文档以 EasyEDA Std plaintext dataStr
  存储（同 ["DOCTYPE","SCH","1.1"] 格式），按 doc UUID 通过
  /api/v2/documents/lists 批量 GET，主体无加密，只组件库走 AES

Pro 2.x 抓取链由 HAR (tmp/prodownload3.har, 178 请求) 反推：

  GET  /api/v4/projects/<P>                     → boards: [{sch, pcb, name}]
  GET  /api/projects/<P>/ticket?uuid=&g_ticket=-1
                                                → 完整项目 manifest
  POST /api/schematic/lists {uuids:[<sch>]}     → sort: [{uuid:<sheet>}]
  POST /api/v2/documents/lists {uuids,docType:1} → schematic plaintext
  POST /api/v2/documents/lists {uuids,docType:3} → PCB plaintext
  POST /api/coppers/search {paths}              → 铺铜层
  POST /api/textpath/search {paths,project_uuid}→ 字体/文字
  POST /api/v2/resources/search {hash,project_uuid} → BLOB 图片

实现：
- crawlers/oshwhub/crawler.py:
  - fetch_pro_source() refactor 成 dispatcher，先 GET project meta
    检查 branch_uuid，null 即旧版走 _fetch_pro_legacy()，非空走
    _fetch_pro_modern()
  - _fetch_pro_legacy() 新增（按上面 9 步流程拉所有 doc + 辅助层）
  - _pro_post_json() POST helper（与 _pro_get_json 对称）
- schemas/project.schema.json: source_format enum 加 easyeda-pro-legacy
- docs/sources/easyeda_pro_source.md rev 4: §1.1 旧版 vs 新版判别表更新、
  §2.7 新增旧版抓取流程 + 实测数据

落盘约定（旧版）：
  source/ticket.json                     完整 manifest
  source/<sheet_uuid>.json               每张原理图（含 dataStr）
  source/pcb_<pcb_uuid>.json             每块 PCB
  source/coppers.json/textpath.json/blobs.json  辅助 PCB 层资源
  source/manifest.json                   索引

实测：
  立创·梁山派      editor=2.1.30, 2 sheets+1 pcb,    1.0 MB,  78 sym/191 fp/128 dev
  立创·泰山派 RK3566 editor=2.1.40, 29 sheets+1 pcb, 0.8 MB, 299 sym/524 fp/295 dev

旧版项目体量比新版小两个数量级（梁山派 1 MB vs RK3576 66 MB）—— 没有
增量 history，组件库走单独端点，本身就是当前快照。

5/5 Pro 项目终极汇总：
  X86 主板          easyeda-pro        3.2.15  7374 docs / 481 MB
  泰山派 RK3566     easyeda-pro-legacy 2.1.40    30 docs / 0.8 MB
  梁山派            easyeda-pro-legacy 2.1.30     3 docs / 1.0 MB
  220V 桌面电源     easyeda-pro        3.2.69   771 docs /  26 MB
  ESP-VoCat         easyeda-pro        3.2.91   278 docs / 7.5 MB

共 8456 docs / ~516 MB plain。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 21:59:25 +08:00

239 lines
8.0 KiB
JSON
Raw Blame History

{
"project_uuid": "7360e73de5dd428e9f29e10573f2d8ac",
"fetched_at": "2026-04-28T13:56:24.057828+00:00",
"editor_version": "2.1.40",
"boards": [
{
"sch": "6394f41146404ead95ba248444940c6c",
"name": "Board",
"pcb": "89ef5a826c7f494ab398b791825aa8f0"
}
],
"documents": [
{
"doc_uuid": "560bd0a5ad4247b99a5038770b717786",
"docType": 1,
"path": "source/560bd0a5ad4247b99a5038770b717786.json",
"size": 2607,
"sha256": "a89f642869ad5b6dfafb559786b88b2ba21d1d4a283d5852063c3f540053910a"
},
{
"doc_uuid": "afd056d94ae848c3a866181a4890394b",
"docType": 1,
"path": "source/afd056d94ae848c3a866181a4890394b.json",
"size": 1908,
"sha256": "425c60b89042564aed24293d6b1f3f3edf91bc10b2a961971d418ef654b5bef5"
},
{
"doc_uuid": "388fe5b93cd749aaae609709b3c665cc",
"docType": 1,
"path": "source/388fe5b93cd749aaae609709b3c665cc.json",
"size": 45844,
"sha256": "edb693d67dc452e26b2962f2a02782008abc1853408de73fd0f61470b652905d"
},
{
"doc_uuid": "8978aa570ee146f8bd69a5fb6c8c60dd",
"docType": 1,
"path": "source/8978aa570ee146f8bd69a5fb6c8c60dd.json",
"size": 30530,
"sha256": "4b34d756f4c43ae391c13c4e089072f9ec4b44bbb098dae313331ecc76beb9ca"
},
{
"doc_uuid": "c25fe0ff97af465ab78960c749c53db1",
"docType": 1,
"path": "source/c25fe0ff97af465ab78960c749c53db1.json",
"size": 52808,
"sha256": "4078193c938073c6101c138c05a1569ff5d59a7c839e291aff6d91a864b326c2"
},
{
"doc_uuid": "81636a72fcce4361b8fd4e9a1a9a4e9b",
"docType": 1,
"path": "source/81636a72fcce4361b8fd4e9a1a9a4e9b.json",
"size": 19318,
"sha256": "c82e0cd4124c36cb07543ecc759cb8d3dc1926d98724f8daf81d4ab00c1792d7"
},
{
"doc_uuid": "ef83fee8093541a5bda0d0e4937b59b9",
"docType": 1,
"path": "source/ef83fee8093541a5bda0d0e4937b59b9.json",
"size": 26094,
"sha256": "1697d4db2ac188cb480ab13736a9fa375918447541fde7d1492dfe70269c32b6"
},
{
"doc_uuid": "759feacd725c47b7bb422003ca237677",
"docType": 1,
"path": "source/759feacd725c47b7bb422003ca237677.json",
"size": 24578,
"sha256": "4189a39c3538dfdf52ce145fabb49993dd2dbe0a449425437e2b8e0aa08db7d0"
},
{
"doc_uuid": "918c7c2b42ce4214b72a92733b1387f0",
"docType": 1,
"path": "source/918c7c2b42ce4214b72a92733b1387f0.json",
"size": 24334,
"sha256": "ddaa09d755057221f583bf91e8a79e15ad82d54dec8b178f210fc9e29a24ee44"
},
{
"doc_uuid": "d14da6d3d80242b0b40f4968d68ba56b",
"docType": 1,
"path": "source/d14da6d3d80242b0b40f4968d68ba56b.json",
"size": 46386,
"sha256": "510eab11d8645f96784b67d0904bef5975d83a3304c19a3341fd2b6c395a30a0"
},
{
"doc_uuid": "0f73056574c0440791bef8566f4ee66f",
"docType": 1,
"path": "source/0f73056574c0440791bef8566f4ee66f.json",
"size": 10669,
"sha256": "a5140d61215f4e51290003553550b95ae799079c8a4b6fa70153ae5439154daf"
},
{
"doc_uuid": "0bc01399ab3146faa20cc538add2623f",
"docType": 1,
"path": "source/0bc01399ab3146faa20cc538add2623f.json",
"size": 8647,
"sha256": "680116ef85ff46cc11eaf2c970b2b471e99685b5dd71fbdc20e29ffd410ec118"
},
{
"doc_uuid": "18527068e7b9496f8ab210bd80badaf8",
"docType": 1,
"path": "source/18527068e7b9496f8ab210bd80badaf8.json",
"size": 49052,
"sha256": "98b3bdeb62ec920f1b383a0d7a0ab305f377c4a7e4f2e8b60d13561c21505e98"
},
{
"doc_uuid": "c21722463acd45a394e68d9065952e3d",
"docType": 1,
"path": "source/c21722463acd45a394e68d9065952e3d.json",
"size": 116026,
"sha256": "a59c84e2245e9bc71255932f1b108434ceb2cd0a458c3f5c061db151795ec6aa"
},
{
"doc_uuid": "63c0af70041349518b4cb52eee44305e",
"docType": 1,
"path": "source/63c0af70041349518b4cb52eee44305e.json",
"size": 25835,
"sha256": "e0e2519e4f09bee493c67ae8f7655a2a8f62443ca7700a4b12e4a37dd5bb3a69"
},
{
"doc_uuid": "6beb4806ff3e497194aa203be71404a4",
"docType": 1,
"path": "source/6beb4806ff3e497194aa203be71404a4.json",
"size": 29251,
"sha256": "a8e44ac4ed97ec2945f912449aae28c3cc393aa755d3490be49b171e57cae336"
},
{
"doc_uuid": "73e72a6798e14da4a7f850412fa8e2aa",
"docType": 1,
"path": "source/73e72a6798e14da4a7f850412fa8e2aa.json",
"size": 50973,
"sha256": "e1ae8dedade3e997cc18eaccbef22a3d76ad0ea59a477c7398d98ce7220986ad"
},
{
"doc_uuid": "ee2b876552f54525babb9f18b84c8412",
"docType": 1,
"path": "source/ee2b876552f54525babb9f18b84c8412.json",
"size": 823,
"sha256": "157f7cbd50c201e7ed00ad64857dbe432382f2a1ac2f5b85ce766f6c1d68c0f8"
},
{
"doc_uuid": "a4d9c80b9ed04fbfa8887614143f69c1",
"docType": 1,
"path": "source/a4d9c80b9ed04fbfa8887614143f69c1.json",
"size": 44032,
"sha256": "b7f97f221ba06028742bc2c5581069d8ba5be3d2b7807f80b7e2a86e72fefe99"
},
{
"doc_uuid": "33158668940f4e6f8fd14bec7c34d622",
"docType": 1,
"path": "source/33158668940f4e6f8fd14bec7c34d622.json",
"size": 9365,
"sha256": "c3c8691ba7b38255eae60b6d1c11353c01789df8b6bc897cf1b84bcf837cd073"
},
{
"doc_uuid": "fa9c85ce30de40c38003ef12b8723134",
"docType": 1,
"path": "source/fa9c85ce30de40c38003ef12b8723134.json",
"size": 41959,
"sha256": "d454a4a938ffcbd52ab62ebda97f7b494d15fe22a9d3cf2636386c78dcc94965"
},
{
"doc_uuid": "c6346fddfbe44e1ca6d7d6b4c797cd81",
"docType": 1,
"path": "source/c6346fddfbe44e1ca6d7d6b4c797cd81.json",
"size": 36524,
"sha256": "13bb231c7d8c1f9bf59a07f9e27eeeb9bca54854e9413ba706b9949007c0fa71"
},
{
"doc_uuid": "60c401439f0242d6b8878d50b7420df7",
"docType": 1,
"path": "source/60c401439f0242d6b8878d50b7420df7.json",
"size": 36153,
"sha256": "7d34efae54fe5667f2dd1a7c13cd28049ef2570fb0e566997d18d6448bdd1b5f"
},
{
"doc_uuid": "12a94c86b6954795a15be0ef9d904ece",
"docType": 1,
"path": "source/12a94c86b6954795a15be0ef9d904ece.json",
"size": 13383,
"sha256": "2bf73d8712dbc904cf081599db13c6d8c93bd7c83891ee36cd4fa9b144d32610"
},
{
"doc_uuid": "617ae0c3ecda4008b528d33a30c09146",
"docType": 1,
"path": "source/617ae0c3ecda4008b528d33a30c09146.json",
"size": 34656,
"sha256": "cf6f067a3227b3111fbf7e5aba962e16b17144f0786568b73b820c7eb4ad4cd9"
},
{
"doc_uuid": "1a7e93d2007948609476b6fced37f663",
"docType": 1,
"path": "source/1a7e93d2007948609476b6fced37f663.json",
"size": 25544,
"sha256": "87a80686f2b538cecd7c4c5627646f585f11caba7ffb74c40d29a20ccdc76179"
},
{
"doc_uuid": "afac947683374ce4a0617e2f378df9e8",
"docType": 1,
"path": "source/afac947683374ce4a0617e2f378df9e8.json",
"size": 13330,
"sha256": "33252f933379357b54467b13908d3d47fab986c9710e8b997ba1a9151aa85e5b"
},
{
"doc_uuid": "87e45b6fd1ab42eda99b95e6beecaa64",
"docType": 1,
"path": "source/87e45b6fd1ab42eda99b95e6beecaa64.json",
"size": 38933,
"sha256": "44b53a3e68f09e112d5a9f3129babd44f15dd48a847511f4429f577696107fa3"
},
{
"doc_uuid": "77ac252964cc4a12a4e0e9fd07851f79",
"docType": 1,
"path": "source/77ac252964cc4a12a4e0e9fd07851f79.json",
"size": 15607,
"sha256": "ebf2284abe4df5ccab1a88bb14829363a4d918c81c5b22c1b80bc6ff145862ad"
},
{
"doc_uuid": "89ef5a826c7f494ab398b791825aa8f0",
"docType": 3,
"path": "source/pcb_89ef5a826c7f494ab398b791825aa8f0.json",
"size": 782,
"sha256": "026d63e79eca3212602dfccdce6cb722b1157ceaf0ab17dcb7ad4d0858b7c971"
}
],
"structure_summary": {
"boards": 1,
"schematic_containers": 1,
"schematic_sheets": 29,
"pcbs": 1,
"symbols": 299,
"footprints": 524,
"devices": 295,
"coppers": 0,
"textpath": 0,
"blobs": 32
},
"aux": {
"blobs_count": 32
}
}
Reference in New Issue View Git Blame Copy Permalink