Knowit
29530e09d2
Topic-targeted pull from local listing index (`name OR introduction`
contains 飞控). 79 std hits in oshwhub_listing_full.jsonl, 2 already
crawled, 77 newly fetched.
dev1 (Guangzhou) walltime:
Step 1 detail scrape ~12s, Step 4 std-source backfill ~80s
(concurrency=5)
Source completeness: 73/77 with editor source, 4 are upstream
attachments-only (no editor session ever attached, source_documents=[]
is genuine — no editor_version on the SSR page either).
Crawler hardening (crawlers/oshwhub/crawler.py):
- count.{like,star,fork,views} are now `.get(..., 0)` defensive.
Listing API omits zero-valued fields for some low-activity entries
(3/77 hit this on first pass, hard-failed with KeyError 'like').
Affects rank_score, pick_top, and metadata.json metrics block.
License mix: 65% GPL 3.0, 11% Public Domain, 11% MIT, ~6% CC variants.
Transport: dev1 → SG via tar+scp (33 MB, ~3 min over lossy
cross-region link). Bypassed gitea push from dev1 because the same
6.5%-loss link tanks single-stream throughput.
35 lines
1.1 KiB
JSON
35 lines
1.1 KiB
JSON
{
|
|
"project_uuid": "21817f8fcf2d4e36b96f8e00c33a5c93",
|
|
"fetched_at": "2026-04-30T10:57:08.684279+00:00",
|
|
"editor_version": "6.4.31",
|
|
"documents": [
|
|
{
|
|
"doc_uuid": "e7d66b9e02b9457083f9ab563784a0f2",
|
|
"docType": 3,
|
|
"master": "d158f8681c8943c9b7ae993a21d3573b",
|
|
"path": "source/e7d66b9e02b9457083f9ab563784a0f2.json",
|
|
"size": 567060,
|
|
"sha256": "d5a7011ab048898e86cff342f8ea21f640beb0878917ca5fdae64938f97662a9"
|
|
}
|
|
],
|
|
"upstream_version_documents": [
|
|
{
|
|
"uuid": "e7d66b9e02b9457083f9ab563784a0f2",
|
|
"title": "M1_MH742_Adaptor_V43_ASC.PcbDoc",
|
|
"description": "",
|
|
"docType": 3,
|
|
"master": "d158f8681c8943c9b7ae993a21d3573b",
|
|
"thumb": "//image.lceda.cn/histories/d158f8681c8943c9b7ae993a21d3573b.png",
|
|
"components": {
|
|
"a29ba53807cf43cab02f2f8a6393998d": 2,
|
|
"df6becfa9964479e9d9250ab07a23ecc": 2
|
|
},
|
|
"updateTime": 1643941579,
|
|
"createTime": 1643941540,
|
|
"histories": [
|
|
"6009255517744ad5a113232591f976ad",
|
|
"d158f8681c8943c9b7ae993a21d3573b"
|
|
]
|
|
}
|
|
]
|
|
} |