Doubles down on what worked in batch-50:
- dev1 (Guangzhou) is primary execution host
- Owner cap=2 for diversity
- --max-source-mb 200 to defend against X86-class outliers
- Pro 2.x deprecated-board fix is already in (commit c3cac97)
- SSH transport for dev1 -> gitea (commit 8220c99)
Candidate pool:
200 picks from A-tier (grade>=3 & like>=10) minus already-crawled 65
Remaining A-tier corpus is 2,741 (Pro 1326 + Std 1415)
173 unique authors, like median 258, grade dist 4:118 / 3:82
Estimated walltime ~25-35 min on dev1 for Step 1-4 (no attachments).
LFS increment ~2.5 GB (source only) or +10 GB if Step 5 attachments
included. Either way well within Gitea's 200 GB migration threshold.
Step 5 (attachment download) deferred — not on the critical path for
EPRO2/Std → KiCad work, can revisit when license-filtered Forge
projection demands it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Avoids HTTPS-over-lossy-link TCP cwnd issues that pinned the previous
push to ~360 KB/s for 10 min on the batch-50 Step 3-4 commit. SSH key
generated on dev1 (~/.ssh/id_ed25519), public key posted to gitea via
/api/v1/user/keys (title "dev1-guangzhou"), origin URL updated to
ssh://git@git.deepknow.site:222/Facere/FacereDataset.git.
Also documents the kernel + git side optimizations applied:
sysctl net.ipv4.tcp_congestion_control=bbr (was cubic)
git config --global http.postBuffer 524288000 (500 MB)
Note: gitea git SSH port is 222, not 22 (22 is the host sshd).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The downstream colleague's "encrypted_external" / "string old format"
projects were Pro 2.x, not Pro 3.x EPRO2. Pro 2.x ships each doc as a
JSON file whose `dataStr` is a plaintext op-stream — one JSON array per
line, e.g. `["COMPONENT","e1","",0,0,0,0,{},0]`. Different wire format
from EPRO2's binary tilde/pipe streams; same Std envelope works for
output.
- tools/epro2/std/pro2_writer.py: parses dataStr line-by-line, keys
objects by id (position 1 for most ops, OPTYPE for singletons),
extracts BBox by walking known coord positions per OPTYPE, derives
layers from LAYER ops directly (Pro 2.x almost matches Std layer
string format already). PCB blobs that are encrypted-external
(`dataStrId` URL + `iv` + `key`, no inline dataStr — Taishan PCB)
return None so the CLI skips with a message instead of stubbing.
- tools/epro2/std/__main__.py: auto-detect via manifest's
editor_version. "2.x" → Pro 2.x writer; otherwise the existing
EPRO2 replay path. CLI surface and output layout unchanged.
- docs/sources/epro2_to_std_mapping.md: adds a Pro 2.x section.
Adapter dispatches on `head.epro_format`: absent / "epro2" gets
dict-shaped objects values, "pro2" gets array-shaped values
(`[OPTYPE, arg1, ...]`). Lists the Pro 2.x-specific OPTYPEs
(FONTSTYLE / LINESTYLE / CONNECT / OBJ / REGION / DIMENSION /
STRING / TEARDROP) the EPRO2 vocabulary doesn't have.
Smoke (re-running --all on all 5 Pro projects): 191 → 222 JSON files.
Liangshan adds 3 (2 SCH + inline 5357-object PCB). Taishan adds 28
(SCH only — PCB skipped, encrypted-external; source/<uuid>.json still
keeps the dataStrId/iv/key for a later fetch+decrypt pass).
84 → 86 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Downstream came back with concrete requirements: don't pre-compute Std
shape[] tilde strings, just dump the raw EPRO2 `objects: {id: payload}`
dict and they'll write a ~100-LoC adapter on their side. Pulling the
tilde-mapping work back saves us from second-guessing positional fields
without their parser to verify against, and shortens our pcb_writer
from ~500 lines to ~40.
Output shape (Std envelope intact, just no `shape[]`):
{
"success": true, "code": 0,
"result": {
"uuid", "puuid", "title",
"docType": 3 | 1,
"components": {},
"dataStr": {
"head": {
"docType": "3" | "1",
"editorVersion": "facere-epro2/0.1 (epro2 <X.Y.Z>)",
"units": "mil",
"epro2_doc_uuid": ...,
"epro2_editor_version": ...,
},
"BBox": {x, y, width, height}, # mil
"layers": [...], # Std layer-string array
"objects": dict(doc.objects), # raw EPRO2, 1:1
"preference": {}, "netColors": [], "DRCRULE": {},
}
}
}
Per-doc spec downstream gave us:
- shape[] dropped (empty placeholder misleads adapter)
- all units mil (no mm conversion — Std canvas already declares mil)
- head.units="mil" so adapter doesn't have to guess
- BBox min/max across known x/y/startX/endX/centerX fields; adapter
can refine by walking path arrays itself
- layers[] keeps Std's 17-line default + inner SIGNAL layers actually
used (21~Inner1.., 22~Inner2..)
- empty stubs preference/netColors/DRCRULE for grep-based triage
New: docs/sources/epro2_to_std_mapping.md with the full EPRO2 OPTYPE →
Std verb table that downstream's adapter authors will copy from. Tables
include the layer-id remapping (the 5↔7 paste/mask flip, 11→10 outline,
12→11 multi, SIGNAL 15+→21+), PCB op mappings, SCH op mappings (marked
best-effort: no Std SCH samples in our corpus), and the 5-Voltage
placeholder COMPONENT → extra net flag trick. Extracted from the
previous Option-3 writer (commit fe6971f) so adapter writers don't
have to reverse-engineer it from source.
ESP-VoCat smoke: 6 PCB + 9 SCH = 15 JSON files, head.units=mil
preserved, no shape[] field present. 82 → 84 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The doc had been growing incrementally as each host got probed; reshape
it as a polished benchmark with TL;DR top, methodology section
(including safety constraints + caveats), per-host detailed tables,
final crawler settings, batch-50 walltime breakdown, and a reproduce
recipe.
Five hosts fully covered:
pro.lceda.cn API 5.0s -> 0.5s (10×)
lceda.cn doc 5.0s -> 0.5s (10×)
oshwhub detail 2.0s -> 1.0s ( 2×)
oshwhub listing 2.0s -> 1.0s ( 2×)
modules.lceda CDN 0.2s (already optimized)
Net effect on batch-50 plan: sleep total ~32min -> ~3min, walltime
~2h -> ~10-15min.
Key finding: the original 5s/req on Pro was set out of "logged-in
account is precious" caution with zero empirical evidence. Sustained
burst probe (25 distinct UUIDs at 0.5s, no recovery) showed 0/25 errors
and median latency 410ms — the caution was unjustified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ladder probe lceda.cn/api/documents/<uuid>: 5 tiers (5/2/1/0.5/0.25s)
× 9 distinct Std doc UUIDs = 45 reqs total, all 200/success. Latency
variance is dominated by payload size (Std docs span 4 KB to 4.5 MB)
not server backpressure. Same posture as Pro API.
Net effect on batch-50 estimate: Std 25 项 × 10 doc calls saved ~19
min wall time (21min sleep -> 2min sleep). Combined plan now projects
~2h -> ~10min walltime exclusive of download bytes.
scripts/probe_rate_limit.py: --host std-doc tier added. Reads doc UUIDs
from /tmp/std_doc_uuids.json (assembled by caller from any source/manifest.json
upstream_version_documents lists). Reusable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Calibrated against ladder probes on 2026-04-29. Findings in
docs/sources/probe_rate_limit_results.md.
SLEEP_PRO 5.0 -> 0.5 (pro.lceda.cn API)
SLEEP_BETWEEN 2.0 -> 1.0 (oshwhub detail/listing)
SLEEP_SOURCE 5.0 unchanged (lceda.cn Std endpoints — not yet probed)
SLEEP_PRO_CDN 0.2 unchanged (modules.lceda.cn — already optimized)
The original 5s rate for Pro API was set out of caution because Pro
requires a logged-in cookie. Empirical sustained-burst probe (25
distinct UUIDs at 0.5s sleep, no recovery): 0/25 errors, median
latency 410ms, p90 932ms. The "Pro is rate-sensitive" assumption was
wrong — server tolerates QPS=2 cleanly.
oshwhub detail HTML pages slowed from p90 6.4s at 1.0s sleep to
p90 15s at 0.5s — server queue backs up. 1.0s is the headroom-safe
water mark.
Net effect on batch-50 estimate: ~1.5h -> ~30min.
scripts/probe_rate_limit.py: rate-limit ladder probe tool. Reusable
for new endpoints (Std source still owes a probe). Designed for safety:
30s tier recovery, low rep counts on auth hosts, bail on first non-200.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two CLI gates needed before scaling Pro batch beyond top-5:
--skip-ext mp4,qt,mov (attachment filter)
Skips video extensions in attachment download. Phase 1 measurements
showed mp4+qt occupy ~54% of attachment storage. Entry still recorded
in metadata.json with skipped:ext:<token> so we can re-fetch later if
the policy changes. Honors both server-declared `ext` and filename
suffix, case-insensitively.
--max-source-mb N (Pro source size cap)
Trips inside the chain replay loop on encrypted-blob total. On trip:
raise ProjectOversizeError, wipe partial source/, append a row to
data/state/oshwhub_pro_oversize.jsonl. Lets us shortlist 50+ Pro
projects without one X86-board-class outlier (~500 MB) blowing the
LFS budget. Std and Pro 2.x legacy are not capped (both <2 MB in
sample).
Verified:
- cap=0 trips on first blob (1.2 MB), source/ wiped, state recorded
- cap=100 runs full ESP-VoCat (7.5 MB plain, 278 docs)
- skip-ext microtest: 8/8 cases (case-insensitive, declared/suffix
fallback, empty-token edge cases)
Plan + frozen candidate list for the next 50 projects:
- docs/plans/oshwhub_batch50.md
- data/state/oshwhub_batch50_candidates.jsonl (gitignore exception added)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Colleague-facing explainer at docs/sources/pro_crawl_vs_export.md.
Addresses the "I see 278 .epro2 files but my browser only downloaded
one" confusion: web download is a ZIP container (extension is a UX
choice, not a format), our crawl produces per-doc message streams.
Both carry equivalent EPRO2 data; only real gap is IMAGE/ binary
previews which we don't fetch yet.
Why per-doc and not ZIP: the ZIP path has no public endpoint —
three HARs confirm the export button fires zero HTTP requests, it's
pure client-side JSZip on data already loaded by the editor. Our
crawler hits the same chain endpoints the editor uses internally,
which delivers per-doc streams.
Log entry references the 278 vs 266 doc-count delta for ESP-VoCat
(we walk full history chain, web export is a current snapshot).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Probed listing API and learned: total field is exposed (Pro=21,202 / Std=12,493),
pageSize accepts >=1000 (full corpus = 35 requests / 71s), sort param is silently
ignored. Dump all listings via scripts/dump_listing_index.py to local jsonl so
downstream batch-selection no longer hits the API.
Why: needed quantitative anchors before scaling Pro batch beyond top-5. License
is detail-page only (~19h serial scan), so we want to filter on grade/like
*locally* first to shortlist before paying that cost. Quality-tier counts now
known: A-tier (grade>=3 & like>=10) = 2,806 across both origins.
- scripts/dump_listing_index.py: one-shot scraper, polite QPS, streams to jsonl
- docs/sources/oshwhub_listing_full.md: human-readable report with growth
trends, quality tiers, owner concentration, and storage-budget anchors
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
