|
One-time static crawl of RTD-served docs.vyos.io/en/1.2/ and /en/1.3/
(spec Sec6, amended Sec15a). tools/scrub.py strips RTD platform JS,
analytics, ad units, and the RTD flyout; localizes absolute self/RTD
URLs; injects the vyos-documentation version-picker bundle.
tools/snapshot.sh drives crawl -> scrub -> picker/PDF -> parity ->
sitemap.xml generation -> per-version Pagefind indexing.
Live run: 1.2 = 113 pages, 1.3 = 152 pages, both parity-clean (no
dangling internal links), zero residual RTD/analytics markers across
all 265 pages. 1.3's PDF (29.2 MiB, over the 25 MiB Workers per-asset
cap) staged to r2-staging/ (gitignored) for the Task 4.2 R2 upload
step, not committed as a Worker asset. crawl/ scratch dir gitignored.
wget's default User-Agent triggers a Cloudflare managed challenge on
docs.vyos.io regardless of request rate; snapshot.sh spoofs a browser
UA and throttles requests (observed empirically, not in the original
brief).
|
