diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 83 |
1 files changed, 82 insertions, 1 deletions
@@ -1 +1,82 @@ -# vyos-docs-legacy-snapshot
\ No newline at end of file +# vyos-docs-legacy-snapshot + +Static snapshot of the legacy VyOS documentation versions **1.2** and **1.3**, +taken from the live ReadTheDocs-hosted `docs.vyos.io/en/1.2/` and +`docs.vyos.io/en/1.3/` trees before docs.vyos.io cuts over to Cloudflare +Workers (per the `2026-07-02-cloudflare-migration-fresh-spec` design, §6 and +§15a). + +## Why this repo exists (one-time nature) + +RTD's build pipeline for 1.2 and 1.3 will be retired as part of the docs +platform migration. Those two versions are LTS/legacy and no longer receive +doc changes, so instead of standing up a live Sphinx build for them on the +new platform, we take **one static crawl** of the RTD-rendered output, scrub +out RTD-specific chrome (platform JS, analytics, ad units, the RTD version +flyout), inject our own version-picker bundle so the pages stay navigable +into the new picker UI, and commit the result as a static asset tree. That +tree is served by a small dedicated Worker (`vyos-docs-legacy`, deployed by +Task 4.2) — no crawler, no Sphinx, no CI rebuild. This repo's `snapshot/` +directory is the source of truth going forward; there is no "re-crawl on a +schedule" step. + +**This is a one-time operation.** `snapshot/` is not meant to be regenerated +routinely — 1.2 and 1.3 are frozen. Re-run `tools/snapshot.sh` only if: + +- The initial snapshot had a defect (missed pages, bad scrub, broken links) + that must be corrected from the still-live RTD source, or +- RTD's hosting of 1.2/1.3 changes in a way that requires re-baselining + before RTD is decommissioned. + +Once RTD stops serving `docs.vyos.io/en/1.2/` and `/en/1.3/`, `tools/snapshot.sh` +can no longer be re-run at all — the source it crawls will be gone. At that +point this repo's `tools/` become historical record only. + +## Layout + +```text +tools/ + snapshot.sh # crawl -> scrub -> picker/PDF -> parity -> sitemap -> pagefind + scrub.py # HTML scrubbing (RTD chrome removal + URL localization + picker injection) + test_scrub.py # TDD unit tests for scrub.py +snapshot/ # committed output: en/1.2/, en/1.3/ (scrubbed HTML, static assets, + # per-version pagefind/, per-version sitemap.xml) +r2-staging/ # 1.3's PDF (29.2 MiB, over the 25 MiB Workers per-asset cap) — + # uploaded to R2 separately by the Task 4.2 deploy flow, NOT + # committed as a Worker static asset. Not committed to this repo + # either (large binary; staged locally for the R2 upload step). +crawl/ # scratch — raw wget mirror output, wiped and regenerated by every + # snapshot.sh run, NEVER committed +snapshot.new/ # scratch — build-then-swap staging dir; moved onto snapshot/ only + # on full success, NEVER committed +``` + +## Re-running the snapshot + +Prerequisites: `wget`, `curl`, `python3.11+`, Node (for `npx pagefind`), and a +local checkout of `vyos-documentation` (for the version-picker bundle files +at `docs/_static/{js,css}/version-picker.*`). + +```bash +DOCS_REPO=/path/to/vyos-documentation bash tools/snapshot.sh +``` + +`DOCS_REPO` defaults to `$HOME/GitHub/vyos-documentation`. The script is +idempotent: it wipes `crawl/` and `snapshot.new/` before every run, crawls +`/en/1.2/` and `/en/1.3/` from the **live** `docs.vyos.io`, scrubs every +mirrored HTML file via `tools/scrub.py`, copies in the picker bundle, +downloads 1.3's PDF into `r2-staging/` (validating its `%PDF-` header), +hard-fails on any dangling internal link (parity check — the crawl graph is +the source of truth since 1.2/1.3 never had a `sitemap.xml`), generates a +`sitemap.xml` per version from the crawl inventory, builds a **per-version** +Pagefind index (no cross-version search contamination), and — only once both +`1.2` and `1.3` complete every step successfully — atomically swaps +`snapshot.new/` onto `snapshot/`. A failed run therefore never leaves the +committed `snapshot/` tree partially updated. + +Run the unit tests with: + +```bash +python3.11 -m venv .venv && . .venv/bin/activate && pip install pytest +python -m pytest tools/test_scrub.py -v +``` |