1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
|
# vyos-docs-legacy-snapshot
Static snapshot of the legacy VyOS documentation versions **1.2** and **1.3**,
taken from the live ReadTheDocs-hosted `docs.vyos.io/en/1.2/` and
`docs.vyos.io/en/1.3/` trees before docs.vyos.io cuts over to Cloudflare
Workers (per the `2026-07-02-cloudflare-migration-fresh-spec` design, §6 and
§15a).
## Why this repo exists (one-time nature)
RTD's build pipeline for 1.2 and 1.3 will be retired as part of the docs
platform migration. Those two versions are LTS/legacy and no longer receive
doc changes, so instead of standing up a live Sphinx build for them on the
new platform, we take **one static crawl** of the RTD-rendered output, scrub
out RTD-specific chrome (platform JS, analytics, ad units, the RTD version
flyout), inject our own version-picker bundle so the pages stay navigable
into the new picker UI, and commit the result as a static asset tree. That
tree is served by a small dedicated Worker (`vyos-docs-legacy`, deployed by
Task 4.2) — no crawler, no Sphinx, no CI rebuild. This repo's `snapshot/`
directory is the source of truth going forward; there is no "re-crawl on a
schedule" step.
**This is a one-time operation.** `snapshot/` is not meant to be regenerated
routinely — 1.2 and 1.3 are frozen. Re-run `tools/snapshot.sh` only if:
- The initial snapshot had a defect (missed pages, bad scrub, broken links)
that must be corrected from the still-live RTD source, or
- RTD's hosting of 1.2/1.3 changes in a way that requires re-baselining
before RTD is decommissioned.
Once RTD stops serving `docs.vyos.io/en/1.2/` and `/en/1.3/`, `tools/snapshot.sh`
can no longer be re-run at all — the source it crawls will be gone. At that
point this repo's `tools/` become historical record only.
## Layout
```text
tools/
snapshot.sh # crawl -> scrub -> picker/PDF -> parity -> sitemap -> pagefind
scrub.py # HTML scrubbing (RTD chrome removal + URL localization + picker injection)
test_scrub.py # TDD unit tests for scrub.py
snapshot/ # committed output: en/1.2/, en/1.3/ (scrubbed HTML, static assets,
# per-version pagefind/, per-version sitemap.xml)
r2-staging/ # 1.3's PDF (29.2 MiB, over the 25 MiB Workers per-asset cap) —
# uploaded to R2 separately by the Task 4.2 deploy flow, NOT
# committed as a Worker static asset. Not committed to this repo
# either (large binary; staged locally for the R2 upload step).
crawl/ # scratch — raw wget mirror output, wiped and regenerated by every
# snapshot.sh run, NEVER committed
snapshot.new/ # scratch — build-then-swap staging dir; moved onto snapshot/ only
# on full success, NEVER committed
```
## Re-running the snapshot
Prerequisites: `wget`, `curl`, `python3.11+`, Node (for `npx pagefind`), and a
local checkout of `vyos-documentation` (for the version-picker bundle files
at `docs/_static/{js,css}/version-picker.*`).
```bash
DOCS_REPO=/path/to/vyos-documentation bash tools/snapshot.sh
```
`DOCS_REPO` defaults to `$HOME/GitHub/vyos-documentation`. The script is
idempotent: it wipes `crawl/` and `snapshot.new/` before every run, crawls
`/en/1.2/` and `/en/1.3/` from the **live** `docs.vyos.io`, scrubs every
mirrored HTML file via `tools/scrub.py`, copies in the picker bundle,
downloads 1.3's PDF into `r2-staging/` (validating its `%PDF-` header),
hard-fails on any dangling internal link (parity check — the crawl graph is
the source of truth since 1.2/1.3 never had a `sitemap.xml`), generates a
`sitemap.xml` per version from the crawl inventory, builds a **per-version**
Pagefind index (no cross-version search contamination), and — only once both
`1.2` and `1.3` complete every step successfully — atomically swaps
`snapshot.new/` onto `snapshot/`. A failed run therefore never leaves the
committed `snapshot/` tree partially updated.
Run the unit tests with:
```bash
python3.11 -m venv .venv && . .venv/bin/activate && pip install pytest
python -m pytest tools/test_scrub.py -v
```
|
