docs: Cloudflare Workers hosting pipeline (apex, content workers, CI, previews) (#2140)

* docs-infra: scaffold Cloudflare workers workspace (versions.json v2, matrix, toolchain)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: record full Phase-0 plan decision in workers/PLAN.md

🤖 Generated by [robots](https://vyos.io)

* docs-infra: shared content worker — asset serving, cache classes, X-Docs-Build, canary no-store

🤖 Generated by [robots](https://vyos.io)

* docs-infra: run worker script before assets; test fetch entrypoint

🤖 Generated by [robots](https://vyos.io)

* docs-infra: apex manifest loader + dispatch map + runtime binding guard (TDD)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: apex redirects (aliases, PDF, trailing-slash) + special paths (TDD)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: PDF redirect honors pdf:null and preserves query

🤖 Generated by [robots](https://vyos.io)

* docs-infra: apex UA gate — allowlist-wins, log-only AI crawlers, empty block list at launch

🤖 Generated by [robots](https://vyos.io)

* docs-infra: apex router (pipeline §3.2), themed 404/503, /kb seam, env configs + congruence test

🤖 Generated by [robots](https://vyos.io)

* docs-infra: add missing-User-Agent regression test for apex UA gate

🤖 Generated by [robots](https://vyos.io)

* docs-infra: R2-streaming preview worker — MIME map, noindex, no-store (TDD)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: preview 404 no-store + fetch handler tests

🤖 Generated by [robots](https://vyos.io)

* docs-infra: bootstrap script — binding-target workers must exist before apex deploys

🤖 Generated by [robots](https://vyos.io)

* docs-infra: apex run_worker_first, lockfile for npm ci, PDF Location from manifest

🤖 Generated by [robots](https://vyos.io)

* docs-infra: derive html_baseurl from DOCS_VERSION_SLUG with RTD fallback (canonical gate prereq)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: version picker + status banner + language scaffold (vanilla JS, TDD pure core)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: picker preserves query+hash across switch; valid breadcrumb markup

🤖 Generated by [robots](https://vyos.io)

* docs-infra: Pagefind search wrapper with runtime base-path + preview prefix handling (TDD)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: pagefind wrapper — asset-failure notice + UI stylesheet load

🤖 Generated by [robots](https://vyos.io)

* docs-infra: gate Pagefind searchbox to CF builds (RTD keeps stock search until cutover)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: deploy sanity gates — limits, critical pages, count-delta, canonical (TDD)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: hermetic gate tests via fixture versions.json

🤖 Generated by [robots](https://vyos.io)

* docs-infra: docs-build workflow — candidate/smoke/promote two-stage deploy + registry + rollback

Two-stage CF Workers pipeline: build in pinned container, assemble artifact,
sanity gates, deploy candidate, scoped pre-traffic smoke via canary apex,
promote (rollback-id capture, hostname purge, registry upload), post-promote
probe + auto-rollback. DOCS_CF_LIVE repo variable gates every docs.vyos.io
production interaction pre-cutover.

scripts/docs_gates/smoke.py adds one authorized check beyond the spec: the
version's index.html probe asserts the #vyos-search mount div is present in
the response body, guarding CI silently forgetting DOCS_VERSION_SLUG (which
would otherwise ship stock RTD search without the Pagefind gate noticing).

🤖 Generated by [robots](https://vyos.io)

* docs-infra: build docs image in-workflow with buildx cache (v4.1 — digest pin dropped)

Plan v4.1 amendment: the ghcr.io digest-pinned image does not exist (workflow
would hard-fail at the first docker step on every push). Replace the BUILD_IMAGE
env placeholder with an in-workflow docker build from docker/Dockerfile via
docker/setup-buildx-action@v3 + docker/build-push-action@v6 (context: docker/,
load: true, tags: docs-build:local, GHA cache from/to). The checked-out commit
is the pin; buildx GHA cache keeps repeat builds cheap. Sphinx-build step swaps
to docs-build:local; inner script unchanged.

🤖 Generated by [robots](https://vyos.io)

* docs-infra: apex/preview deploy workflow — canary auto, production behind environment approval

* docs-infra: apex-deploy concurrency guard (per-ref, cancel-in-progress)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: fork-safe PR preview pipeline — approval record, R2 prefixes, label consumption, cleanup

* docs-infra: nightly preview sweep — pipefail + per-prefix failure isolation

🤖 Generated by [robots](https://vyos.io)

* docs-infra: nightly canary QA — per-entry sweep + URL-parity corpus vs RTD

🤖 Generated by [robots](https://vyos.io)

* docs-infra: parity sweep scoped to CF-built versions; transport-error resilience

🤖 Generated by [robots](https://vyos.io)

* docs-infra: one-off bootstrap workflow (binding targets — runs once on this push)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: remove one-off bootstrap workflow (bootstrap complete)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: one-off canary apex + preview deploy (route targets for Task 3.6 step 2c)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: remove one-off canary deploy workflow (targets live)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: address Phase-0 CodeRabbit findings (canonical gate, error caching, registry pointer, validation)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: strengthen manifest tests (full dispatch iteration, mutation-free validate)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: address GitHub CodeRabbit review (pointer-after-probe, fail-closed sweeps, block-precedence UA gate, preview hardening)

🤖 Generated by [robots](https://vyos.io)

* docs-infra: adversarial review fixes — error no-store, probe retry, PR-list membership, preview dotted-segment

🤖 Generated by [robots](https://vyos.io)

* docs-infra: serve oversized legacy PDF from R2 via apex (spec §5 fallback)

The 1.3 PDF (29.2 MiB) exceeds the 25 MiB static-asset cap and is absent
from the legacy content Worker's build, so /_/downloads/en/1.3/pdf/ (and
the picker's PDF link) 301'd into a dead-end 404 post-cutover. Add the R2
object fallback spec §5 already documented but never implemented: a
DOCS_PDFS R2 bucket binding on the apex Worker, a manifest pdf_r2_key
field (1.3 only), and a router step ahead of version dispatch that streams
the object with its own cache class (canary/error still force no-store).

🤖 Generated by [robots](https://vyos.io)

* docs-infra: PDF R2 fallback honors Range + If-None-Match, preserves ETag

🤖 Generated by [robots](https://vyos.io)

1 files changed, 93 insertions, 0 deletions

diff --git a/scripts/docs_gates/gates.py b/scripts/docs_gates/gates.py
new file mode 100644
index 00000000..73f4b23e
--- /dev/null
+++ b/scripts/docs_gates/gates.py
@@ -0,0 +1,93 @@
+"""Deploy-blocking sanity gates (spec §7.1).
+
+Gates: file-count vs plan cap (<= 80% of 100k), per-file < 25 MiB, index.html +
+critical-page presence, page-count delta vs previous deploy, canonical URLs must
+start with the https://docs.vyos.io/en/<slug>/ prefix, declared PDF present,
+Pagefind non-empty.
+Exit 0 = deployable; exit 1 = blocked (one line per failed gate on stderr).
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+FILE_CAP = 100_000
+CAP_FRACTION = 0.8
+MAX_FILE = 25 * 1024 * 1024
+COUNT_DELTA_MIN_RATIO = 0.5  # new build must have >= 50% of previous page count
+CANONICAL_RE = re.compile(r'<link\s+rel="canonical"\s+href="([^"]+)"')
+
+
+def _fail(msgs: list[str], msg: str) -> None:
+    msgs.append(msg)
+    print(f"GATE-FAIL: {msg}", file=sys.stderr)
+
+
+def run(artifact: Path, slug: str, versions: Path, previous_meta: Path | None,
+        critical: list[str]) -> int:
+    fails: list[str] = []
+    root = artifact / "en" / slug
+    manifest = json.loads(versions.read_text())
+    entry = next((v for v in manifest["versions"] if v["slug"] == slug), None)
+    if entry is None:
+        _fail(fails, f"slug {slug} not in versions.json")
+        return 1
+
+    files = [p for p in artifact.rglob("*") if p.is_file()]
+    if len(files) > FILE_CAP * CAP_FRACTION:
+        _fail(fails, f"file count {len(files)} > 80% of {FILE_CAP} cap")
+    for p in files:
+        if p.stat().st_size > MAX_FILE:
+            _fail(fails, f"{p.relative_to(artifact)} exceeds 25 MiB")
+
+    for rel in ["index.html", *critical]:
+        if not (root / rel).is_file():
+            _fail(fails, f"critical page missing: en/{slug}/{rel}")
+
+    pagefind = root / "pagefind"
+    if not pagefind.is_dir() or not any(pagefind.iterdir()):
+        _fail(fails, "Pagefind index missing or empty")
+
+    if entry.get("pdf"):
+        expected = artifact / entry["pdf"].lstrip("/")
+        if not expected.is_file():
+            _fail(fails, f"declared PDF missing: {entry['pdf']}")
+
+    pages = [p for p in root.rglob("*.html")]
+    if previous_meta is not None and previous_meta.is_file():
+        prev = json.loads(previous_meta.read_text())
+        if prev.get("page_count") and len(pages) < prev["page_count"] * COUNT_DELTA_MIN_RATIO:
+            _fail(fails, f"page count collapsed: {len(pages)} vs previous {prev['page_count']}")
+
+    want = f"https://docs.vyos.io/en/{slug}/"
+    for p in pages:
+        m = CANONICAL_RE.search(p.read_text(errors="ignore"))
+        if m is None:
+            _fail(fails, f"missing canonical link in en/{slug}/{p.relative_to(root)}")
+            break  # one example is enough to block
+        if not m.group(1).startswith(want):
+            _fail(fails, f"bad canonical in en/{slug}/{p.relative_to(root)}: {m.group(1)}")
+            break  # one example is enough to block
+
+    return 1 if fails else 0
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--artifact", type=Path, required=True)
+    ap.add_argument("--slug", required=True)
+    ap.add_argument("--versions", type=Path, required=True)
+    ap.add_argument("--previous-meta", type=Path, default=None)
+    ap.add_argument("--critical-list", type=Path,
+                    default=Path("scripts/docs_gates/critical-pages.txt"))
+    a = ap.parse_args()
+    critical = [line.strip() for line in a.critical_list.read_text().splitlines()
+                if line.strip() and not line.startswith("#")]
+    return run(a.artifact, a.slug, a.versions, a.previous_meta, critical)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())