ci: IS-572: re-sync ported Cloudflare Workers pipeline files with rollingclaude/cf-port-circinus

The category-1 files in this port are byte-identical copies from `rolling`.
`rolling` has since moved: [vyos-documentation#2209](https://github.com/vyos/vyos-documentation/pull/2209)
merged as `3a1c6c30`, thirteen rounds of hardening on exactly these files.

Re-take all 14 category-1 paths from `origin/rolling` via
`git checkout origin/rolling -- <paths>`, so byte-identity holds by
construction rather than by hand-editing:

  .github/workflows/docs-build.yml
  scripts/docs_gates/{gates,parity,smoke,test_gates,test_parity,test_smoke}.py
  workers/.gitignore
  workers/apex/src/{index,special,uagate}.ts
  workers/apex/test/{router,uagate}.test.ts
  workers/apex/ua-policy.json

Thirteen of the fourteen carry
[vyos-documentation#2209](https://github.com/vyos/vyos-documentation/pull/2209)
exactly — the pre-change tree was byte-identical to `3a1c6c30^` for those
paths. `workers/.gitignore` additionally picks up the one-line `test-results/`
entry from
[vyos-documentation#2212](https://github.com/vyos/vyos-documentation/pull/2212);
inert on circinus, since only the deliberately-unported `apex-deploy.yml`
writes that directory.

Deliberate exclusions are unchanged: `docs-canary-qa.yml` (cron runs on the
default branch only, so it is not ported even though
[vyos-documentation#2209](https://github.com/vyos/vyos-documentation/pull/2209)
touched it on `rolling`), `apex-deploy.yml`, and the `docs-preview-*`
workflows. `docs/conf.py` stays hand-merged and circinus-specific, with its
ReadTheDocs fallback intact.

🤖 Generated by [robots](https://vyos.io)

1 files changed, 78 insertions, 0 deletions

diff --git a/workers/apex/test/uagate.test.ts b/workers/apex/test/uagate.test.ts
index f4c16c66..1989f847 100644
--- a/workers/apex/test/uagate.test.ts
+++ b/workers/apex/test/uagate.test.ts
@@ -17,8 +17,86 @@ describe("UA gate (§3.2.1) — ships log-only for AI crawlers", () => {
   it("unknown UA → allow (fail-open for humans)", () => {
     expect(uaVerdict("Mozilla/5.0 (X11; Linux x86_64) Firefox/128.0", policy)).toBe("allow");
   });
+  it("Applebot is allowed but Applebot-Extended is logged — most-specific match wins", () => {
+    // Apple's AI-training crawler token CONTAINS the search crawler's, so plain
+    // substring matching with a fixed allow-before-log precedence let the allow entry
+    // swallow it: the AI crawler was allowed AND never logged, unlike every other AI
+    // crawler in the log list.
+    expect(uaVerdict("Mozilla/5.0 (compatible; Applebot/0.1; +http://www.apple.com/go/applebot)", policy)).toBe("allow");
+    expect(uaVerdict("Mozilla/5.0 (compatible; Applebot-Extended/0.1)", policy)).toBe("log");
+  });
+  it("Google-Extended is not a UA token — it must not sit in the UA policy at all", () => {
+    // Google-Extended is a robots.txt user-agent control token; it never appears in a
+    // User-Agent header, so an entry for it could only ever be dead weight.
+    // Compared case-INSENSITIVELY on both sides: bestMatch() lowercases every policy entry
+    // before matching, so "google-extended" would be functionally identical to the token
+    // this guard exists to keep out — but toContain() compares primitives by strict
+    // equality, so a lowercase variant would sail past a case-sensitive assertion and
+    // quietly restore the entry. Match the matcher's own case semantics.
+    const entries = [...policy.allow, ...policy.log, ...policy.block].map((e) => e.toLowerCase());
+    expect(entries).not.toContain("google-extended");
+  });
   it("block takes precedence over allow on a UA matching both lists", () => {
     const dualMatch = { ...policy, allow: ["Googlebot"], block: ["Googlebot EvilScraper"] };
     expect(uaVerdict("Mozilla/5.0 (compatible; Googlebot EvilScraper/1.0)", dualMatch)).toBe("block");
   });
+
+  // --- allow-vs-log contests. Pinned verdicts for the four UAs that distinguish every
+  // candidate rule, so a future tweak to the precedence cannot silently drop telemetry. ---
+
+  it("a UA carrying BOTH a log token and a longer allow token is logged, not allowed", () => {
+    // "GPTBot/1.0 DuckDuckBot" matches allow "DuckDuckBot" (11 chars) and log "GPTBot" (6).
+    // Under the longest-match rule the longer ALLOW needle won and the ua-log event never
+    // fired; under the original allow-before-log rule it also won. A UA presenting two
+    // different crawlers' tokens is precisely the shape worth recording, and `log` costs
+    // nothing but a log line — the request is served either way.
+    expect(uaVerdict("GPTBot/1.0 DuckDuckBot", policy)).toBe("log");
+  });
+
+  it("pinned verdicts for the four discriminating UAs", () => {
+    expect(uaVerdict("Mozilla/5.0 (compatible; Applebot/0.1)", policy)).toBe("allow");
+    expect(uaVerdict("Mozilla/5.0 (compatible; Applebot-Extended/0.1)", policy)).toBe("log");
+    expect(uaVerdict("GPTBot/1.0 DuckDuckBot", policy)).toBe("log");
+    expect(uaVerdict("Mozilla/5.0 (compatible; Googlebot/2.1)", policy)).toBe("allow");
+  });
+
+  it("a narrow allow entry no longer overrides a matched log entry — log wins outright", () => {
+    // This branch used to return "allow" when the matched allow entry strictly CONTAINED
+    // the matched log entry, so a policy could carve a narrow allow out of a broad log
+    // entry. Removed as spoofable (see the next test). Both rows are now "log", which is
+    // the safe verdict — the request is still served either way; only telemetry differs.
+    const carveOut = { allow: ["Bytespider-Search"], log: ["Bytespider"], block: [] };
+    expect(uaVerdict("Bytespider-Search/1.0", carveOut)).toBe("log");
+    expect(uaVerdict("Bytespider/1.0", carveOut)).toBe("log");
+  });
+
+  it("the removed carve-out was spoofable by quoting both tokens independently", () => {
+    // The concrete bypass. Containment was tested between the two matched ENTRIES, never
+    // against the UA's own token structure, so a request-controlled string naming both
+    // tokens separately matched allow "Bytespider-Search" and log "Bytespider", satisfied
+    // the containment test, and bought the AI crawler an `allow`. It must be logged.
+    const carveOut = { allow: ["Bytespider-Search"], log: ["Bytespider"], block: [] };
+    expect(uaVerdict("Bytespider/2.0 Bytespider-Search/1.0", carveOut)).toBe("log");
+  });
+
+  it("dropping the carve-out leaves every SHIPPED-policy verdict unchanged", () => {
+    // The vendor pair the shipped policy actually depends on runs the other way round: log
+    // "Applebot-Extended" is LONGER than allow "Applebot", so the allow entry never
+    // contained the log entry and log already won. No pair in ua-policy.json took the
+    // removed branch, so its removal is behaviour-preserving for what we ship.
+    expect(uaVerdict("Mozilla/5.0 (compatible; Applebot/0.1)", policy)).toBe("allow");
+    expect(uaVerdict("Mozilla/5.0 (compatible; Applebot-Extended/0.1)", policy)).toBe("log");
+  });
+
+  it("an entry present in BOTH lists resolves to log, not allow", () => {
+    // Equality is not containment. Listing the same token twice is an authoring error, and
+    // `log` is the resolution that cannot lose data.
+    const contradictory = { allow: ["CCBot"], log: ["CCBot"], block: [] };
+    expect(uaVerdict("CCBot/2.0", contradictory)).toBe("log");
+  });
+
+  it("block still short-circuits ahead of the allow-vs-log contest", () => {
+    const all3 = { allow: ["DuckDuckBot"], log: ["GPTBot"], block: ["EvilScraper"] };
+    expect(uaVerdict("GPTBot/1.0 DuckDuckBot EvilScraper", all3)).toBe("block");
+  });
 });