What the rule checker checks, and where every rule came from

From: Brent Bartosch
Date: 2026-08-05

You asked three things in Tuesday's call: how many checks there are, whether they are weighted equally, and which ones carry the most importance. All three are answered below. The second one has an answer you will not expect, so it is first.

How the score is actually calculated

Nothing is weighted. The score is the number of checks that passed divided by the number that ran, as a percentage. Every check counts exactly once. There are no weights anywhere in the code - pipeline/qa_gate/core.py:539 is the whole calculation:

score = round(100 * passed / total)

This matters because the call notes recorded it as "weights were AI-generated from shared documents, not yet validated." That is not the case. There are no weights to validate. Flat scoring was a deliberate choice, and whether it stays flat is a decision for you, not an assumption for me. See the decisions at the end.

Severity does exactly two things, and neither is weighting:

1. It orders the findings list so the worst thing is at the top.
2. A CRITICAL finding fails the page regardless of score. A page needs score >= 80 and zero CRITICAL findings to pass (core.py:541).

So a page can score 95/100 and still not pass. That is the only place importance is expressed today, and it is binary rather than graded.

The 80 threshold is not arbitrary: it maps to the engagement's own "80% first-pass QA" target.

How many checks: 63 across four page types

A page is only ever checked against its own page type's list. The counts differ because the team's documents specify different amounts for each: the feature and solution page templates are fully specified in the sprint SOP, while the blog workflow is still undocumented, so the blog list is deliberately short.

How well each rule is sourced, stated up front

Of the 63 checks, 11 carry a line-level citation in the code - the document name and line you can go read. The rest are covered by a statement recorded once per page type rather than on each rule, quoted in full in each section below.

This is worth saying plainly because it is the weakest part of the answer to your question. Blog and comparison rules were written against your documents and each one names its line. The feature and solution rules were written against the SOP's template tables, and the line numbers were recorded once for the whole page type instead of per rule. No value in this document was invented, but the two halves are not equally easy for you to audit, and you should know which is which before you spend time on it.

Blog

8 checks. 3 CRITICAL, 2 HIGH, 3 MEDIUM. Source: pipeline/qa_gate/blog_page.py.

7 of 8 carry a line-level citation naming the team document and line they came from.

The remaining 1 do not carry their own line reference. They are not invented - the values were read off the template tables in the document quoted below - but the line is recorded once for the whole page type instead of on each rule, so you cannot spot-check those rows the way you can the rest. Flagged here rather than left for you to find.

CANON, and what it is: - audit/data/docs/blog-qa-process.md (+ the older revision doc-2.md) is the team's Blog QA Process. It is a PROCESS document - a 3-the AI reviewer (Qwen -> Kimi -> Claude consolidation) plus a manual final-review checklist. It specifies NO page structure and NO character or element counts for blogs. - audit/data/docs/marketing-team-sprint-sops.md specifies Feature/Solution, Use Case and Industry page structures in detail. It contains NO blog section. Its "Rules for standardization" chapter (line 1474 onward) is the one page-type-agnostic part, and CTA rules there apply to any asset. - audit/data/docs/rule-book-qa-rubric.md (Qasim's rulebook) is about topical clusters, LLM citations and server-log retrieval measurement. It states no on-page blog rules at all - no meta title or description lengths, for blogs or anything else. - audit/data/docs/pmm-automation-wishlist.md line 24 puts "Create asset-specific workflows ... for blogs" at P0 in the team's OWN backlog: the blog workflow is, by their admission, not documented yet.

What the blog check deliberately does NOT check

These are omissions on purpose. Where the team's own documents are silent, no number was invented, because a number invented here would look like the team's rule forever.

• No blog H1 / meta title / meta description character ranges exist in any team doc. The 20-40 / 50-60 / 145-160 ranges are stated only for feature, use-case and industry pages (marketing-team-sprint-sops.md:64-65, 546-547, 927-928). Blog metadata is therefore presence-checked only.
• No meta-title separator convention for blogs. Live blog titles are inconsistent: 'EZO Q3 2024 Recap - EZO.io', 'The Crowdstrike Outage and the Blue Screen of Death | EZO', and 'EZO Tops 2026 Capterra & Software Advice Rankings' with no suffix (kb/ezo.io/corpus/blog/*.md front matter). MetaTitleSeparator is deliberately not in the ruleset.
• No blog slug word count. Live slugs range from 'leap-24-takeaways' to a 20-word webinar slug, and dates appear in slugs (the SOP's 'avoid dates' rule is stated for feature pages only).
• No blog section structure. The canon never says a blog has an intro, a key-takeaways box, an FAQ block, or a specific section sequence. blog-qa-process.md:63 flags 'missing summaries' and :66 'missing examples, visuals, tables, FAQs ... where they would improve understanding' - both explicitly conditional judgment calls, so they stay with the LLM-relay layer, not the rule checker.
• No word count, reading level, or paragraph-length rule. 'The article is easy to scan and read' (blog-qa-process.md:179) has no threshold behind it in any doc.
• No minimum internal-link count. :178 says opportunities are 'considered'.
• No blog FAQ count. The 8-10 FAQ rule is a feature-page rule (marketing-team-sprint-sops.md, FAQs section).
• No image / alt-text rule for blogs. The SOP states ~100 char alt text for the feature hero image only; pmm-automation-wishlist.md:43 wants alt text verified pre-publish but states no standard.
• Everything the blog QA process is actually about - factual accuracy, logical consistency, unsupported claims, redundancy, transitions, search intent - is judgment, run through their Qwen -> Kimi -> Claude relay (blog-qa-process.md steps 2-4). This ruleset is the rule checker only and does not pretend to replace that relay.

Feature page

21 checks. 10 HIGH, 10 MEDIUM, 1 LOW. Source: pipeline/qa_gate/feature_page.py.

0 of 21 carry a line-level citation naming the team document and line they came from.

The remaining 21 do not carry their own line reference. They are not invented - the values were read off the template tables in the document quoted below - but the line is recorded once for the whole page type instead of on each rule, so you cannot spot-check those rows the way you can the rest. Flagged here rather than left for you to find.

Feature / Solution page ruleset, encoded from the Marketing Team Sprint SOPs.

The feature and solution page templates share the SOP's own table values, and the line numbers are recorded against the solution page ruleset (solution_page.py, quoted in the solution section below). Same numbers, recorded once:

CANON, and what the rule checker takes from it: - Template publish order and tables: 13 sections, H1 20-40 (QA checklist line 588), meta title 50-60 (:589), meta description 145-160 (:590), hero supporting heading 40-50 (:91), hero description 200-230 (:92, :595), 3 stats (:168), stats H2 50-70 (:164), 6 benefit cards (:213), benefits H2 50-60 (:209), 5-8 logos (:263), 4-6 workflow blocks (:313), 2-4 trust badges (:136), 2-3 testimonials (:368), 3-5 resources (:425), 8-10 FAQs (:474, :619), final CTA H2 30-50 (:543). - Section 4.2 competitive context (pmm-page-brief-process-v1.md:219): "solution pages can carry a light wedge (why this category, why us) without becoming a comparison page. Keep it to one to two sentences maximum." Encoded twice: the wedge field is capped at two sentences (HIGH), and a head-to-head comparison_table section on a solution draft is CRITICAL - that is the comparison page's defining section, and a solution draft that carries one has become a comparison page.

Comparison page

12 checks. 7 CRITICAL, 3 HIGH, 2 MEDIUM. Source: pipeline/qa_gate/comparison_page.py.

4 of 12 carry a line-level citation naming the team document and line they came from.

The remaining 8 do not carry their own line reference. They are not invented - the values were read off the template tables in the document quoted below - but the line is recorded once for the whole page type instead of on each rule, so you cannot spot-check those rows the way you can the rest. Flagged here rather than left for you to find.

CANON, and what the rule checker takes from it: - Section 4.6 core principles: source-first, claim-by-claim verification, evidence discipline, honesty over aggression, externally safe. - Phase 5 (external-safety pass): every negative/competitor-gap claim must carry one validated label - "Not publicly documented", "Requires configuration / paid tier" (only if citable), or "Via integration" with the integration named - and scope must be qualified ("via CMDB", "some Elite setups") over absolute "they can't do X". - Evidence discipline: no claim ships without a citable source. Competitor numbers are quantitative claims; the citation markers accepted here are brief_gate's own (CITATION_RE / CLAIM_RE are imported, not re-invented, so page check and brief check cannot drift). - The fairness pass (Phase 7 "fairness"; the exemplar opens with "Snipe-IT is a solid, open-source ITAM tool" and carries a whole "Who Snipe-IT Is Right For (Be Honest)" section): at least one competitor strength is acknowledged. - The feature matrix ("Head-to-Head Comparison", the exemplar's 13-row capability table): a comparison page without one is not a comparison.

What the comparison page check deliberately does NOT check

These are omissions on purpose. Where the team's own documents are silent, no number was invented, because a number invented here would look like the team's rule forever.

• Phases 1-4 (internal-source grounding, competitor positioning map, review mining, line-by-line claim research) are upstream brief research, not page properties. The claim register with per-row confidence and evidence links is a brief artifact and belongs to the brief layer.
• Phase 6 synthesis (narrative spine, risks, counters, wedge pillars) and Phase 7's improvement loops and second-LLM red-flag pass are judgment and process; they live in the AI reviewer, not the rule checker.
• 'Never assert a competitor lacks something they have' is a truth check. The rule checker verifies labeling and sourcing, not whether the source is right - factual verification stays with the verification loops.
• Negative-claim detection is marker-based (NEGATIVE_RE plus cross/'No ...' cells). A competitor gap phrased without those markers, or aimed at an unnamed rival, will not be caught here; the AI reviewer reads for it.
• Our own numbers are not checked here: draftgen's grounding policy limits the draft to brief facts and FixPass verifies edits against the FactStore, so a duplicate check would drift from that source of truth.
• No character ranges or element counts: Section 4.6 states none for comparison pages (the exemplar's 13 rows is an instance, not a rule), so no CharRange/ElementCount appears and a test asserts it.
• Landmine lists ('Snipe-IT doesn't support SSO' - it does) and approved terminology are per-competitor facts from the exemplar's Landmines section, not a generic canon rule; encoding one competitor's list as everyone's would be wrong.
• TCO framing validation ('validate with Finance or product' in the exemplar) and legal/brand review before publish are human sign-offs, not machine checks.

Solution page

22 checks. 1 CRITICAL, 11 HIGH, 9 MEDIUM, 1 LOW. Source: pipeline/qa_gate/solution_page.py.

0 of 22 carry a line-level citation naming the team document and line they came from.

The remaining 22 do not carry their own line reference. They are not invented - the values were read off the template tables in the document quoted below - but the line is recorded once for the whole page type instead of on each rule, so you cannot spot-check those rows the way you can the rest. Flagged here rather than left for you to find.

CANON, and what the rule checker takes from it: - Template publish order and tables: 13 sections, H1 20-40 (QA checklist line 588), meta title 50-60 (:589), meta description 145-160 (:590), hero supporting heading 40-50 (:91), hero description 200-230 (:92, :595), 3 stats (:168), stats H2 50-70 (:164), 6 benefit cards (:213), benefits H2 50-60 (:209), 5-8 logos (:263), 4-6 workflow blocks (:313), 2-4 trust badges (:136), 2-3 testimonials (:368), 3-5 resources (:425), 8-10 FAQs (:474, :619), final CTA H2 30-50 (:543). - Section 4.2 competitive context (pmm-page-brief-process-v1.md:219): "solution pages can carry a light wedge (why this category, why us) without becoming a comparison page. Keep it to one to two sentences maximum." Encoded twice: the wedge field is capped at two sentences (HIGH), and a head-to-head comparison_table section on a solution draft is CRITICAL - that is the comparison page's defining section, and a solution draft that carries one has become a comparison page.

What the solution page check deliberately does NOT check

These are omissions on purpose. Where the team's own documents are silent, no number was invented, because a number invented here would look like the team's rule forever.

• Section 4.2's ICP requirement ('one primary ICP per solution page. The brief names the ICP explicitly and maps every section to their pains and KPIs') is a brief-layer fact plus a mapping judgment. The ICP lives in the brief, checked there; whether every section speaks to that ICP is the AI reviewer's read, not a page field the rule checker can count.
• 'Problem -> solution narrative, not feature catalogue. Features appear only as evidence for value claims' (Section 4.2) is narrative judgment. The rule checker verifies the section skeleton; it cannot tell a narrative from a catalogue.
• Proof-point fit ('favour industry-specific stats, customer logos from the target segment, and testimonials from the named ICP role') is relevance judgment. The rule checker counts 3 stats, 5-8 logos, 2-3 testimonials; segment fit stays with the AI reviewer and PMM review.
• Wedge detection is field-based: the two-sentence cap reads the dedicated top-level 'wedge' field, and the comparison boundary is structural (no comparison_table section). Competitive prose woven into the hero or workflow copy is not detected here - that needs the AI reviewer.
• Breadcrumb wording conflicts and is not resolved here: the template (section 2) mandates 'Features' as the parent category, but the solution exemplar's breadcrumb is 'AssetSonar > IT Asset Management' (feature-solution-pages-index.md:238) and live solution pages publish under /solutions/. No solution breadcrumb convention is stated anywhere, so breadcrumb content is presence-only via section order - nothing invented.
• The template's hero table says H1 20-30 (line 91) but its own QA checklist says 20-40 (line 588). The rule checker uses the checklist's 20-40 - the same value the feature page rules and the sprint SOPs use - and the conflict is flagged here rather than silently resolved.
• CTA casing conflicts exactly as it does for feature pages: the template mandates ALL-CAPS buttons (TALK TO AN EXPERT / TRY IT FOR FREE / SCHEDULE A CALL) while the standardization chapter and CTAConformance require sentence case - and the solution exemplar itself uses sentence case ('Schedule a call'). Same open conflict as FEATURE_CONFLICTS; not resolved here.
• Per-item ranges inside collections are canon-stated but not yet machine-checked, matching the feature page rules's depth: stat label 25-40 and context 80-140, benefit card title 30-35 and description 160-170, workflow description 350-360 and tab/title 30/40, testimonial quote 150-350, FAQ question 50-100 and answer 300-500 (max 560), mid-page CTA headlines 30-50, image alt text ~100. Extending CharRange over collection items is a shared feature+solution upgrade, not a solution-only rule.
• Approval-state facts are not page properties: badge currency and approval, logo usage permission, stat source documentation, exact approved testimonial quotes, and the QA checklist's 5/5 Product Accuracy / 4/5 overall scores are human sign-offs and verification-loop work, not machine checks.

The meta description question, worked through

You flagged a meta description at 162 characters showing as a "critical issue" against a 145-160 range, said it is not actually a critical SEO problem, and that you could not find the source of the range in your own documents. Three separate things there, and they have three different answers.

1. The range is real and cited, but it is not from your documents. It comes from the sprint SOP, at marketing-team-sprint-sops.md:64-65 for feature pages, :546-547 for use-case pages and :927-928 for industry pages, plus the solution page QA checklist at :590. That is why it is not in your files. It is part of the website standardisation canon, so whether it should apply at all is a conversation with Rubab rather than a code change.

2. It was never marked critical. A length overage produces a MEDIUM finding. The default severity for a character range check is MEDIUM (core.py:155), and CRITICAL fires only when the field is missing entirely. At 162 characters you would have seen MEDIUM. If something showed you the word critical, that is a presentation bug and I need to know which document or comment you were reading, because the check itself is behaving correctly.

3. Blogs already have no range at all. Blog meta descriptions are checked for presence only, deliberately, because no blog character counts exist in any team document. That is recorded in the blog omissions above.

Decisions I need from you

These are yours, not mine, and I would rather have them on the record than guess:

1. Flat or weighted? Every check currently counts once. If you want some checks to matter more, that is a real change to how the score is calculated and I would want to scope it with you rather than pick weights myself. Picking them myself is how the call notes came to say the weights were AI-generated.
2. Meta description range: keep, widen, or drop it? It is Rubab's canon, so this needs her too. If minor overages should not count against a page, the honest fix is to widen the range or lower the severity, not to leave a rule in place that everyone ignores.
3. Which of these checks should be CRITICAL? Today CRITICAL is the only way a check can outrank the others, and the current assignment came from reading your documents, not from you. The comparison page list in particular has 7 of 12 as CRITICAL, which is worth your eyes.

One thing that is already answered

You asked which model runs the fan-out query and retrieval tree prompt, since the prompt was written for GPT. It runs on openai/gpt-5.2 (pipeline/fanout.py:71, pipeline/config.py:24) - the same lineage you wrote it for, routed through EZO's OpenRouter account rather than a separate OpenAI key.

Generated by scripts/build_rule_inventory.py on 2026-08-05. Every count, threshold, severity and citation in this document is read out of the live code at build time, so re-running it after a rule change produces a correct document rather than a stale one. Nothing here was typed by hand.