Build a Python evidence gate for AI slop

#1. Introduction

A detector can’t tell whether a draft contains original thinking. A review gate can check for a clear position, evidence, and a concrete example, then flag gaps for a person to review.

In this tutorial, we’ll build a small Python command-line tool that checks those signals. It won’t claim to determine who wrote the text.

#What you’ll learn

You’ll define a lightweight editorial contract for a draft, check its Markdown sections and evidence links with Python’s standard library, and flag a few generic phrases. You’ll also run the check locally, or add it to a development workflow.

#Prerequisites

You’ll need Python 3.11 or later, a terminal, and a Markdown editor. The tool uses no third-party packages.

#Step 1 - Define what a useful draft must contain

A checker needs a contract: a small set of conditions it can verify consistently. For this example, every draft needs three sections named Position, Evidence, and Example.

The Position section should state what the author thinks. Evidence should include at least one source link. Example should give the reader a concrete case. The checker can confirm that the sections contain text and that the evidence section contains a link. It can’t decide whether the position is thoughtful, the source is reliable, or the example is genuinely useful.

That limit is deliberate. A script can flag missing signals; a person still has to judge their quality.

#Step 2 - Create a sample draft

Save this Markdown as draft.md. The heading names are part of the contract, so use them as written. You can change the contract later, but the script and drafts need to agree.

## Position

A review gate is more useful than an authorship detector because it can point
to specific gaps that a reviewer can evaluate.

## Evidence

The source describes the reported result and its context:
[Read the source](https://your-source.example/report)

## Example

For a draft about a product change, explain what changed and show one case where
a user would notice the difference.

The link is a placeholder. Replace it with a source relevant to your draft. The sample has a position, a linked source, and an example; that doesn’t make those sections good. It makes them available for review.

#Step 3 - Write the Python checker

Save the following code as evidence_gate.py. It looks for the three required headings, checks that each has content, searches the evidence section for a URL or Markdown link, and reports a short list of generic phrases.

import re
import sys
from pathlib import Path

REQUIRED_SECTIONS = ("Position", "Evidence", "Example")
GENERIC_PHRASES = (
    "many people believe",
    "it goes without saying",
    "this speaks for itself",
)
LINK_PATTERN = re.compile(
    r"\[[^\]]+\]\(https?://[^)\s]+\)|https?://\S+",
    re.IGNORECASE,
)


def get_section(markdown: str, wanted: str) -> str:
    lines = markdown.splitlines()
    content = []
    in_section = False

    for line in lines:
        heading = re.match(r"^#{1,6}\s+(.+?)\s*#*\s*$", line)

        if heading:
            if in_section:
                break
            in_section = heading.group(1).strip().casefold() == wanted.casefold()
        elif in_section:
            content.append(line)

    return "\n".join(content).strip()


def check_draft(markdown: str) -> list[str]:
    issues = []

    for name in REQUIRED_SECTIONS:
        section = get_section(markdown, name)
        if not section:
            issues.append(f"missing or empty section: {name}")

    evidence = get_section(markdown, "Evidence")
    if evidence and not LINK_PATTERN.search(evidence):
        issues.append("Evidence section has no link")

    lowered = markdown.casefold()
    for phrase in GENERIC_PHRASES:
        if phrase in lowered:
            issues.append(f"generic phrase to review: {phrase}")

    return issues


def main() -> int:
    if len(sys.argv) != 2:
        print("Usage: python evidence_gate.py <draft.md>", file=sys.stderr)
        return 2

    draft_path = Path(sys.argv[1])
    try:
        markdown = draft_path.read_text(encoding="utf-8")
    except OSError as error:
        print(f"Could not read {draft_path}: {error}", file=sys.stderr)
        return 2

    issues = check_draft(markdown)
    if issues:
        for issue in issues:
            print(f"FAIL: {issue}")
        return 1

    print("PASS: draft meets the configured checks.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The section parser stops collecting text when it reaches the next Markdown heading. That means a section’s content is limited to the area under its heading, rather than accidentally including the rest of the document.

The link pattern accepts either a Markdown link or a plain URL. The phrase list is intentionally short and editable. These phrases are prompts for review, not proof that a person or a model wrote the draft.

#Step 4 - Run the check

Run the script from the directory containing both files:

python evidence_gate.py draft.md

With the sample draft, the output is:

PASS: draft meets the configured checks.

Now remove the link from the Evidence section or delete the Example heading and run the command again. The script prints a failure for the missing check and returns a nonzero exit code. That makes it possible to use the tool in a local workflow or call it from another command that checks the exit status.

A pass only means the configured checks passed. It doesn’t mean the draft’s claims are accurate, the source supports them, or the example adds useful detail.

#Step 5 - Adapt the contract to your workflow

The section names and phrase list are policy choices, not universal rules. If your drafts use different headings, update REQUIRED_SECTIONS. If you don’t want phrase matching, remove GENERIC_PHRASES and the loop that checks it.

Keep the tool’s job narrow. A useful gate points a reviewer to a missing section, absent link, or phrase worth revisiting. It shouldn’t score originality or label a draft as AI-written. Those judgments require context the script doesn’t have.

One useful next step is to run the checker on a few drafts and adjust the contract when it flags something that isn’t meaningful to your team. The goal is a small, explainable review aid, not a verdict.

#Key takeaways