Custom classifiers

Every team has rules that exist only in people's heads. "Never touch the billing service without a second reviewer." "We don't hand-edit generated files." "Anything touching the auth middleware is high-risk, period." The rules are real, but nothing enforces them except memory and code review vigilance.

Custom classifiers turn those rules into checks. You describe what to look for in plain English — no code, no DSL — and Firetrail runs your check on every pull request alongside the built-in ones.

Creating one

From your project's Custom Classifiers page, create a classifier with three things:

• a name — what shows up on PRs (e.g. "Hardcoded Secret Scanner"),
• a description — a note for your team about why this check exists, and
• a task prompt — the instruction itself, in your own words.

A good task prompt reads like something you'd tell a new teammate:

"Flag any PR that adds a hardcoded credential, API key, or token — anything that looks like a secret committed into the repo."

"Flag changes to files under app/services/billing/ — that area has caused production incidents and needs a careful review."

"Flag raw SQL string interpolation. We use parameterized queries everywhere."

On each PR, your classifier reports a verdict and its evidence, just like a built-in check. Each classifier also tracks its own stats — how many PRs it has run on and how many it flagged — so you can tell whether it's earning its keep. You can edit, pause, or delete one at any time; edits apply to future analyses.

Advisory by design

Custom classifiers show their findings on every PR, but they never change the Future Debug Cost. Only built-in checks move the score.

This is deliberate. The score means the same thing on every Firetrail project — a 62 is a 62. If custom checks could add points, every team's scores would drift apart and comparisons (including your own history) would quietly stop meaning anything. So your checks get full visibility where reviews happen, and the headline number stays universal.

Cost and limits

Each active custom classifier is one extra AI call per analysis, which shows up on Cost & usage like any other check.

How many you can have depends on your plan: Pro includes 10, Business has no limit. (Free and Starter don't include custom classifiers.)