Write agent guidelines for custom failure reasons

When automatic failure categorization is enabled in Agents > Settings, the mabl agent assigns a failure reason to every failed run. The Agent guidelines field on a custom failure reason is how you tell the agent when to choose that reason. This article explains how the agent uses your guidelines and offers recommendations for making them effective.

How the agent uses your guidelines

For every failed run, the agent infers a root cause from the evidence it can gather, including the failure-state screenshot, console and network logs, the assertion that failed, the test's results across recent runs, correlated deployment events, and the test definition itself. It then builds a list of every enabled failure reason in your workspace, both default and custom, reviews each custom reason's agent guidelines, and assigns the single reason that best matches.

Three principles follow from this:

• The agent decides from the root cause, not the wording of the error. The same surface error can have completely different root causes — a test that fails with "element not found" could be a regression, a stale selector, or a timing issue. The agent investigates the evidence behind the error, not the error text.
• The agent always assigns exactly one reason. If nothing fits, it falls back to the default "Other issue." A custom reason is chosen only when your guidelines make a clear case for it. Vague guidelines, such as "assign for unusual failures", give it nothing to act on.
• Guidelines inform the decision, but don't force it. The agent weighs your guidelines against all of the evidence, and the reason it picks is a suggestion that anyone triaging the run can change. When the agent picks the wrong reason, sharpen the guidelines and it improves on the next run.

Write an effective guideline

The best guidelines read like a short decision rule for someone triaging the failure:

1. Lead with when to assign it — start with "Assign when…" or "Assign for…".
2. List the concrete signals — the specific evidence that points to this reason, such as an error code or vendor domain in the network log, a message in the screenshot, or a pattern across recent runs. Don't just restate the description ("Assign when there's a vendor outage" tells the agent nothing new), and don't lean on the error text alone — many root causes produce the same "timeout." Describe the evidence and the root cause it reveals.
3. Disambiguate from the closest default reason — say "Prefer this over Environment issue when…" so the agent doesn't default to a reason that superficially matches. Default reasons are listed in failure reasons.
4. Ground it in evidence the agent actually has — the sources listed above. Don't reference information the agent can't see, such as your feature-flag dashboard or a Jira ticket.
5. End with a concrete example — one sentence describing a failure that should get this reason.

Stay within the 1000-character limit. Shorter is usually sharper.

Examples

Third-party vendor outage — a failure caused by an external service you depend on, not your own app:

Assign when the failure traces to a named third-party service the app embeds or calls — a payment gateway (Stripe, Braintree), an identity provider (Okta, Auth0), or a maps/chat widget — rather than our own code or infrastructure. Signals: the network log shows a 5xx or timeout on a request to the vendor's own domain, or the failure screenshot shows a vendor error message ("Payment provider unavailable") while the page and the test's earlier steps otherwise worked. Prefer this over Environment issue, which is for our own app or infrastructure being broadly down, and over Network issue, which is a transport-level failure reaching our app itself. Example: checkout fails at the "Pay" step with a 503 from js.stripe.com in the network log, while the earlier cart and shipping steps in the same test passed.

Test data problem — the test and the app are both fine, but the data the test relies on was stale, missing, or already used:

Assign when the test logic and the app are both correct but the data the test depends on is missing, stale, or already consumed: a login with a locked or expired account, a coupon or gift card already redeemed, a search for a record that was deleted or is out of stock, or an amount that no longer matches a shared fixture. The failure screenshot typically shows the app behaving correctly and rejecting the input (e.g. "This code has already been used"), not a broken page. Prefer this over Test implementation issue, which is for stale selectors or wrong assertions in the test itself, and over Regression, which is for the app's own behavior changing. Example: a promo-code step fails because the single-use code was consumed by an earlier run, and the app correctly shows "Coupon expired".

Feature flag or experiment — an A/B experiment or gradual rollout served a variant the test didn't expect:

Assign when the failure is caused by an A/B experiment or a gradual feature rollout serving a different variant than the test expects, not by a permanent app change. Signals: the same test passes and fails across recent runs with no correlated deployment; the failure screenshot shows a different-but-valid layout, copy, or an extra element (not a broken or half-loaded page); or the network log shows an experiment-assignment call (LaunchDarkly, Optimizely, Split) or a variant cookie indicating this session was bucketed into a non-default variant. Prefer this over Regression, which is a permanent change that begins at a specific deploy and keeps failing, and over Timing issue, where the screenshot shows the expected UI that simply had not finished loading. Example: a signup test intermittently fails to find the "Continue" button because half of sessions get an experiment that renames it "Next".

Learn more

• Failure reasons — default and custom failure reasons, and the fields on a custom reason.