Write a check

A check is any program that reads one request and prints one result. bay runs it on the exact candidate, in a sandbox, and judges only the result it prints. This page names the files a repository keeps for bay, then the check contract, field by field.

What lives in .bay/

bay reads the configuration, the policy, .gitattributes and the check programs from main, so a change never judges itself.

A first check

This check runs one command on the candidate and reports it as one unit. It uses only Python's standard library, so any machine with Python 3 runs it. Commit it with the policy entry below, then test it before you merge it.

#!/usr/bin/env python3
"""A bay check: run one command on the candidate, and report it as one unit."""
import json, subprocess, sys, time

request = json.load(sys.stdin)                    # bay.check.request/1
command = sys.argv[1:]                            # the policy's command, after this script
started = time.monotonic()
try:
    run = subprocess.run(command, capture_output=True, text=True)
    verdict = "pass" if run.returncode == 0 else "product_failure"
    output = run.stdout + run.stderr
except OSError as error:                          # the tool is missing: not the change's fault
    verdict, output = "infra_failure", str(error)
print(output, file=sys.stderr)                    # people read stderr; bay reads stdout
unit = {"id": " ".join(command), "verdict": verdict, "seconds": round(time.monotonic() - started, 2)}
if verdict != "pass":
    unit["reason"] = output.encode()[-1800:].decode(errors="ignore")  # bay shows at most 2,048 bytes
json.dump({
    "schema": "bay.check.result/1",
    "exercised": request["candidate"],            # the exact commit and tree it ran on
    "plan": {"selected": 1, "completed": 1},
    "checks": [unit],
}, sys.stdout)

Name it in the policy

# .bay/policy.toml on main
[[check]]
id = "unit"
environment = "linux-x86_64"      # a runner that serves it lists it in bay.checkEnvironment
command = ".bay/check.py npm test"
adapter = [".bay/check.py"]        # every file that decides a pass, run from the base
expected_minimum = 1               # the least units the check must report
timeout_s = 600
path = ["/usr/local/bin", "/usr/bin", "/bin"]

What bay sends

bay writes one bay.check.request/1 document to the check's stdin. A check ignores a field it does not know.

What a check sends back

The check prints one bay.check.result/1 document on stdout. People read stderr; bay never does.

• A verdict is pass, product_failure, infra_failure or unknown.
• product_failure blames the change. infra_failure blames the machine or a tool.
• A reason tells the author why a unit did not pass. bay shows at most 2,048 bytes.
• A check that cannot run here prints {"schema": "bay.check.result/1", "unavailable": "<reason>"} instead.

How bay judges the result

• An exit code alone is never evidence. With no valid result, the check is unknown.
• A result for another commit or tree is rejected whole, as unknown.
• Any unit with product_failure fails the check, and the change does not merge.
• Fewer units than expected_minimum, or fewer completed than selected, is unknown, never a pass.
• A unit with infra_failure makes the check infra_failure; a reason over 2,048 bytes does too.
• unavailable keeps no record. The change waits with CHECK_UNAVAILABLE for a machine that can run it.

Checks that hold up

These habits come from checks that run every day on large repositories.

• One program, several modes: name the mode in command, such as .bay/check.py lint and .bay/check.py test web.
• List every judge in `adapter`: helpers, baselines and schemas too. bay runs them from the base.
• Cheap checks first: a failed check ends the run, so a fast lint spares the slow tests behind it.
• Wake only what a change touches: when_paths makes an app's tests required only when its paths change.
• Copy only what a check reads: paths limits the tree bay builds for it, so a large repository checks quickly.

• Name the machine's settings: each variable in env, each tool folder in path.
• Say `unavailable` when no worker can run it: a busy queue is never a pass, nor the change's fault.
• Keep secrets out of `reason`: bay writes it into Git, so redact tokens and keep the tail of the output.
• Set `expected_minimum` from a real run: the measured count minus a margin, never zero.