Characterize Failure Tokens Before You Extract a Predicate

A messy validator is unsafe until reject paths freeze. Characterization of failures still beats a pretty extract. Change one predicate only after the matrix holds. The damage pattern Happy-path tests miss most damage from refactors. Callers depend on failure shape, not structure. Exit codes remain part of the public contract. Messages remain part of operator grep habits. A tidy diff can still swallow a reject branch. A rename can still alter a stderr line. A helper extract can still change an exit code. What this method freezes This workflow freezes observable rejection before structure. It then permits one predicate extract, nothing more. Three fields define each characterization row in this method. Those fields are payload, exit code, and stderr token. Do not freeze helper names or internal layout. Do not freeze comment text or import order. Teaching fixture, not production history The next module is teaching code, not production. Treat outputs as unexecuted examples until you run them. # example_validator.py — illustrative fixture only import json import sys ALLOWED_KINDS = { " user " , " service " , " batch " } MAX_NAME = 32 def run ( argv ): if len ( argv ) < 2 : print ( " usage: validator " , file = sys . stderr ) return 2 path = argv [ 1 ] try : with open ( path , encoding = " utf-8 " ) as handle : payload = json . load ( handle ) except OSError : print ( " unreadable payload " , file = sys . stderr ) return 2 except json . JSONDecodeError : print ( " invalid json " , file = sys . stderr ) return 2 if not isinstance ( payload , dict ): print ( " object required " , file = sys . stderr ) return 1 kind = payload . get ( " kind " ) name = payload . get ( " name " ) dry = payload . get ( " dry_run " , False ) if kind not in ALLOWED_KINDS : print ( " unknown kind " , file = sys . stderr ) return 1 if not isinstance ( name , str ) or not name : print ( " name required " , file = sys . stderr ) return 1 if len ( name ) > MAX_NAME : print ( " name too long " , file = sys . stderr ) return 1 if any ( ch in name for ch in " / \\ " ): print ( " name has path chars " , file = sys . stderr ) return 1 if dry not in ( True , False ): print ( " dry_run must be bool " , file = sys . stderr ) return 1 print ( f " ok: { kind } : { name } : { int ( dry ) } " ) return 0 if __name__ == " __main__ " : raise SystemExit ( run ( sys . argv )) Enter fullscreen mode Exit fullscreen mode The fixture mixes parsing, policy, and printing on purpose. That mix is the usual local mess. Extra keys are accepted. Nested objects are not. Encode the reject matrix as data Store the matrix beside the fixture, not comments. Comments drift during later cleanup and review passes. Checked-in matrix files get grepped by the harness. { "cases" : [ { "id" : "no_args" , "mode" : "no_file" , "exit" : 2 , "stderr_token" : "usage: validator" }, { "id" : "missing_path" , "mode" : "missing" , "exit" : 2 , "stderr_token" : "unreadable payload" }, { "id" : "bad_json" , "raw" : "{" , "exit" : 2 , "stderr_token" : "invalid json" }, { "id" : "array_root" , "payload" : [ 1 , 2 ], "exit" : 1 , "stderr_token" : "object required" }, { "id" : "bad_kind" , "payload" : { "kind" : "admin" , "name" : "ok" }, "exit" : 1 , "stderr_token" : "unknown kind" }, { "id" : "empty_name" , "payload" : { "kind" : "user" , "name" : "" }, "exit" : 1 , "stderr_token" : "name required" }, { "id" : "long_name" , "payload" : { "kind" : "user" , "name" : "n012345678901234567890123456789012" }, "exit" : 1 , "stderr_token" : "name too long" }, { "id" : "path_chars" , "payload" : { "kind" : "user" , "name" : "acme/ops" }, "exit" : 1 , "stderr_token" : "name has path chars" }, { "id" : "dry_string" , "payload" : { "kind" : "user" , "name" : "ok" , "dry_run" : "true" }, "exit" : 1 , "stderr_token" : "dry_run must be bool" }, { "id" : "happy" , "payload" : { "kind" : "batch" , "name" : "nightly" , "dry_run" : true }, "exit" : 0 , "stdout_token" : "ok:batch:nightly:1" } ] } Enter fullscreen mode Exit fullscreen mode Happy path stays in the same table on purpose. A reject-only file hides accidental accept flips. One table shows both directions after each edit. Harness that records tokens, not vibes The harness below is an unexecuted example. Copy it, then run it on your tree. Do not trust pasted output as measured evidence. # reject_matrix.py — illustrative characterization harness import json import subprocess import sys import tempfile from pathlib import Path ROOT = Path ( __file__ ). resolve (). parent SCRIPT = ROOT / " example_validator.py " MATRIX = ROOT / " reject_matrix.json " SNAPSHOT = ROOT / " reject_matrix.snapshot.json " def run_case ( case ): with tempfile . TemporaryDirectory () as tmp : payload_path = Path ( tmp ) / " payload.json " argv = [ sys . executable , str ( SCRIPT )] mode = case . get ( " mode " ) if mode == " no_file " : pass elif mode == " missing " : argv . append ( str ( Path ( tmp ) / " missing.json " )) else : if " raw " in case : payload_path . write_text ( case [ " raw " ], encoding = " utf-8 " ) else : payload_path . write_text ( json . dumps ( case [ " payload " ]), encoding = " utf-8 " ) argv . append ( str ( payload_path )) proc = subprocess . run ( argv , capture_output = True , text = True ) return { " id " : case [ " id " ], " exit " : proc . returncode , " stdout " : proc . stdout . strip (), " stderr " : proc . stderr . strip (), } def token_ok ( actual , case ): if case . get ( " stderr_token " ) and case [ " stderr_token " ] not in actual [ " stderr " ]: return False if case . get ( " stdout_token " ) and case [ " stdout_token " ] not in actual [ " stdout " ]: return False return actual [ " exit " ] == case [ " exit " ] def main ( write = False ): cases = json . loads ( MATRIX . read_text ( encoding = " utf-8 " ))[ " cases " ] rows = [ run_case ( case ) for case in cases ] report = [] failed = 0 for case , actual in zip ( cases , rows ): ok = token_ok ( actual , case ) failed += int ( not ok ) report . append ({ " id " : case [ " id " ], " ok " : ok , ** actual }) if write : SNAPSHOT . write_text ( json . dumps ( report , indent = 2 ) + " " , encoding = " utf-8 " ) print ( json . dumps ({ " failed " : failed , " rows " : report }, indent = 2 )) return 1 if failed else 0 if __name__ == " __main__ " : write = " --write " in sys . argv raise SystemExit ( main ( write = write )) Enter fullscreen mode Exit fullscreen mode Token checks beat full-string equality for this fixture. Usage lines may gain a path later. Exit codes still must match exactly. Numbered workflow 1. Inventory reject classes from current behavior List every reject class the module currently emits. Use stderr tokens and exit codes as labels. Do not inventory intended design from memory. Inventory what HEAD prints today. python3 example_validator.py echo $? python3 example_validator.py /tmp/missing.json echo $? printf '{' > /tmp/bad.json python3 example_validator.py /tmp/bad.json echo $? Enter fullscreen mode Exit fullscreen mode 2. Turn each class into a tiny payload Turn each class into a JSON fixture file. Keep each payload tiny and obviously invalid. One payload per class keeps later diffs readable. Combined payloads hide which check moved. 3. Run the harness once against HEAD Run the harness on the current tree first. Save the report as the locked snapshot. python3 reject_matrix.py --write git add reject_matrix.json reject_matrix.snapshot.json example_validator.py git commit -m "Lock reject tokens before predicate extract" Enter fullscreen mode Exit fullscreen mode Uncommitted snapshots do not protect later diffs. A dirty matrix is not a gate. 4. Classify each row before touching code Use the table below as a stop rule. Skip extracts that need new behavior. Observed row Safe next edit Unsafe next edit Token and exit stable Extract one boolean predicate Rename stderr text Extra keys already accepted Leave extra-key policy untouched Add a surprise denylist dry_run rejects strings Keep bool check in the same place Coerce "true" to boolean Path chars rejected Extract has_path_chars only Expand to Unicode slash folds Happy row still prints ok: Keep print format unchanged Switch to JSON logs Missing file returns 2 Keep IO errors as 2 Merge IO errors into 1 If the wanted change is policy, stop this workflow. Characterization protects behavior, not product intent. Policy needs an explicit spec, not a snapshot. 5. Extract one predicate, not a layer Extract one boolean helper, not a new layer. Keep print statements and return codes in place. def has_path_chars ( name ): return any ( ch in name for ch in " / \\ " ) Enter fullscreen mode Exit fullscreen mode Wire it in one call site only. Do not move JSON loading in the same patch. Do not move exit-code mapping in the same patch. if has_path_chars ( name ): print ( " name has path chars " , file = sys . stderr ) return 1 Enter fullscreen mode Exit fullscreen mode That is the whole structural change. Anything larger is a second change. 6. Re-run the matrix as a byte gate The snapshot must match on tokens and exits. Any stderr drift is a failed extract. python3 reject_matrix.py python3 - << ' PY ' import json from pathlib import Path live = json.loads(Path("reject_matrix.snapshot.json").read_text()) # Re-write live only after a deliberate matrix update. print("locked_rows", len(live)) PY git diff --exit-code -- example_validator.py reject_matrix.json Enter fullscreen mode Exit fullscreen mode If the helper extract is correct, only the validator file changes. Matrix files stay quiet. Quiet matrix files are the passing signal. 7. Stop after one green extract Do not chain a second extract in the same change. One green matrix is the whole change. A second extract hides which edit broke a token. Split commits keep the bisect cheap. Where a free model belongs A coding model can propose the predicate extract. It cannot own the reject matrix. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that pair to draft the helper on a scratch tree after HEAD is locked. Keep the matrix runner on the repo you actually ship. Treat model output as a patch candidate. Merge only when reject_matrix.py reports zero failures. Prompt the model with the frozen table, not with vibes. Paste the one helper boundary. Forbid stderr edits in the same request. Propose a patch that extracts has_path_chars only. Do not change stderr strings. Do not change exit codes. Do not touch JSON loading. Stop after one helper. Enter fullscreen mode Exit fullscreen mode If the patch edits more than the predicate, discard it. Breadth is the usual failure mode here. Failure analysis the matrix still misses This table does not cover locale. It does not cover concurrent writers. It does not cover symlink tricks on the payload path. Unicode homoglyphs can still pass the path check. MAX_NAME counts Python characters, not graphemes. Extra keys remain untested policy. dry_run missing is not the same as false . The fixture treats missing as false today. Record that fact as a row if callers depend on it. Add that row only as a new commit. Expanding the matrix is not an extract. Mixing both hides the cause of a miss. Commands for a disposable loop Use a throwaway clone for model drafts. Keep the locked snapshot in the real clone. git clone --local . /tmp/validator-scratch cd /tmp/validator-scratch python3 reject_matrix.py --write # apply a model patch here, then: python3 reject_matrix.py Enter fullscreen mode Exit fullscreen mode Compare exits between clones with the same matrix file. Different tokens mean the patch is not small. Delete the scratch clone after the gate fails or passes. Limitations This method does not replace typed contracts. It does not replace fuzzing. It does not replace a real parser test suite. Token matching can hide extra stderr noise. A second line can appear and still pass. Tighten tokens only when operators grep exact lines. Subprocess tests miss in-process monkeypatches. They also miss import-time side effects. If loading the module prints, add an import row. The workflow assumes deterministic stderr. Timestamps in errors break snapshots. Strip clocks before you freeze tokens. Who should not use this approach Do not use this as an authorization test. Do not use this as a crypto parser test. Do not use this as a privacy boundary test. Teams without a committed snapshot should not start extracting. Teams changing product policy should not hide behind characterization. Teams that cannot run subprocesses locally should pick in-process tests. If the validator already has stable unit tests, use those first. A reject matrix is for untested mess. It is not a badge for clean modules. Closing rule Freeze failure tokens, then extract one predicate. Re-run the matrix. Stop. If a scratch model loop helps, keep it behind that gate. The matrix stays the authority, not the diff.