What Four Measured Experiments with a System One Model Actually Showed

The inverted scanner: when 64 patterns miss what one question catches

A prompt-injection scanner built on roughly 64 regular expression patterns was fed four synthetic agent skill files. One file contained an active, multi-stage attack: an instruction override, a credential harvesting command, and an explicit instruction to conceal the exfiltration from audit logs. The regex scanner flagged it—but only because the payload happened to include an incidental phrasing habit common in raw model output.

When that single incidental phrase was reworded, leaving all three hostile behaviors completely untouched, the file scanned 100% clean. Zero detections.

At the same time, the scanner evaluated a documentation file written specifically to teach human reviewers how to recognize prompt-injection attacks. That educational document was flagged at maximum risk. The scanner’s ranking had completely inverted: the benign teaching document was treated as a critical threat, while the live, functional attack payload passed straight through.

We ran both files through TypeSafe’s System One model, Jev. The model scored the original attack and the reworded attack identically, rating both at maximum threat severity. It rated the educational tutorial as safe. A single binary probability question (noul) carried the entire distinction:

Is this text an instruction addressed to the agent reading the file, or a description of such an instruction written for a human?

The syntactic scanner parsed tokens; the typed judgment model classified intent and communicative direction. That single result (recorded in prompt-swap bead bd-wsfy.5) illustrates why System One models are attracting serious attention. But it also raises the engineering question that most launch-week commentary avoids: where does this primitive actually hold up against production code, where does it fail, and where does plain deterministic code beat it?

---

Measurement 1: Test admissibility and the structural shape trap

In our DevSecOps infrastructure repository (devsecops-infra), we enforce an explicit repository rule: a test is admissible only if it can fail when a deployment fails.

Automated code quality tools often reward the wrong properties. A unit test that merely asserts a hardcoded configuration value or mirrors a YAML schema is short, neat, cyclomatic-complexity-free, and passes linters with flying colors. A real behavioral integration test—one that spins up a harness, probes an endpoint, exercises error handling, and tears down state—is longer, uses helper functions, and frequently triggers arbitrary structural linter warnings.

To test whether Jev could distinguish real verification from superficial assertion mirroring, we selected nine real test functions from our codebase. Before submitting any query to the model, a human reviewer evaluated and pre-recorded the admissibility verdict for each function. Recording the human baseline first was essential to keep the evaluation objective.

The model was asked to score each test against the written repository standard using a score query on a calibrated rubric.

The result: nine of nine matched the human audit.

Crucially, the scores did not cluster near the boundary. The model separated the admissible tests from the inadmissible assertions with a wide, empty margin:

• Admissible behavioral tests: scored cleanly above 0.85, correctly identifying that the test assertions tracked runtime effects, exit codes, and live daemon state changes.
• Inadmissible structural tests: scored below 0.30, flagging that the test asserted only on dictionary keys or in-memory fixtures without exercising a failure mode.

The model succeeded because evaluating whether a test asserts on behavior versus shape is fundamentally a semantic classification problem. Linters see the syntax tree; Jev evaluated whether the assertion had an observable causal link to deployment success.

---

Measurement 2: Routing and the asymmetry most write-ups miss

A common pattern in agent workflows is routing a user or agent task description to the specific governance procedure, runbook, or skill designed to handle it (tested in prompt-swap bead psw-ej55).

Keyword search over documentation and skill catalogs frequently breaks down. When an engineer or an agent searches using exact technical terms (e.g., postgresql-dump-restore), a standard search index finds the entry. But when the exact same task is phrased naturally (“we need to pull a snapshot of the production database before rolling out the migration”), keyword search often returns zero hits.

We tested Jev against eight natural language task descriptions: six describing tasks covered by existing procedures, and two describing novel tasks governed by no existing workflow.

Using a choice query over the available catalog, Jev: - Correctly identified and selected the governing procedure on six out of six positive tasks with confidence at or above the 0.80 automated threshold. - Correctly abstained on the two ungoverned tasks, returning confidence scores well below 0.50.

The asymmetric error principle

In building this router, our initial implementation treated all classification outcomes symmetrically. That was an architectural mistake.

The cost of classification errors is sharply asymmetric: 1. False positive routing: If the model routes a request to procedure A when it should have gone to procedure B, an engineer or agent notices the mismatch during execution and incurs the small friction of a redirected search. 2. False negative abstention: If the model returns “nothing applies” when a mandatory governance procedure actually exists, the agent assumes the workspace is unregulated and proceeds to hand-roll an unreviewed, ad-hoc change.

Because an improper abstention can bypass safety controls, the threshold for returning “none” cannot be a simple majority probability. In our production router, a confident “none” (confidence ≥ 0.85) is accepted; an uncertain “none” automatically degrades to returning the top candidate procedures for human confirmation.

---

Measurement 3: Specification conformance on live converge logs

The most challenging evaluation we conducted (tracked in devsecops-infra bead dso-8wtyy) asked whether Jev could evaluate specification conformance against raw system execution logs.

Our infrastructure repository maintains an extensive Ansible Role and Playbook Standard comprising 111 distinct specification clauses. Of those 111 requirements, 19 describe behaviors that can be observed directly in Ansible execution output (such as task order, idempotence on second apply, and probe execution in check mode).

We fed Jev the exact task execution trace from four test scenarios involving the sifs_occmd tool installation role: 1. Clean baseline: standard playbook execution. 2. Seed A (Idempotence violation): a command task was modified to omit changed_when: false, causing the second apply to report changed=1 instead of changed=0 (violating clause RPS-073). 3. Seed B (Phase ordering violation): the installation tasks were imported before preflight verification tasks, so mutations occurred before environment readiness was checked (violating clause RPS-018). 4. CNPG repeat run: a raw log from a complex Cloud Native PostgreSQL cluster deployment where a repeat apply reported changed=16.

Across 76 total clause judgments, the model’s performance revealed both its genuine strengths and its strict boundaries:

• Seed A was caught decisively: On clause RPS-073 (“A second apply reports changed=0”), Jev returned contradicts with 0.99 probability at 0.99 confidence.
• CNPG drift was caught: In the complex PostgreSQL log, Jev flagged repeat changes (contradicts 0.90) and detected that preflight tasks performed mutations (contradicts 0.67).
• Zero false contradictions: Across all clean runs, Jev never generated a false contradiction at or above 0.50 confidence. When evidence was absent from the log, it returned insufficient at 1.00 confidence rather than hallucinating an answer.
• Seed B was completely missed: On clause RPS-018 (“Run preflight before the first mutation”), Jev returned contradicts at only 0.03 probability, completely missing that two mutating installation tasks ran ahead of the preflight block.
• One false positive conformity: On clause RPS-084 (“Run play-level verification after all roles converge”), Jev reported supports at 0.68 confidence on a playbook that completely lacked play-level verification tasks.

The verdict on specification gates

The conclusion was unequivocal: a typed judgment model cannot serve as a blocking CI gate for specification conformance.

The only violations Jev caught with high confidence were those where a simple regex over the log recap (changed=0) would have caught them anyway. When the specification required reasoning about sequential order across log entries (Seed B), the model failed. Replacing a deterministic check with a model judgment merely replaces an unverified input with an unverified opinion.

Where Jev excelled was as an advisory triage reviewer: it generated specification coverage rankings that immediately highlighted unverified clauses in our standards, pointing engineers directly to where deterministic assertions were missing.

---

What the primitives are and how they behave

Understanding where to apply these models requires looking past marketing terms like “System One” and examining the actual API primitives.

Unlike autoregressive language models (GPT-4, Claude, Gemini) that generate text token-by-token using causal attention, TypeSafe’s Jev is a non-generative, typed model. It evaluates inputs against structured questions and returns numerical probabilities:

Input State (code, logs, diff, text) + Typed Questions
                     │
                     ▼
       ┌───────────────────────────┐
       │   Jev Forward Pass (GPU)  │  Single parallel evaluation
       └───────────────────────────┘
                     │
         ┌───────────┼───────────┐
         ▼           ▼           ▼
       noul       choice       score
     (0.0-1.0)  (distribution) (ordinal)

1. noul: Evaluates a single binary proposition and returns a float representing the probability (0.0 to 1.0) that the condition holds.
2. choice: Selects one categorical label from a declared array of options, returning a complete probability distribution over all options alongside a calibrated confidence metric.
3. score: Evaluates the input against an ordered rubric of two to ten levels, returning an interpolated continuous score.

Single-pass parallel questions

In generative models, asking ten questions requires either ten separate API requests or generating a long text response that consumes output tokens and latency.

In Jev, questions are evaluated in a single parallel forward pass. You can submit twenty independent noul and choice questions alongside the input state in one payload. Adding questions increases compute negligibly compared to the base state encoding.

The vendor confidence bands

When integrating typed judgments into production systems, the vendor’s documented calibration bands provide clear operational cutoffs:

• ≥ 0.80 (Automated): High certainty. Safe for automated execution in low-risk paths (triage, label tagging, cache invalidation).
• 0.50 – 0.79 (Advisory): Moderate confidence. Useful for advisory flags, ranking candidate lists, or queuing items for human review.
• < 0.50 (Abstain): Low certainty. The model cannot reliably distinguish the outcome; the system must fall back to deterministic defaults or human intervention.

Academic grounding

This architecture reflects a broader finding in machine learning research: evaluation requires significantly less model capacity than generation.

Recent work by Li et al. (ICLR 2026, Rethinking Model Capacity in Evaluation) demonstrated that intermediate representations in compact models carry strong evaluative signals that are lost when forced through causal text generation heads. Furthermore, formal benchmarks (arXiv:2406.08660) continue to demonstrate that compact, task-focused classification architectures routinely outperform large, zero-shot generative models on domain-specific classification while using orders of magnitude less compute.

---

Field reports: what practitioners on X are actually building

Beyond our internal test suite, engineers across the community have spent launch week stress-testing Jev against different problem shapes. These reports (catalogued in devsecops-infra/docs/reference/typesafe-system-one-use-cases.md) highlight where the technology works in practice and where the boundaries lie.

1. Search reranking with user-stated plain text criteria

One of the most practical external implementations was built by Aleksandr Sarantsev (running live at jev.foglight.co), who implemented personalized search reranking where users express their relevance criteria in conversational plain text. Instead of hardcoding boolean filters (e.g., price brackets or category checkboxes), the user enters criteria like “commercial properties with owner financing and absentee management.” Jev evaluates extracted candidate snippets against the criteria in parallel, returning calibrated relevance probabilities. A similar implementation by maintainer @Xuanwo filters open GitHub issues and pull requests according to bespoke reviewer priorities.

2. Empirical cost and latency advantages

Practitioner @Braxxxx benchmarked Jev against Claude 3.5 Sonnet and Opus on a keep-or-skip classification task across 100 extracted social posts. Opus was measured at 63 times the cost and 4.7 times the latency, while Sonnet was 28 times the cost and 7.1 times the latency. Note the critical constraint: the benchmark tested classification over pre-extracted text, not document OCR or screenshot parsing.

3. Browser automation, not “computer use”

Several early reviewers attempted to use Jev for desktop automation, expecting an alternative to Anthropic’s Computer Use. Practitioners @Nav and @AVB clarified the boundary: Jev has no vision capabilities. It cannot process screenshots or mouse coordinates.

What it can do is browser automation: code extracts interactive DOM elements and their accessibility trees, and Jev executes a choice query over the candidate list to select the next action. As @AVB noted, this requires the decision space to be known a priori. Furthermore, practitioners warned against classifying DOM nodes one-by-one: candidates must be batched into a single request, or HTTP latency erases any performance advantage.

4. Calibrated confidence as a search heuristic

Stress-testing the calibration quality, researcher Aviz Maeir applied Jev to hard Sudoku puzzles. Rather than using greedy filling based on the highest single prediction, the solver used the full probability distribution returned by choice to guide a tree search, successfully cracking boards where greedy approaches deadlocked. This confirms that discarding the probability distribution throws away the model’s most valuable signal.

5. The twelve-factor agent design frame

Software architect @dexhorthy contextualized typed models within the broader 12-Factor Agents framework. Specifically: - Tools Are Structured Outputs: A tool call is simply a structured prediction; execution belongs strictly to code. - Own Your Control Flow: Business logic, retry loops, and state transitions belong in deterministic code, not delegated to an autoregressive model. - Small, Focused Agents: Replace monolithic LLM loops with pipelines of deterministic extractors, compact classifiers, and targeted agent invocations.

6. Skeptical and dissenting perspectives

Healthy engineering evaluation requires listening to the skeptics: - Practitioner @KC+AI questioned whether extension-based classifiers were already achievable with open models like GLiNER. - @Felipe Infante de Castro noted that models like qwen3-reranker already compute calibrated relevance scores with open weights. - @al'amin ai warned that a proprietary, hosted classification API is not an open world model, leaving developers dependent on closed provider weights. - @FleetingBits summarized the consensus cleanly: fast, low-cost classification is valuable in production, but marketing hype should not be mistaken for empirical verification.

7. Practitioner video teardowns and live demos from X

Beyond static commentary, practitioners published video teardowns, benchmark sessions, and integration walkthroughs demonstrating how System One models behave under real workloads:

---

The honest counterweight: where deterministic code wins

Every tool has an operational boundary. In our architecture, Jev was evaluated against the deletion test: would deleting the model leave the requested outcome unmet or unproven?

In a systematic audit of our own DevSecOps quality toolchain, 13 candidate integration sites were explicitly rejected, while only a handful were approved.

1. Mutation testing destroys model opinions

Where a deterministic oracle exists, using a machine learning model is an anti-pattern. Consider test suite quality: you could ask a model to “judge whether this test suite is rigorous.” But that judgment is merely an opinion.

In contrast, mutation testing (e.g., mutants4rs or mutants4ts) systematically inserts syntactical faults into the codebase and executes the test suite. If a mutant survives, the test suite demonstrably failed to catch broken code. That is an empirical proof, not a probability. A model has no place in the mutation loop.

2. What stays in code

Deterministic algorithms belong in native code: - Set reconciliation: determining diffs between deployed resources and Git state belongs in hash comparisons and set theory. - Arithmetic & thresholds: comparing latency percentiles, error budgets, or billing totals belongs in standard relational operators (>, <). - Graph traversal: dependency ordering and topological sorting in build systems belong in Kahn’s algorithm, not in a neural network. - Lexicon lookups: verifying exact identifiers, environment variables, or schema fields belongs in map lookups.

Replacing any of these with a model introduces latency, variance, and cost with zero engineering benefit.

3. Correcting the limits: what 255 actually bounds

It is easy to misread vendor documentation during launch week. In our own early planning notes, an engineer recorded that Jev had a hard limit of “255 questions per API call,” citing the vendor’s announcement post.

That figure was real, but it was attached to the wrong noun.

As stated directly in the vendor launch post, 255 is the maximum cardinality ceiling on the options of a single choice question. You cannot submit a choice question with 300 enumerated categories.

There is no documented maximum limit on the number of questions in a request. Instead, the real technical constraint (documented in Valyu’s practitioner guide) is a token budget: - 64,000 total tokens covering the input state and all submitted questions combined. - 32,000 tokens for the state plus the single longest question.

Checking vendor primary sources before committing architectural limits into code remains a fundamental engineering responsibility.

---

The practical implementation guide

If you are incorporating a typed System One model into an existing software pipeline, structure your integration around three battle-tested architectural patterns:

┌──────────────────────────────────────────────────────────────┐
│                    Incoming Request / State                  │
└──────────────────────────────┬───────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────┐
│ 1. Deterministic Extraction & Filtering (Code)               │
│    - Parse AST / DOM / log files                             │
│    - Strip irrelevant boilerplate                            │
│    - Enforce hard boundary rules                             │
└──────────────────────────────┬───────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────┐
│ 2. Speculative Parallel Fan-Out (Jev)                        │
│    - Submit state + all primary and conditional questions    │
│    - Parallel execution in one GPU pass                      │
└──────────────────────────────┬───────────────────────────────┘
                               │
         ┌─────────────────────┴─────────────────────┐
         ▼                                           ▼
  Confidence ≥ 0.80                           Confidence < 0.80
┌──────────────────────────────┐            ┌──────────────────────────────┐
│ 3a. Automated Code Execution │            │ 3b. Fallback Cascade         │
│     - Deterministic branch   │            │     - Human review queue     │
│     - Fast-path execution    │            │     - Frontier LLM triage    │
└──────────────────────────────┘            └──────────────────────────────┘

Pattern 1: Speculative fan-out

Because questions are evaluated in parallel at virtually zero marginal cost, do not make sequential round-trips. Submit your primary classification questions alongside downstream conditional questions in the initial request:

{
  "state": "diff --git a/services/auth.py b/services/auth.py...",
  "questions": [
    {"id": "is_security_sensitive", "type": "noul", "query": "Does this patch modify authentication, token parsing, or credential handling?"},
    {"id": "risk_tier", "type": "score", "rubric": "1: trivial docs, 2: minor fix, 3: core logic, 4: security boundary"},
    {"id": "audit_action", "type": "choice", "options": ["auto_approve", "require_peer_review", "require_secops_review"]}
  ]
}

If is_security_sensitive comes back false, code simply discards the audit_action result. You save an entire network round-trip.

Pattern 2: The calibrated cascade

Use typed models as a high-throughput filter in front of expensive resources: 1. Deterministic code: Filters out obvious cases (known static assets, passing unit tests, exact schema matches). 2. System One model: Evaluates the ambiguous middle ground. If confidence ≥ 0.80, proceed immediately along the automated path. 3. Frontier model / Human review: Only the difficult remainder (confidence < 0.80 or high-stakes sensitive actions) is escalated to an autoregressive model like Claude 3.5 Sonnet or a human engineer.

Pattern 3: Code-owned composite scoring

Never ask a model to provide a single “holistic opinion” on a complex artifact. Ask granular, independent questions, and combine their probabilities using deterministic formulas in your own code:

Final Risk = w₁ ⋅ P(unauthenticated) + w₂ ⋅ P(exposes data) + w₃ ⋅ P(breaks idempotence)

This keeps the weighting, thresholding, and business logic entirely in your codebase, where it can be adjusted, tested, and audited without retraining or re-prompting the model.

Pattern 4: Containment-gated SecOps automation

In security operations and infrastructure remediation, the boundary between automated intervention and human escalation is defined by confidence calibration. Rather than letting an unconstrained agent run arbitrary remediation commands, production incident pipelines use typed models to classify alerts against pre-approved playbooks:

Under this model, read-only telemetry enrichment runs immediately. Low-risk containment actions (such as temporary rate-limiting) execute automatically when confidence exceeds 0.85. Disruptive interventions (such as revoking API keys or isolating database nodes) halt at the confidence gate and require explicit human confirmation.

---

Conclusion: a missing primitive, not a silver bullet

The value of TypeSafe’s System One model is not that it replaces frontier LLMs, and certainly not that it replaces deterministic compilers or test runners.

Its value is that it provides a fast, inexpensive, typed bridge between messy human text and deterministic code. It replaces brittle 64-pattern regex lists with semantic intent classification. It routes natural language queries when keyword search fails. And it provides calibrated probability distributions that code can safely branch on.

Treat it as an operational component: measure its performance against your own codebase, establish human baselines before believing metrics, keep your deterministic oracles strictly in code, and always verify what the limits actually bound.

---

Sources and References: TypeSafe System One Launch Post · TypeSafe API Documentation · Valyu Practical Guide to Jev · Li et al., ICLR 2026 (Model Capacity in Evaluation) · Fine-Tuned Small Models vs Zero-Shot Generative Classifiers · Twelve-Factor Agents Architecture · Aleksandr Sarantsev Plain-Text Reranking Demo.