# Strukto (full) > Strukto is the structured layer for AI agents: a unified virtual filesystem workspace that mounts data sources, services, and tools behind one bash interface. This file contains the full text of our published research and news posts for AI crawlers and LLM context. ## Company - Homepage: https://www.strukto.ai/ - Research index: https://www.strukto.ai/research - News index: https://www.strukto.ai/news - About: https://www.strukto.ai/about ## Founder Zecheng Zhang — Founder (Stanford, ex-AWS, YC alumni). Background: ML systems, relational databases, structured data, LLM evaluations, AI agents. LinkedIn: https://www.linkedin.com/in/zechengzhang/ — X: https://x.com/zechengzh ## Contact - Booking: https://cal.com/strukto --- ## Research # SEAR: Schema-Based Evaluation and Routing for LLM Gateways **URL:** https://www.strukto.ai/research/sear **Published:** 2026-04-23 **Venue:** The ACM Conference on AI and Agentic Systems **Authors:** zecheng, andrew-zheng, lawrence-xu **External links:** CAIS: https://www.caisconf.org/program/2026/papers/sear-schema-based-evaluation-and-routing-for-llm-gateways/ — arXiv: https://arxiv.org/abs/2603.26728 ## Abstract Evaluating production LLM responses and routing requests across providers in LLM gateways requires fine-grained quality signals and operationally grounded decisions. We present **SEAR**, a schema-based evaluation and routing system for multi-model, multi-provider LLM gateways. SEAR defines an extensible relational schema with cross-table consistency links and around one hundred typed, SQL-queryable signals covering context, intent, response characteristics, issue attribution, and quality scores. To populate this schema reliably, SEAR proposes self-contained column instructions, in-schema reasoning, and a multi-stage judge pipeline that produces database-ready structured outputs. Combined with gateway operational metrics, these records enable flexible SQL-based analysis, diagnosis, and routing recommendations over production traffic. Across thousands of production sessions, SEAR achieves strong signal accuracy on human-labeled data and supports practical routing decisions, including large cost reductions with comparable quality in offline replay. ## Motivation Existing approaches to LLM-as-judge evaluation fall into four broad categories, each with well-known limitations: - **Unstructured / free-text judges** produce commentary that is difficult to aggregate across sessions. - **Single-score evaluators** collapse all quality dimensions into one rating, preventing drill-down into specific failure modes. - **Rubric-based evaluators** apply a fixed, coarse scoring scheme that does not decompose into per-signal diagnostics. - **Template-based pipelines** fragment across teams and store untyped score–reason pairs that are hard to aggregate or compare. Routing faces a parallel challenge. Learned routers optimize an objective and return a recommendation, but their decisions remain black-box, with no per-signal explanation for why a given model suits a given task. Production teams must also trade off provider, cost, latency, and throughput explicitly, and typically prefer asynchronous policy refresh with offline validation over opaque per-request routing. ## System Overview ![SEAR system architecture and database schema](/research/sear/cais-1.png) A central LLM gateway sits between applications and providers. Every request logs operational metrics (latency, time-to-first-token, token counts, cost, cache usage, error type) to a `gateway_metrics` table. A configurable fraction of traffic is sampled off the serving path and sent to the **SEAR judge**, a reasoning-LLM pipeline that writes structured signals into four semantic evaluation tables: - **`context_info`** (45 columns): request-side context and user intent, including language, domain, task type, and requirements like tool use, code, or multi-step reasoning. Includes 20 static features (modality flags, message and token counts) derived directly from raw logs. - **`llm_response_info`** (19 columns): what the model actually produced, including tool invocation, code generation, reasoning behavior, and refusals. Overlapping dimensions with `context_info` enable gap analysis ("code requested but not produced"). - **`issue_attribution`** (20 columns): for each shared dimension, attributes gaps to their likely source: user input, context, model behavior, or mixed causes. - **`evaluation`** (31 columns): ordinal severity per signal and overall quality dimensions (relevance, completeness, coherence, instruction following, factual accuracy, safety, overall quality). All columns are typed: booleans, categorical enums, or ordinal enums with explicit level definitions. Integer and floating-point scores are deliberately avoided, since prior work shows LLM judges cluster and degrade on wide numeric scales (e.g., 90 vs. 93). ### Cross-table Design The four tables mirror one another along semantic dimensions. For a signal family such as `tool_call`, the schema records whether tool use was _required_, whether it was _produced_, _who is responsible_ for any gap, and _how severe_ the gap is. Two properties follow directly from this design: 1. **Consistency checks.** Disagreements across linked columns surface as SQL join violations. Flagged records can then be re-judged with a stronger model or filtered out. 1. **Signal traceability.** Each signal can be traced through all four tables, from request to response to attribution to severity. ## Schema-Conforming Judge Pipeline ![Multi-stage LLM judge pipeline](/research/sear/cais-2.png "small") Generating a hundred-column structured output in a single call is unreliable. SEAR addresses this with three design choices: 1. **Self-contained column instructions.** Each column description specifies its definition, evidence scope (which messages to inspect), value-assignment rules, examples, and edge cases that separate it from neighboring columns. This reduces inter-column interference during generation. 1. **In-schema reasoning.** Rather than a separate chain-of-thought call (which doubles the number of LLM invocations from four to eight for the full schema), SEAR places a temporary `reasoning` text field as the first property of the JSON schema. Because generation follows schema order, the model emits reasoning before the signal columns in a single autoregressive pass; the field is dropped before database insertion. The reasoning prompt is structured as a self-check: identify the task, derive signals step by step, and verify consistency. 1. **Multi-stage pipeline.** Tables are generated in foreign-key order (context → response → attribution → evaluation), each stage receiving the conversation plus all upstream structured outputs. Each call emits 19–31 columns instead of roughly one hundred, improving generation stability. ## Data-Driven Evaluation Because evaluation signals and gateway metrics live in the same queryable layer, downstream analyses reduce to standard SQL: - **Model evaluation** joins issue-attribution and severity columns from the evaluation tables with the gateway table to compare candidate models along any task-type or domain slice (e.g., coding tasks over the last thirty days). - **Provider evaluation** ranks providers on task quality with median latency as a secondary criterion, for any workload. - **User evaluation** aggregates user-side risk indicators (safety-sensitive content, ambiguous instructions, noisy context) to trigger guardrails or adjust sampling rates. Because records are timestamped, the same queries can be windowed to track temporal trends and detect model or provider drift. ## Data-Driven Routing The same records also drive routing. Rather than a black-box per-request classifier, SEAR derives policy updates from accumulated observations: - **Model routing.** For a given slice and quality–cost trade-off, recommend the cheapest model whose aggregate quality is within a target margin of the best-performing model (or that outperforms a currently deployed baseline). - **Provider routing.** For a given model, select providers whose quality is within 5% of the best and rank them by median time-to-first-token. This asynchronous pattern matches production practice: offline recommendations that can be reviewed, replayed, and validated before deployment, rather than opaque per-request decisions on live traffic. ## Experimental Results We evaluate SEAR on **3,000 production sessions drawn from three organizations (A, B, and C)**, with distinct workload profiles (multilingual, roleplay, and translation-heavy, respectively). 300 sessions (100 per organization) are held out and fully human-annotated across all semantic evaluation columns by two senior engineers. ### Judge Accuracy Across six configurations (GPT-5-mini and GPT-5.2 at low/high reasoning effort, with and without in-schema reasoning), the best configuration — **GPT-5.2 at high reasoning effort with in-schema reasoning** — achieves: - Error rate **0.0851**, Hamming loss **0.0794** - Boolean accuracy **95.5%**, boolean micro-F1 **0.899** - Categorical accuracy **93.4%** Broken down by table, boolean-signal accuracy exceeds 91%, categorical-signal accuracy exceeds 92%, and ordinal-signal accuracy ranges from 80% to 86%. ### Routing Case Study For Organization C, where most traffic falls into a simple-complexity slice currently served by `claude-haiku-4-5` ($1.00/M input, $5.00/M output), SEAR's routing query ranks candidates by composite quality (the sum of six ordinal quality signals, maximum = 18): | Model | Quality | $/M in | $/M out | | --------------------------- | --------- | ------ | ------- | | gemini-2.5-flash-lite | **17.57** | 0.10 | 0.40 | | claude-haiku-4-5 (deployed) | 17.00 | 1.00 | 5.00 | | grok-4-1-fast | 16.86 | 0.20 | 0.50 | | qwen3-80b | 15.66 | 0.15 | 1.20 | Replaying 100 sessions with the top-ranked candidate and comparing outputs pairwise against the deployed model yields **72 ties, 12 wins for the routed model, and 16 wins for the original**, indicating effectively tied quality at **90% lower input cost and 92% lower output cost**. This result should be interpreted as a targeted case study rather than a definitive benchmark — one organization, one workload slice, and an offline replay of 100 sessions. Even so, it demonstrates that SEAR-derived queries can identify substantially lower-cost candidates without clear quality degradation, and that logged traffic is sufficient to drive offline policy updates even with limited data. ## Full Schema ![Full SEAR evaluation schema with foreign-key relationships](/research/sear/full_schema.png "small") ## Discussion Co-locating evaluation signals and gateway metrics in a single SQL-queryable data layer enables a closing feedback loop: SEAR signals accumulate, routing policies are refreshed from those signals, routed traffic generates new signals, and quality estimates improve over time. Because judging runs asynchronously on sampled traffic off the serving path, stronger (and slower) judge models can be used without affecting request latency. We view the combination of a typed relational schema, schema-conforming generation via in-schema reasoning, and foreign-key-ordered multi-stage decomposition as a principled foundation for flexible, interpretable LLM evaluation and routing in production gateway settings. --- ## News # Optimizing S3 for AI Agents That Use Object Storage as a Filesystem **URL:** https://www.strukto.ai/news/mirage-s3-agent-filesystem **Type:** News **Published:** 2026-05-13 **Authors:** zecheng **Categories:** AI Agent, Virtual Filesystem, Agentic System, Open Source, S3, Cache, S3 Versioning **External links:** Docs: https://docs.mirage.strukto.ai — Snapshot & Rollback: https://docs.mirage.strukto.ai/home/snapshot — GitHub: https://github.com/strukto-ai/mirage — X: https://x.com/zechengzh/status/2054671432245035144 — LinkedIn: https://www.linkedin.com/posts/zechengzhang_s3-aiagent-agenticsystem-activity-7460434416453128192-HQUd — Medium: https://medium.com/@strukto-ai/optimizing-s3-for-ai-agents-that-use-object-storage-as-a-filesystem-b053134bf84f — Substack: https://strukto.substack.com/p/optimizing-s3-for-ai-agents-that S3 is one of the most useful places to store logs, datasets, exports, media, and model artifacts. It is durable, cheap, and widely supported. But S3 is not a filesystem. It is object storage. That difference matters when an AI agent is trying to inspect data, repeat a run, or recover the exact bytes it saw earlier. An agent usually does not want to think in buckets, continuation tokens, SDK clients, retries, byte ranges, and checkpoint files. The model already has a strong prior for paths and shell commands. It wants to ask simple questions: ```bash ls /s3/logs/2026/ head -n 3 /s3/events.jsonl grep error /s3/app/*.jsonl wc -l /s3/export/users.csv ``` That is the gap Mirage tries to close. ## The Simple Idea Mirage mounts S3 into a virtual filesystem path, then lets the agent use normal shell commands against that path. ![Mirage S3 diagram: an agent issues shell commands, Mirage routes them through an S3 mount, cache, and streaming reads that stop when the command is satisfied.](/mirage-s3-light.svg) The agent sees files and directories. Under the hood, Mirage turns those requests into S3 operations: | Agent asks for | Mirage maps it to | | ---------------------------- | --------------------------------------- | | `ls /s3/logs/` | S3 prefix listing | | `stat /s3/file.jsonl` | object metadata | | `head -n 3 /s3/events.jsonl` | a streaming preview that can stop early | | `grep error /s3/*.jsonl` | command dispatch over matching objects | | `cat /s3/data.csv` | byte read, stream, or cached file read | ## Why Cache Matters Agents behave adaptively. They list a directory, inspect one promising file, change plans based on what they find, check nearby metadata, preview a few lines, then branch into the next command. Without a cache, every small question becomes a remote request. On S3 that means extra latency, pagination, and repeated downloads of the same bytes. Mirage separates two kinds of reuse: - **Index cache:** directory listings, object metadata, and known missing paths. - **File cache:** file bytes that were already read by earlier commands. That split matters because `ls` and `stat` should not need to download object bytes. A repeated `head` or `grep` should not need to fetch the same content again either. ## Streaming as the General Path Many agent commands only need a slice of a file. `head -n 3` does not need a whole log archive. It needs enough bytes to find three newline boundaries. A byte-counted preview such as `head -c 200` has an even clearer stop condition. The general primitive here is streaming with a command-level stop condition. Mirage can read an object as a stream, feed it into the command, and stop consuming once the command is satisfied. That is a semantic optimization at the command-to-storage layer. Mirage does not need to understand the business meaning of the data. It only needs to understand that `head` can stop after enough lines, `wc` can count as bytes stream through, and a pipeline can stop when the downstream command is done. For `head -n 3`, the stop condition is three newline boundaries. For `head -c 200`, it is 200 bytes. For `grep payment_failed /s3/events.jsonl | head -n 20`, the downstream `head` stops the pipeline once it has 20 matching lines. That is what makes shell composition work: ```bash cat /s3/events.jsonl | grep payment_failed | head -n 20 ``` Streaming is the right default because it handles pipes, unknown file sizes, arbitrary line lengths, and commands that process data progressively. That makes it more general than planning a fixed byte range up front. A byte range is useful when the caller already knows the exact byte bounds it wants. In a pipeline like `cat /s3/events.jsonl | grep payment_failed | head -n 20`, AI agents do not know where the first 20 matching lines are before the command starts. The right behavior is to stream through the object, let `grep` filter progressively, and stop when the downstream `head` is satisfied. ## What Mirage Does Today Mirage optimizes streaming based on command semantics. It streams data into commands and lets each command stop when it has enough output. For example, `head -n 3` should not behave like `cat`. Mirage reads only until the command has the three lines it asked for, then stops. The optimization stays below the interface. The agent writes normal shell commands; Mirage decides how much of the S3 object to read and when to stop. ## What the Unoptimized Path Costs The direct S3 request charge is usually not the scary part. AWS lists S3 Standard `GET` requests at `$0.0004` per 1,000 requests in a pricing example, so one million preview requests is about `$0.40` in request charges. The expensive part is moving and processing bytes you did not need. AWS also shows S3 data transfer out to the internet at `$0.09/GB` in a pricing example. Actual bills vary by region, transfer tier, free tier, storage class, and whether the consumer runs inside the same AWS Region. The request count can look harmless while the byte shape is the real problem. Processing one full 1 GB object for a three-line preview is very different from a command that streams and returns once the preview is done, especially when an agent repeats that pattern thousands or millions of times. Even when same-region AWS data transfer is free, avoiding unnecessary full-object reads still reduces latency, parser work, memory pressure, and token exposure. ## What Is Different From a Normal S3 Mount A traditional S3 mount exposes object keys as files and mostly stops there. Mirage treats the command and the file type as part of the read path, which matters for formats that are not useful as raw bytes. For `.parquet`, a normal Unix-style read is not helpful to an agent. `cat` returns binary; `head` does not naturally mean "show me the first rows"; `grep` has no schema context. Mirage can make those commands format-aware: ```bash head /s3/events.parquet cat /s3/events.parquet ``` The object still lives in S3. The virtual filesystem gives Mirage room to return a typed view of it: schema, columns, and table previews instead of an opaque binary blob. That is the part a normal POSIX-style mount does not provide. ## Rolling Back With S3 Object Versions S3 already has a useful rollback primitive: when bucket versioning is enabled, every overwrite creates a new object version. Mirage can reuse that instead of asking the agent to invent its own checkpoint scheme. This matters because agents do multi-step work. They inspect source data, write derived artifacts, notice a bad transform, and then need to get back to the exact S3 object version they used earlier: ```bash head -n 20 /s3/events.jsonl grep payment_failed /s3/events.jsonl > /analysis/payment_failures.txt cp /analysis/payment_failures.txt /s3/reports/payment_failures.txt ``` If `/s3/events.jsonl` or `/s3/reports/payment_failures.txt` changes later, the agent should not have to guess which bytes were part of the earlier state. It needs a rollback handle. When Mirage reads an S3 object, it can record two backend-native markers: - `ETag`, used as a fingerprint to detect that the live object changed. - `VersionId`, used as a rollback pin to fetch the exact object version again. For rollback, Mirage does not need to copy every S3 object into a separate checkpoint bucket. If an earlier read captured a `VersionId`, Mirage can bind the virtual path back to that version and recover the same bytes from S3's own history, even if the current head of the key was overwritten later. Without S3 bucket versioning, Mirage can still use fingerprints to detect drift: it can tell the caller that the live object no longer matches what the agent read before. With bucket versioning enabled, Mirage can go further and recover the original bytes by reusing S3's own version history. That is the important versioning story for agents on S3: keep the agent's interface as `/s3/...`, but make rollback point at S3's native object versions whenever the bucket exposes them. ## Why This Pain Point Is Showing Up Now [Amazon S3 Files](https://aws.amazon.com/s3/features/files/) is a good signal that this problem is real: applications, agents, teams, and shell tools increasingly expect S3 data to behave like files and folders. The pain is not only API ergonomics. It is also cost shape. The [S3 pricing page](https://aws.amazon.com/s3/pricing/) for S3 Files splits out high-performance file-system storage plus data access charges for reads, writes, and sync. That is a reminder that file-like access to object storage is not free magic; where data is cached, copied, read, and synchronized matters. Agents make the cost problem sharper because they ask many small repeated questions. If a tiny preview turns into a full-object read, the agent pays in bytes moved, latency, memory, parser work, and sometimes tokens. Mirage's bet is that a filesystem-shaped command layer can avoid a lot of that waste: keep S3 as object storage, stream when composition needs it, and cache repeated work. ## Next This is the first version of the S3 story. The next useful step is a small benchmark: first-run `ls` with an empty cache, repeated `ls` after the listing is cached, first-run `stat`, repeated `stat`, streaming `head`, and repeated `grep`. --- # Introducing Mirage: A Unified Virtual Filesystem for AI Agents **URL:** https://www.strukto.ai/news/introducing-mirage **Type:** News **Published:** 2026-05-06 **Authors:** zecheng **Categories:** AI Agent, Virtual Filesystem, Agentic System, Open Source, Product Launch **External links:** Docs: https://docs.mirage.strukto.ai — GitHub: https://github.com/strukto-ai/mirage — X: https://x.com/zechengzh/status/2052105012172792061 — LinkedIn: https://www.linkedin.com/posts/zechengzhang_aiagents-llm-opensource-share-7457867220971331584-jxMk — Medium: https://medium.com/@strukto-ai/introducing-mirage-a-unified-virtual-filesystem-for-ai-agents-a6ba24c51231 — Substack: https://strukto.substack.com/p/introducing-mirage-a-unified-virtual ## What Mirage is Modern AI agents talk to dozens of backends: object stores, document services, databases, chat platforms, ticketing systems. Each one ships its own SDK, its own auth model, its own retry semantics. Wiring all of that into an agent today usually means one of two things: write a custom tool function per service, or run a fleet of MCP servers and hope they compose. **Mirage is a virtualization layer for AI agents, mounting data sources, services, and tools as one virtual filesystem with a bash interface.** Mount each backend at a path, then reach every backend with the same handful of Unix-like tools the LLM already knows. `cat`, `grep`, `head`, `wc`, pipes, and redirects all work across S3, Google Drive, GitHub, Slack, Postgres, and Redis, side-by-side under one root. ```bash grep alert /s3/log.jsonl | wc -l cp /drive/spec.md /github/repo/docs/ cat /slack/channels/oncall__C04F1/2026-05-06/chat.jsonl | head -20 ``` One bash command spans every backend. The agent reasons about one abstraction instead of N SDKs and M MCPs. ## Try it in one minute If you want the fastest path before reading the architecture details, install the SDK for your stack and mount a first workspace locally: ```bash npm install @struktoai/mirage-node uv add mirage-ai ``` For TypeScript apps, start with `@struktoai/mirage-node` on servers or `@struktoai/mirage-browser` in browser and edge runtimes. For Python, use `mirage-ai`. The [product page](/mirage) has fuller examples for mounts, credentials, snapshots, and framework adapters. ## How it fits together Mirage sits between the agent and the infrastructure. The agent issues bash commands or VFS calls; a dispatcher routes each call to the right backend; a two-layer cache (RAM index + file bytes) keeps repeated reads off the network. The whole workspace can be snapshotted to a `.tar`, cloned for parallel runs, and rolled back like git. ![Mirage architecture: agent talks to Mirage Bash and VFS, through a Dispatcher and Cache, to Infrastructure and Remote services.](/mirage-arch-light.svg) The architecture is intentionally boring. There's no new protocol to learn. There are no new vocabulary items for the LLM. Mirage maps every backend to filesystem semantics that have been stable since 1973. ## MCP vs a Real Filesystem The Model Context Protocol (MCP) is a useful standard for tool definitions. It does not, however, give the agent a _filesystem_. It gives the agent a list of tools, each one a function with a JSON schema. To compose two tools, the agent has to learn each function's vocabulary, manage state between calls, and handle errors per-tool. That works, but it scales by N: every new backend is a new tool surface. A unified filesystem flips the problem. Backends become directories. Operations become bash. The agent already knows how to compose `find | xargs grep | sort | head`. The LLM was pre-trained on that exact pattern. | | MCP server-per-backend | Mirage unified filesystem | | -------------------------------------------- | ----------------------------------- | ---------------------------------- | | Vocabulary the agent has to learn | One per server, growing with N | Bash. Already known. | | Composition across backends | Custom tool that orchestrates calls | Native bash pipes | | State between calls | Tool-specific, often re-fetched | Two-layer cache, hits across calls | | Snapshot / rollback / clone | Not in spec | One API call (`ws.tar`) | | Agent code to add a new backend | Author + register a new server | Mount one resource at a path | | Fits inside FastAPI / Express / browser apps | Server has to run separately | Embedded library, drops in | MCP is good for what it is: a transport for tool descriptions. But for agents that have to _read, write, and pipe across_ heterogeneous data, a real filesystem with a real shell is a better primitive. ## Mirage vs a Real Filesystem A real filesystem hands the agent raw bytes. `cat foo.parquet` on Linux dumps binary; the agent has to know the format, ship a parser, and probably waste tokens on a corrupted-looking blob. Permissions, mounts, and tool behavior are whatever the host OS happens to have configured. Mirage controls how the agent sees the files. The same `cat` command goes through Mirage's command registry, which knows the resource type at each mount point and can dispatch to a format-aware handler. `cat` on a `.parquet` returns a tabular preview, on a `.wav` returns a transcription, on a `.h5` returns a tensor summary. The agent reasons in bash; Mirage handles the decoding underneath. ```bash $ cat /s3/podcasts/episode42.wav | head -5 [00:00:01] welcome back to the show, today we're talking about [00:00:06] virtual filesystems for AI agents and why the [00:00:11] file abstraction keeps coming back. our guest is [00:00:16] going to walk us through what mirage actually does [00:00:21] under the hood and how it composes with bash. ``` That extra control surface unlocks a few things a stock filesystem cannot: - **Polymorphic bash on every format.** `cat`, `grep`, `head`, and `wc` work on `.parquet`, `.csv`, `.json`, `.mp3`, `.wav`, `.h5`, and more. The agent stops needing a format-specific tool per data type. - **Customization per workspace.** Register your own commands and resources. A research workspace can expose `cat` differently on `.h5` than a production workspace does, without changing the agent's prompt. - **Sandboxing and observability.** Every read and write goes through a single dispatcher, which means rate limits, audit logs, and policy checks live in one place instead of scattered across N tool implementations. - **Reproducibility.** The whole workspace, including its mount tree and command registry, snapshots into a `.tar`. The same agent run works the same on a teammate's laptop. Real filesystems are great primitives. Mirage is a real filesystem the agent's _operator_ gets to shape: same bash vocabulary, much more control over what the agent actually sees. ## What's next Mirage is open source today on [GitHub](https://github.com/strukto-ai/mirage). Python (`mirage-ai`) and TypeScript (`@struktoai/mirage-node`, `@struktoai/mirage-browser`) SDKs are live, plus a CLI. Adapters drop the workspace into OpenAI Agents SDK, Vercel AI SDK, LangChain, Pydantic AI, CAMEL, Mastra, and OpenHands. Read the full docs at [docs.mirage.strukto.ai](https://docs.mirage.strukto.ai), or jump straight to [the product page](/mirage). If you want to follow along, the launch threads on [X](https://x.com/zechengzh/status/2052105012172792061) and [LinkedIn](https://www.linkedin.com/posts/zechengzhang_aiagents-llm-opensource-share-7457867220971331584-jxMk) are good places to ask questions.