/5 min read
From 6 Hours to 2: Fixing the Memory Wall in Batch AI Pipelines
Why throwing more GPUs at batch inference doesn't work — and the three infrastructure optimizations that actually cut our pipeline runtime by 66%.
Hi, I'm Sowmith.👋
I build theinfrastructurethat takes AI fromdemo to production.
At Fractal.ai I'm building the harness around production agents — tool registry, evals, the plumbing around the prompt.
Before that, ML infra at Amazon — pipelines, serving, and deployment for inspection models running on millions of packages a day.
The stack is the moat.
Ship the harness, not the prompt.
If it's not measured, it's not real.
Guardrails are a feature.
From demo to production
A demo is one prompt that works. Production is the session, harness, sandbox, eval, and guardrails that make it work again tomorrow when the model changed.
Model drift — an LLM's behavior shifts as providers ship new versions. Prompts, tool calls, and evals built on one model may not survive the next.
Chaos
gpt-5.3-codexClaude Opus 4.7Llama 4Responses APIAssistants API (deprecated)MCP server (half-built)pgvectorprompt_v12.mdscratch.ipynb.env.localagent.pyTODO: eval harnessTODO: retry logicuseless_log.txtLangChain
Orchestration
Session
Append-only event log + memory
Harness
The loop: plan, call, retry, exit
Tool Registry(MCP)
Typed, discoverable, permissioned tools
Evals & Guardrails
CI for model swaps
Production
- User
- Router
- Harness
- Response
Agentic infrastructure is the layer between the model and the user — session, harness, tool registry, evals. It's where most of the engineering cost of an AI product now lives.
Writing
All posts/5 min read
Your Text-to-SQL Is Lying to You
Academic benchmarks show 90%+ accuracy. Production systems hit 10-31%. The dangerous gap is semantic errors that look correct but aren't.
/6 min read
GraphRAG Looks Great Until Entity Resolution Breaks
Vector RAG fails at multi-hop reasoning. GraphRAG fixes it — until entity resolution breaks, and errors compound exponentially across every query.
/5 min read
We Moved from React to Svelte 5 and Cut Our Bundle by 60%
We migrated a B2B UI library from React to Svelte 5. The bundle dropped 60%, FCP improved 400ms, and the real trade-offs were not what we expected.
/4 min read
Modern React Patterns for Scalable Applications
Explore advanced React patterns including compound components, render props, and custom hooks that help build maintainable and scalable applications.