Agents, Harnesses & Loops - a list of topics

If you didn’t already hear “agent” so far, congrats on escaping to a place where the entire LLM community couldn’t find you.

We started this year with agents, moved to harnesses, and are now talking about loop engineering. We still have half a year to go!

Keywords aside, it’s all software engineering to bypass the limitations of current LLMs and make them work for production problems. Each harness is a different solution to context size, integrations with existing systems, and a framework to make the entire non-deterministic technology deterministic and durable.

I am listing the levels of agentic engg as a guide of how I am thinking about it. I am not going into details of each one (You can ask LLMs to explain topics that are alien. Also, here’s one resource that builds an agentic harness from scratch for purely learning purposes.)

Level 1 - Agents & sessions:

  • Agent

    • Core LLM loop

    • Basic knobs: max turns, system prompt, retries, structured outputs, effort levels, thinking

    • Common tools like web search, web fetch

    • Hardening: model routing, OTEL, usage metering, streaming

  • Session

    • Conversation history, tool-call history, tool-call middleware

    • Save & resume: Continue the session with the same harness and convo state.

    • Hardening: Tool call secret proxies

Level 2 - Harness: Giving the agent access to knowledge and ability to perform actions

  • Tools

    • Programmatic tools: Custom defined. In-process runs.

    • Tools to work on a sandbox: grep, read, edit, shell execution, etc

    • MCP tools: Bring in external integrations

  • Permissions - tool call approvals, denials, and human approvals

  • Sandboxes: Local in-process (like just-bash), Cloud sandboxes, Sandboxes customized per task/ playbook

Level 3 - Customizing a harness for a use case:

  • Sub agents (their own agentic loops)

    • Isolated agents - its own convo history, harness, tools, etc.

    • Main agent sees the result of the subagent.

    • These are predefined sub-agents passed to the main agent for its use at its discretion.

  • Common sub-agents

    • Deep research

    • Explore, Plan (used by coding agents)

  • Context management

    • Truncating oversized tool output

    • Programmatic tool calling

  • Hardening: Tool-call middleware (like express middlewares), durability via tools like Temporal

  • Extending sandboxes: File systems in the sandbox, AGENTS.md, Skills (skill.md front matter), MEMORY.md

  • Teaching an agent when to use tools: Skills (sending expert knowledge into the agent), Memory as knowledge space for auto-improving agents.

Level 4 - Executing an agent in a customized harness towards a goal:

  • Triggers: run on x event with event trigger input

  • Scheduled/Recurring: run at x time/ run at x interval

  • Goals: auto nudge the agent until the goal is met (separate evaluator, smallish model, judging after every turn)

  • Plugins: Bring well crafted integrations for custom playbooks like Github for coding agents.

  • Channels: Bring agents into existing channels where users already are, like Slack

Level 5 - Multi-agent orchestration:

  • Agent Teams & mailboxes: lead agent supervising peer sessions, shared task list, ~5-10 long-running agentic teams.

  • Agentic Map Reduces: Coordinated subagents for specific use cases. Ex: Devin code review agents.

  • Agentic workflows: Code scripts with agentic loops and old-school code going hand in hand in deterministic orchestration. Extreme customization and Repeatable. From one agent to hundreds, running for hours in cohesion.

Level 6 - Harnessmaxxing:

  • Evals for agents and synthetic evals (A bit late in my leveling :P)

  • Running an agent that optimizes a harness config / prompts until the evals are maxxed

    • Auto research → come up with X1 → X2 built on top of X1 → X3 → X4. Stop after y turns. Pick the X that’s best.

    • Meta harness → look at all X’es so far and all performance issues so far to create next candidate

    • GEPA → Remove non-dominating X’es (a dominates over b if a is better than golden on all evals). Among the rest, pick based on probability to improve further.

Level 7 - Building well-crafted agents for popular use cases:

  • Coding agents: Mistral vibe, Codex, OpenCode, Claude Code

  • Agents for industries: cyber security

  • Agents for different modalities: visual reasoning pipelines, voice-specific agents for ultra-low latency

Level 8 - Loop engg/ Harness maker:

  • Creating a multi-agent harness using another large model and giving it the entire agent/harness infra at disposal.

  • Apply similar techniques as harness maxxing, just that we are now iterating an entire agentic pipeline instead of just prompts.

  • This is just too costly where we are today to build from scratch. A human building to some point and then creating a creation/optimizer loop is economical as of today.

Level 9 - Agentic finetuning:

  • Teaching an agent to use a harness better

  • Distilling knowledge of using a harness from LLM to SLMs

  • Context distillation from the same model to be able to codify the context/harness into its weights

What else am I missing?

built with btw btw logo