In today's newsletter:

• Technical LLM interview question!
• 10 must-use slash commands in Claude Code.
• [Hands-on] Build a Real-time Federated Data Engine for Agents.

TODAY'S ISSUE

Agents

​Technical LLM interview question!​

You have 80,000 agent trajectories from production. You need to find top 100 worth reviewing to improve your agent.

No LLM allowed to evaluate trajectories. How will you do this?

Let’s look at some approaches.

The simplest solution one could start with is random sampling. Pick 100 random trajectories and review.

But most production agents handle routine requests just fine, so you end up wasting a big chunk of your annotation budget.

Another approach can filter for longer conversations since 10+ user messages means more complexity.

But longer conversations skew heavily toward outright failures. You’ll surface obvious breakdowns but miss subtle issues hiding in conversations where the agent technically succeeded.

A recent paper from DigitalOcean takes a new approach.

It computes lightweight behavioral signals directly from the trajectory data using deterministic rules.

The signals fall into three groups:

1) Interaction signals:

• If a user rephrases the request or corrects the agent, that’s misalignment.
• Agent repeating itself is stagnation.
• User abandoning the agent is disengagement.
• User confirming something worked is satisfaction.

All are detected through normalized phrase matching and similarity checks.

2) Execution signals:

• A tool call that doesn’t advance the task is a failure signal.
• Repeated calls with identical or drifting inputs indicate a loop.

These are straightforward to extract from execution logs.

3) Environment signals, like rate limits, context overflow, and API errors.

• Useful to diagnose but not for training since they reflect system constraints, not agent decisions.

Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review.

On τ-bench, they compared all three approaches on 100 trajectories:

• Random sampling hit a 54% informativeness rate.
• The length-based heuristic reached 74%.
• Signal-based sampling reached 82%.

This means roughly 4 out of every 5 trajectories are genuinely useful to improve the agent.

In fact, among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs. 41.3% for random.

These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don’t break the task but still matter for optimization.

The whole framework runs without any LLM overhead and can sit always-on in a production pipeline.

If you want to see this in practice, this signal-based approach is already integrated into Plano, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place.

​Here’s the Plano GitHub repo →​

​Here’s the paper on arxiv →​

👉 Over to you: What is your approach to solve this?

Claude

​10 must-use slash commands in Claude Code​

Setting up shell aliases is such a natural part of working in a terminal that most developers do it almost reflexively. If you run a command often enough, you alias it.

With Claude Code prompts, though, devs typically skip this step entirely and keep retyping the same 10-15 line instructions from memory, like their code review checklist, test gen constraints, pre-commit scan...and all this session after session.

The real cost isn’t just the repetition you do as a dev, but the prompt drift.

Every time you retype a prompt from memory, the wording shifts slightly. For instance, you might forget a constraint or phrase the expected output format differently.

With shell commands, this doesn’t matter because they’re deterministic, but with an LLM, slightly different phrasing may produce noticeably different output.

Claude Code’s custom commands fix both problems.

You can save a markdown file in .claude/commands/, and it becomes a slash command you can invoke with identical instructions every time.

The prompts are version-controlled through Git, so your whole team runs the same commands, and when someone improves a prompt, everyone gets the update on their next pull.

This is the same pattern Boris Cherny described in his thread on Claude Code workflows, where his every repeated workflow becomes a command, checked into Git, and shared with the team:

Let’s walk through how to set them up, then the 10 commands that have been most useful in my workflow. We'll demo each one on a real ML inference service (FastAPI, scikit-learn, Alembic) so you can see the actual output, with full prompt templates you can drop into your own project.

How custom commands work

A custom command is a Markdown file inside a .claude/commands/ directory. The filename becomes the command name.

The file content is the prompt that gets sent to Claude when you run the command. You can use $ARGUMENTS as a placeholder for anything typed after the command name.

For instance, running “/dissect src/auth/session.ts” substitutes $ARGUMENTS with “src/auth/session.ts“.

You can also inject dynamic context using shell commands with the !command syntax:

Claude runs those shell commands before processing the prompt, so the context is always fresh.

Lastly, an optional YAML frontmatter at the top of the file lets you pre-approve tools (so Claude doesn’t ask for permission on every git call), set a model override, or add a description:

That’s the entire system, which includes a markdown file, an optional YAML header, and $ARGUMENTS for dynamic input.

Below are the 10 commands we’ve found most useful in practice:

The newsletter ahead is a bit too long to share over email due to size constraints.

​We have shared the full setup guide, with usage videos and prompts here →​

hands-on

​Build a Real-time Federated Data Engine for Agents​

Real-time sync for agents is incredibly difficult, especially when your data is scattered across dozens of sources.

Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything.

By the time your agent retrieves the data, it’s already outdated.

Imagine what it would look like if your Postgres database was updated 5 minutes ago, a MongoDB collection changed 2 minutes ago, but your agent is still pulling from yesterday’s snapshot.

This is why most production RAG systems fail.

There’s a better approach:

MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL, without moving any data.

Here’s what makes it different:

• Your data stays in place. No ETL pipelines or data duplication
• Query Postgres, MongoDB, REST APIs, and more using consistent SQL
• JOIN across different sources in real-time with a unified interface
• Works with both structured and unstructured data

And here’s the best part:

You don’t even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting.

The breakthrough for AI agents is simple:

When data updates at the source, your agent gets fresh results immediately without any sync delays, stale embeddings, or custom code for each integration.

You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get.

In the video below, we have given you a complete walkthrough of what we just discussed and how to actually do it.

Make sure you watch this till the end.

​You can find the MindsDB GitHub repo here →​

THAT'S A WRAP

Partner with US

ADVERTISE TO 900k+ AI Professionals

Our newsletter puts your products and services directly in front of an audience that matters, including thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.

Get in touch today by replying to this email.

Today’s email was brought to you by Avi Chawla and Akshay Pachaar.

​Update your profile | Unsubscribe​

Looking for more? Unlock our premium DS/ML resources.

© 2026 Daily Dose of Data Science