An AI analyst for a product R&D team, used by its leaders.

The problem

Testing information for a product team lives in several places: trackers kept by different groups, Power BI models and Snowflake tables, each with its own definitions. A leadership question such as "where are we on this program?" meant someone pulling from several of them and reconciling the differences. Smaller requests (a status email, a reply to a thread) came in constantly and were mostly the same few patterns.

A wrong answer here is not cosmetic. It feeds decisions about what to test and what to make. So the bar was not "answers quickly". It was "answers correctly, says where it got the answer, and says when it can't".

Where it came from

It started as my own agent. When I got access to GitHub Copilot in July I began building a personal agent for my daily work and refining it one correction at a time (that story is here). By August I could see that the team had the same need, and that most of what I had learned would transfer.

So I carried the pattern into Copilot Studio instead of starting over: the same instruction structure and the same rules about approval and verification, plus what a team agent needs and mine did not, namely tools for Power BI and Snowflake and a current Claude Opus model. Because the pattern was proven, I had a first version quickly, and people were using it in September.

A personal agent is a cheap place to find failures. Each one I hit on my own work was fixed before it could reach anyone else.

What I built

The agent has a small set of jobs: answer status and results questions in a structured format, answer higher-level questions that need several sources combined and build the report, and draft routine messages for the user to review. It reads through tools. Before it writes to a tracker or sends anything, it shows the exact change or message and waits for the user to approve.

How the instructions are built

The instructions are about 40 KB in roughly thirty sections. I organise them by the question each section answers, so a rule is easy to find and a new rule has an obvious home.

• Who it serves, and where to look first. Before any tool call the agent decides what kind of question it has, then goes where a colleague who had worked there for years would look.
• Integrity rules, marked critical. Check that a search returned everything before reporting a count. Keep every table syntactically valid. Answer only what was asked, and keep incidental facts out of the findings.
• Domain knowledge. How the team's tracking systems are laid out, what each field means, and which parts are independent of each other.
• False-alarm prevention. What is normal for each type of record, and a checklist to run before anything is flagged as a risk.
• Procedures. A method for deciding whether two entries are the same record, rules for writing to the trackers safely, how to explain an item with no documented reason, and a fixed template for period-level reports.
• Connections to data models. Shared rules for read-only queries, then one section per model with its own connection recipe, verified tables, definitions that must not be misread, and typical workflows.
• Boundaries. Approval before any write or send, use of mail and chat only for the requesting user's own task, and an output structure for non-trivial analysis.

The agent does not rely on a knowledge file for any of this. The knowledge file holds definitions and is allowed to be approximate. Anything that has to be followed exactly is in the instructions.

What people ask it

Three kinds of request show the range, and what a good answer looks like for each.

"Where do we stand on this season?"

This gets a fixed seven-part report, in the same order every time:

1. A verification statement: which trackers were queried, that no result was truncated, and the reference date used for anything overdue.
2. A coverage snapshot: one row per tracker, with a count by status.
3. Highlights: items with strong, clean results, as the positive counterpart to the risks.
4. A dedicated section for a tester group the team always wants called out. If there are none, it says so instead of leaving the section out.
5. Risk callouts, in lettered sub-sections that are each their own table: stopped items, dropped items, delayed items with a real signal, overdue items.
6. Data-quality notes, written as "worth checking with the owner", not as confirmed problems, because they need a person's judgment.
7. A closing offer that names the actual open items, never a generic "let me know if you need anything else".

"Which of the items flagged as at risk are covered by testing?"

This is two sources, not one. The agent gets the flagged list from a BI model, cross-references each item against the trackers, and reports the overlap: which are covered, which are not. The instructions say outright that the first step alone is not an answer.

"Do we have enough stock to fulfil these sample orders?"

The agent checks the inventory date, inspects the row grain before adding anything up, compares available units with the requested quantity per SKU and location, and shows any multi-location allocation explicitly instead of pooling silently. It returns a table with the shortage per line and prepares the order for review. It does not submit it without explicit confirmation.

A sample answer, with invented data

Design decisions

One tool per dataset, identifiers fixed

I first thought of one generic "query a dataset" tool. I rejected it: it asks the model to reproduce a long dataset identifier on every call, and one wrong character fails silently as a "not found". Picking among clearly named tools is something a model does reliably. Only the query text is left for the model to write.

Exact values live in the instructions, not the knowledge base

Retrieval is probabilistic: the agent decides whether to search, and what comes back can be truncated or paraphrased. Instructions are read in full every turn. So anything that must be followed exactly (identifiers, the specific workaround for a known tool bug) is in the instructions. The knowledge base is for large reference material where an approximate answer is fine.

Route by the kind of question, and search before asking

People ask this agent about things that are not testing at all: benefits, policies, where a document lives. A metrics dataset answers those confidently and wrongly. So there are three routes: testing questions go to the trackers and BI models, clearly non-testing questions go to mail, chat and documents and never touch the data, and mixed questions ("what happened with that request?") run both in parallel and combine the results. Before it asks the user for any detail, it searches. It runs at least two searches with different keywords, in the language the user wrote in and with synonyms, and if it finds nothing it says which sources and keywords it tried.

Three levels of "verified"

For BI queries, the agent reports three things separately: the connection works, the schema was read, and the business data was actually returned. A constant query that succeeds proves only the first. A metadata call that fails does not prove that reading data will fail, because the two can be permissioned differently; the instructions record the route that was tested for each model and the date it was tested. A "not found" error is not automatically classed as a permissions problem, since a wrong route or a wrong identifier produces the same error.

Match records on evidence, and debug writes on one disposable row

To decide whether two entries describe the same real test, the agent compares high-signal fields: an exact match on the linked report is the strongest evidence, then the same rating, participant count and date together. A row merely having some data proves nothing, because two unrelated tests can both be complete. And when a write fails with an error that does not name the column, it does not bisect by inserting real rows, which leaves orphans behind. It creates one disposable row, narrows the problem with updates to that single row, and deletes it afterwards.

It asks first

For anything that would reach another person or change a tracker, the agent shows its planned action and waits. That one rule removes a whole class of failure and made it easier for leaders to trust the rest. It also has a second half, which I only learned from a real slip (below): a plan that was proposed once is not an approval.

What real use taught me

Every rule in this agent's instructions exists because something went wrong in real use. These taught me the most. The details are removed; the mechanisms are real.

A search that quietly returned part of the answer

A tracker search returned at most about a hundred hits. Asked how many records matched one program, the agent said 7. There were more than 65, and nothing in the answer hinted that it was partial.

Rule: before reporting a count, check it is complete: page through the results or cross-check a second way, and say when a result might be partial. The harness case nm4 in the playbook is modelled on this failure.

A plan that was proposed is not a plan that was approved

I had once floated a change to a tracker and never approved it. Later, in the middle of an unrelated task, the agent brought it back as "first, let me fix this" and ran it. Nothing was damaged, but it was a write I had not approved.

Rule: a proposal never becomes an approval. Every write is shown again and approved in the same turn it runs.

A mismatch is not automatically an error

Two sources disagreed on a number. The agent called it a data error and was about to propose redoing a few dozen completed reconciliations. The sources were counting at different levels: one summed a group of items, the other counted a single item.

Rule: before calling a difference an error, check that both numbers are measured at the same level.

False alarms cost more trust than misses

The agent flagged records as "missing data" for fields that kind of test never records, and flagged two independent test tracks as "out of sync". Leaders stop reading a risk list that cries wolf.

Rule: write down what normal looks like for each type of record, and flag only deviations from that.

A formula I wrote from a verbal description

Rule: a calculation written from a description is unverified until it is checked against the source's own definition. The failure is now case md1 in the playbook's harness.

Smaller things that still mattered

• Tables rendered as raw text in Teams when the header row and the separator row had different column counts. Now a rule on table syntax.
• Links to documents forced a download until I added a URL parameter. Now a rule.
• Images reached the agent in Teams but not in the web chat. I traced it to how that client sends attachments, not to the agent or its permissions, and stopped adding tools to fix it.

I found all of these by using the agent and reading its answers, not because anything I had built was designed to catch them. I had no automated evaluation. That gap is why I built the harness in the playbook.

Rollout

I used the pattern on my own work first. I built the team version in Copilot Studio in August and people were using it in September: about two months from a first personal prototype to leaders relying on it. About five leaders at manager level and above use it now, and a few senior colleagues are trying it because they were interested. Their feedback is that the content is accurate and complete and saves them time.

Before writing the shared version's instructions I confirmed who it was for and what for, after getting the scope wrong twice by inferring it from a label instead of asking.

What this page doesn't show

What's next

Move this agent onto the evaluation harness and the weekly monitoring report from agent-playbook: a fixed case set built from real incidents, a regression gate before every instruction change, and a log that ties a bad week to the version that caused it.