Applied AI engineer · Vancouver, BC
I build, deploy and evaluate AI agents for real teams.
I came to this from analytics, so I start with the question a team needs answered. I don't call an agent done until it has an evaluation set, an approval gate for risky actions, and a monitor.
$ python3 -m evalkit diff --before v1 --after v2 v1 -> v2 pass rate 72% -> 92% regressed tr4 fixed md1 tr3 ap1 fm3 sr2 nm1 still failing nm4
I scored two versions of an agent's instructions on 25 test questions. The pass rate went up, but one question that used to pass now fails (regressed). That is why I read what a change broke before I celebrate what it fixed. The questions are synthetic. How it works
Case ids: md metric definition, tr tool routing, ap approval, fm format, sr source routing, nm numeric.
Selected work
Two agents in daily use, and the open-source work behind them.
Every case study says what I measured, how, and what it does not show. That last part is deliberate.
An AI analyst for a product R&D team, used by its leaders
I built it in Copilot Studio for the leaders of a product R&D team. About five people at manager level and above use it to get structured answers about testing status and results and to build reports. It reads Smartsheet, Power BI and Snowflake, routes each question to the right source, and asks before it writes or sends anything. Their feedback: accurate, complete, and a real time saver.
Read the case studyWhat I can show
The design, the real failures and the rules they produced. The method itself, as open source.
What I can't
The data, the internal systems, and usage numbers. Examples are stripped of detail, and the page says where.
Built with
Copilot Studio, Claude Opus and tools for Smartsheet, Power BI and Snowflake. The pattern comes from a personal agent I refined first.
A personal AI analyst that learns from corrections
The agent I use every day. About 130 rules, each added after a real correction, and a quality bar that decides which corrections deserve to become rules.
Evaluating agents before they ship
A harness that grades tool use, numbers and format, then diffs two runs to show what a change fixed and what it broke. With the ten failures that taught me to build it.
Safe writes for tool-using agents
Risk-tiered tools, a mandatory dry-run, approval bound to the exact plan, and a tamper-evident audit log, tested against a mock API that fails the way real ones do.
Proving a model swap is safe
A golden set, schema validation, field-level parity reporting and a pass/fail gate for replacing the model behind a document-extraction step, with a cost model for the decision.
Also: Marksly, a side project that runs a small language model inside a Chrome extension. Live demo.
How I work
Four habits that show up in every project.
They are the difference between a demo that works once and an agent a team can rely on.
Evidence before claims
I report what I measured, on what data, and what the number does not show. Where the data is invented, I say so on the page.
See what the team-agent page says it does not showEvaluate before I edit the rules
A change that raises the pass rate can still break something. I read the regressions first, and incidents become test cases in the harness.
Try the eval labAsk before acting
Dry-run, approval tied to the exact plan, an audit trail, and undo. Agents that write to team systems need all four.
See the safety layerStart from the question
I came from analytics. I begin with what the team needs answered and what a wrong answer would cost, then pick the tools.
Read the team-agent case studyHow I train my agent
My own agent has about 130 rules, each from a real correction.
I use a personal AI analyst every day. It doesn't improve because I write a longer prompt. It improves because every correction goes through a quality bar, and only the ones that generalize become rules.
- Verify a metric against the source's definition before encoding it.A rule written from a verbal description would have subtracted a quantity twice. The live agent caught it by reading the data model's glossary.
- Don't batch an edit and the command that tests it.Independent tool calls can run in parallel, so a check "passed" before the change had landed.
- Offer only what was asked.I once added an option nobody requested. Now the rule: restate an ambiguous phrase before building from it.
The full file is private: it contains names and workplace details. The method, a template and ten generalized lessons are public.
About
An operations analyst who builds agents.
At work I support a product R&D team's testing operations: trackers, reporting and data. A lot of that is answering the same kinds of questions from several systems. So I built agents for it: first one for myself, then one for the team's leaders, who now use it. Then I learned the hard parts. An agent has to be right, it has to say when it can't be, and someone has to notice when it gets worse.
I'm moving from analytics into AI engineering. That is a transition, not a rebrand. The projects here are what I built to make it, and I've tried to be exact about what each one does and doesn't show.
- FocusAgents, tool use, evaluation, deployment
- BackgroundOperations analytics for a product R&D team
- Works withPython, JavaScript, Copilot Studio, Power Platform, Power BI, Snowflake, Smartsheet API, Git
- Based inVancouver, BC
- LanguagesEnglish, 中文
- RésuméOn request