Commentary / May 31, 2026
How to Measure Collaboration Friction in AI-Assisted Work
A practical how-to article on using transcript-based collaboration friction as one measurement tool for improving agentic work environments.
A modest way to see whether your human-AI work environment is getting better or worse.
We needed a way to measure the impact of changes to our agentic vibing environment, including our continuous improvement efforts. This time, the change was specifically about CLAUDE.md, the local guidance file that tells the agent how to work inside our shared workspace. But the need is bigger than one file. We will use this same measurement habit to understand any meaningful change in the robustness of our environment.
It is easy to say a new instruction “made the agent better.” It is harder to say what changed, where the evidence lives, and whether the improvement was caused by the change. The environment is always moving: models, tools, human habits, project context, outages, task difficulty, and the agent’s local understanding.
So the goal is to build a practical measurement tool that helps us notice whether the collaboration loop is getting smoother or rougher.
Agentic work is about more than the model. It includes the prompt, the repository, the tools, the approval boundaries, the human’s habits, the memory layer, and the structure of the work. If we change one part of that system, we need some way to look back and ask: did this help?
Start With The Collaboration Loop
Model metrics are useful, but they are not enough for day-to-day agentic work. In a real workspace, the agent is interpreting messy instructions, reading files, calling tools, waiting for feedback, handling local constraints, and sometimes recovering from mistakes.
The useful unit is the work loop: the human asks, the agent acts, the environment responds, the human corrects or approves, and the process continues.
That loop leaves a trail. In our case, the trail is the Codex session transcript. Other teams may have different logs, chat exports, task histories, or agent traces. The format matters less than the principle: if you want to understand the collaboration, look at its record.
The transcript cannot prove that the final work was good, that the human was satisfied, or that one change caused one result. But it can reveal visible friction: retries, corrections, blocked turns, repeated command failures, scope misunderstandings, abandoned paths, verification gaps, and moments when the human had to steer harder than expected.
That is enough to be useful, as long as we stay humble.
Define Friction Before Counting It
For our purposes, friction is not “anything that takes time.” Waiting for a long-running test can be normal. Asking a clarifying question can be healthy. Challenging an assumption can improve the work. A clean collaboration does not mean the transcript is silent or perfectly linear.
Friction is an interruption in the work loop: the difference between useful effort and avoidable drag.
That means the categories matter. A tool failure is different from a scope correction. A verification miss is different from an external provider outage. All can slow the work, but they point to different remedies. Without that second layer, the metric is just noise with a decimal point.
Count Affected Turns, Not Just Events
Raw counts can mislead. If one user turn produces five failed tool calls because a local service is unavailable, counting five separate events can make the collaboration look worse than it felt. The real signal may be that one turn was affected by one underlying issue.
That is why we look at affected turns. Affected-turn rate asks a simpler question: how often did the human’s work loop visibly hit friction?
The companion measure is frictionless turns. This is not a claim that the work was perfect. It simply asks how much of the collaboration moved forward without visible interruption. Over time, that can become a useful trend, but only when read beside the cause mix, the work diary, and the kind of work being done.
There are no hero metrics here. A frictionless turn is not automatically good, and a friction-affected turn is not automatically bad. The useful question is whether the visible friction was avoidable, useful, temporary, or outside the agent’s control.
The important word is investigating. The metric points. It does not explain itself.
Separate The Cause Classes
Once you can see affected turns, the next step is to separate likely causes. In our current system, we distinguish collaboration issues, verification issues, agent execution issues, environment and tooling issues, external provider issues, and user-driven changes. These are practical buckets, not perfect truths.
In version one, we assign each friction event to one primary cause class. That keeps the report readable and reduces double-counting. It is also imperfect. A single turn can include a scope correction, a failed tool call, and a verification concern. We are already considering better ways to represent multi-cause friction in future versions, but the first version favors interpretability over taxonomy perfection.
This distinction protects the team from bad conclusions.
If friction rises because the agent is misunderstanding scope, the answer may be better local guidance or clearer task framing. If friction rises because Docker is failing, the answer is not a better prompt. If friction rises because the human is learning a new workflow and correcting more aggressively, that may be a temporary investment rather than a failure.
This is especially important when measuring changes to an agentic environment. If we update CLAUDE.md and then see more friction the next day, that could mean the guidance got worse. It could also mean the guidance made the agent more explicit about assumptions and verification, creating more visible collaboration while improving trust.
Two derived measures help us keep the cause mix visible. Collaboration-adjusted friction means friction likely tied to the human-agent collaboration itself: scope, assumptions, verification, agent execution, and the way the work is being coordinated. External/tooling friction means friction likely caused by local tools, infrastructure, external providers, auth, quotas, network issues, or other conditions outside the collaboration pattern.
Those terms are not universal benchmarks. They are report labels that help us avoid blaming the wrong part of the system.
The number is the beginning of the conversation, not the verdict.
Include Human Signal
If the human is repeatedly correcting scope, showing frustration, expressing uncertainty, or applying urgency pressure, that matters. If the human is showing trust, positive signal, or productive rapport, that also matters. These are transcript-visible cues, not sentiment ground truth, but they help keep the measurement aligned with the real goal.
The goal is not to make friction disappear. The goal is trustworthy progress.
A low-friction environment where the human stops challenging the agent may be worse than a moderate-friction environment where assumptions are surfaced and corrected early. If you optimize only for fewer interruptions, you may teach the system to avoid the corrections that make the work safe.
We learned a lesson when collecting the human signal. In my environment, the OpenAI feedback loop is not exposed in the transcripts that we are scraping. Therefore, for this to work, the human has to provide the “how’s it going” signal in the conversation.
Read The Numbers With The Work Diary
A daily or weekly friction number without context is easy to misread. What kind of work was happening? Was it writing, coding, publishing, infrastructure, research, debugging, or workflow design? Was the team introducing a new tool? Were there external service problems?
The diary turns the number into a clue.
If friction spikes during a complex migration, that may be expected. If it stays high during familiar article edits, that may reveal a real collaboration problem. If external tooling dominates the cause mix, the team should not blame prompt quality.
This is where you need to dig underneath the numbers to understand what they mean. The top-line metric is not the insight. The interpretation is the insight.
Protect The Privacy Boundary
Transcript analysis is powerful because it sees the actual work loop. That also makes it sensitive.
Session logs can contain human prompts, agent responses, local paths, command output, error traces, source code, customer references, API URLs, environment variable names, and private emotional signals from the human side of the conversation. A useful internal report can include more detail. A shareable report needs a different policy.
In our process, we always produce two reports. The internal report is confidential and can include more diagnostic detail for the people doing the work. The clean report is designed from the start for broader sharing. It uses aggregate findings, date-level trends, cause-class summaries, sanitized work categories, caveats, and methodology notes to explain the approach without exposing private work.
That separation matters. A clean report should not be treated as a lightly redacted internal report. Redaction after the fact is easy to get wrong, especially because summaries can preserve sensitive meaning even when exact strings, paths, and names have been removed.
That privacy boundary is part of the how-to. If the measurement cannot be shared safely, it will either stay hidden or leak more than intended.
Use It As One Instrument, Not The Dashboard
This is just one measurement tool, and it is not perfect. It will miss things. It will misclassify things. It will sometimes make a day look better or worse than it felt. It can see transcript-visible friction, not the full human experience or the final quality of the work. But it still offers valuable insight.
It gives us a way to compare before and after when we change our environment. It helps separate agent behavior from tooling instability. It encourages us to ask whether a process change reduced avoidable drag or merely moved it somewhere else. It gives us a reason to inspect the work loop instead of relying on vibes alone.
No single number should become the dashboard. Affected-turn rate, frictionless-turn rate, collaboration-adjusted friction, external/tooling friction, and human signal are all instruments. They make sense together, in context, with the work diary beside them.
The practical habit is simple: change the environment deliberately, measure the collaboration loop, read the number with the diary, inspect the cause mix, and decide what to adjust next.
That is how I want to use this for CLAUDE.md. Not as proof that one guidance edit made the agent better by a precise amount, but as a way to see whether the working relationship changed in a visible, interpretable way.
Vibing without principles wastes time. Measuring without humility creates false confidence. A modest collaboration-friction audit sits in the middle: useful enough to guide improvement, humble enough not to become theater.
Appendix: How To Build Your Own Friction Audit
Everyone’s agentic environment is different. We use Codex session logs. You may use Claude Code logs, Cursor chat history, Continue traces, terminal transcripts, ticket comments, Slack threads, or some other record of human-agent work. Do not copy our tool mechanically. Copy the method.
The goal is to give yourself, or your agent, enough structure to build a first useful version.
How We Started
We began with a practical question: if we change the rules of the collaboration environment, how will we know whether the change helped?
For us, that question came up while revising CLAUDE.md, the workspace guidance file that tells the agent how to behave. We wanted the agent to surface assumptions, keep changes scoped, verify before declaring completion, preserve local agreements, and explain meaningful risk before action. Those rules sounded right, but sounding right is not the same as working better.
Our planning sessions converged on a few decisions:
- Measure the collaboration transcript, not the model in isolation.
- Treat the output as directional evidence, not scientific proof.
- Count affected turns, not only raw friction events.
- Separate collaboration problems from verification problems, agent execution misses, environment/tooling failures, external provider issues, and user-driven changes.
- Track visible human signal so “less friction” does not accidentally mean “less useful challenge.”
- Keep private internal reports separate from clean shareable reports.
- Make the first version plain Markdown, not a polished dashboard.
That last point mattered more than it may sound. A beautiful dashboard can make weak evidence look stronger than it is. We wanted the report to feel useful, but humble.
A Practical Build Plan
If you want to build your own version, start with the smallest useful loop.
First, choose the transcript source. Find the place where your agentic work already leaves a record. You need the human prompt, the agent response, tool calls or actions if available, timestamps if available, and enough session identity to group turns together.
Second, define the unit of analysis. We used the user turn because it maps well to the human work loop. A user asks for something, the agent acts, and the transcript shows whether that turn moved cleanly or hit visible friction.
Third, define your friction signals. Start simple. Look for correction language, retry loops, failed tool calls, scope reversals, verification challenges, abandoned paths, timeout patterns, and explicit user frustration. Do not try to solve sentiment analysis on day one.
Fourth, assign cause classes. For a first version, assign one primary cause class to each friction event so the report stays readable. A practical starter set is:
- collaboration
- verification
- agent execution
- environment/tooling
- external provider
- user-driven
Fifth, calculate both raw events and affected turns. Raw events tell you where to debug. Affected turns tell you how often the collaboration loop was interrupted.
Sixth, add a small work diary. Even a rough label such as “writing,” “coding,” “publishing,” “infrastructure,” or “workflow design” makes the numbers easier to interpret.
Seventh, generate two outputs. The internal report can include more detail for the team. The clean report should remove transcript snippets, exact local paths, customer names, secrets, session IDs, unreleased project details, and private emotional content.
Eighth, read the output like a diagnostic, not a grade. Ask what changed, what caused the change, and whether the friction was avoidable, useful, temporary, or outside the agent’s control.
Ninth, keep every daily report and read them together. The daily report tells you what happened in one window of work. The master trend analysis tells you whether the environment is changing over time.
A Starter Prompt For Your Agent
If you want an agent to build the first version for you, give it something like this:
Build a small local collaboration-friction audit for my agent transcripts.
Goal:
Measure visible friction in human-agent collaboration so I can compare workflow changes over time. This should be directional, not a productivity benchmark.
Input:
Use the transcript or session logs available in this workspace. First inspect the format and explain what fields are available.
Output:
Create a plain Markdown daily report, a machine-readable JSON daily report, and a master trend report that reads all daily reports.
Metrics:
- sessions scanned
- user turns
- agent turns
- tool calls or actions, if available
- raw friction events
- affected user turns
- frictionless user turns
- affected-turn rate per 100 user turns
- frictionless-turn rate per 100 user turns
- cause-class counts
- visible human signal counts
Cause classes:
- collaboration
- verification
- agent execution
- environment/tooling
- external provider
- user-driven
Requirements:
- Keep the implementation simple.
- Do not claim causality or productivity improvement.
- Include a caveat near the top of the report.
- Separate internal detail from clean shareable output.
- Recreate the master trend report from the daily reports so it is reproducible.
- Redact secrets, tokens, local sensitive paths, customer names, and private transcript snippets from shareable reports.
- Add focused tests or validation examples for the parser and metric calculations.That prompt will not produce the same tool in every environment, and it should not. The point is to force the agent to inspect the local record, define the measurement boundary, and build a first version around your actual workflow.
Sample Output Shape
Your first report does not need to be fancy. A useful report can look like this:
Collaboration Friction Audit
Period: 2026-05-30 to 2026-05-30 Source: agent session transcripts Sessions scanned: 2 User turns: 29 Agent turns: 123 Tool calls: 379 Events detected: 12 Affected user turns: 9 Frictionless user turns: 20 Human signals detected: 15 Observed transcript time: 2.4h
This is a directional heuristic. It measures visible collaboration friction in the transcript, not code quality, productivity, causality, or agent competence.
Scorecard
| Metric | Value |
|---|---|
| Raw event density | 41.4 / 100 user turns |
| Affected-turn rate | 31.0 / 100 user turns |
| Frictionless-turn rate | 69.0 / 100 user turns |
| Collaboration-adjusted friction | 6.9 / 100 user turns |
| External/tooling friction | 34.5 / 100 user turns |
Cause Mix
| Cause Class | Count | Interpretation |
|---|---|---|
| Collaboration | 1 | Scope, retry, assumption, rollback, or correction signals |
| Verification | 0 | Test, proof, or validation challenge |
| Agent execution | 9 | Avoidable command, path, or local execution miss |
| Environment/tooling | 1 | Shell, Docker, sandbox, local service, cache, or filesystem issue |
| External provider | 0 | API, auth, quota, rate limit, network, or provider issue |
| User-driven | 1 | Intentional abort, scope change, or user-directed interruption |
Work Diary
| Date | Work Types | Turn Signal |
|---|---|---|
| 2026-05-30 | Agent ops / metrics | 31.0% affected, 69.0% frictionless |
The numbers above are only an output shape. Your real values will depend on your transcript format, how you define a turn, how you assign primary cause classes, and how conservative your friction detection is.
Master Trend Analysis
A daily report is useful, but it is not the end of the measurement loop. The daily report answers: what happened in this slice of work? The master trend analysis answers: what pattern is emerging across many slices of work?
This matters because any single day can be misleading. One day may include a hard migration. Another may be a writing day. Another may include an external outage. Another may be the first day after changing the agent guidance. If you read only one daily report, you may confuse local weather for climate.
The master trend report should read all of the daily reports and rebuild the larger picture from them. I prefer rebuilding the master report each time instead of hand-editing it, because the trend should be reproducible. The daily reports are the evidence. The master report is the synthesis.
At minimum, the master trend analysis should include:
- the date range covered
- number of daily reports included
- total user turns, agent turns, and tool calls
- daily affected-turn rate
- daily frictionless-turn rate
- raw event density by day
- cause-class totals and cause-class movement over time
- human signal totals and notable movement
- work diary summary by date
- days with unusual spikes or drops
- notes about known environment changes
The most useful part is not the aggregate average. The useful part is the shape of the change.
For example, if affected-turn rate drops for three weeks after a guidance change, that is worth studying. If raw event density rises but collaboration-adjusted friction falls, the environment may be noisier while the human-agent relationship is improving. If environment/tooling friction dominates every spike, the team should invest in the workspace, not the prompt. If human correction pressure rises while frictionless turns also rise, the human may be steering more sharply but getting better throughput.
The master report should also call out before-and-after windows. If you changed CLAUDE.md on May 30, compare the days before May 30 with the days after May 30. Do not claim proof. Say what changed, what else may have changed, and what follow-up inspection is needed.
A readable master trend report might look like this:
Master Collaboration Trend
Period covered: 2026-05-24 to 2026-05-30
Daily reports included: 7
Total user turns: 184
Total agent turns: 731
Total tool calls: 2,108
This is a directional trend summary. It aggregates visible transcript friction from daily reports. It does not prove productivity, causality, code quality, or agent competence.
Trend Summary
| Metric | 7-Day Value | Direction |
|---|---|---|
| Average affected-turn rate | 28.4 / 100 user turns | down |
| Average frictionless-turn rate | 71.6 / 100 user turns | up |
| Average raw event density | 39.2 / 100 user turns | mixed |
| Collaboration-adjusted friction | 8.1 / 100 user turns | down |
| External/tooling friction | 31.1 / 100 user turns | flat |
Daily Turn Signal
| Date | Affected-Turn Rate | Frictionless-Turn Rate | Main Cause |
|---|---|---|---|
| 2026-05-24 | 35.0 | 65.0 | environment/tooling |
| 2026-05-25 | 31.8 | 68.2 | agent execution |
| 2026-05-26 | 29.4 | 70.6 | collaboration |
| 2026-05-27 | 26.1 | 73.9 | verification |
| 2026-05-28 | 24.7 | 75.3 | agent execution |
| 2026-05-29 | 20.0 | 80.0 | user-driven |
| 2026-05-30 | 31.0 | 69.0 | environment/tooling |
Interpretation
The seven-day pattern suggests collaboration friction may be improving, but the final day shows a tooling-related spike. That spike should not be read as a failure of the collaboration rules without checking the work diary. The better question is whether the same environment/tooling issue repeats across future reports.
Again, the values above are not universal benchmarks. They are an example of the kind of trend synthesis that helps the team know where to look next.
What To Improve After Version One
Once the first report works, resist the urge to jump straight to charts. Better definitions are more valuable than better visuals.
The next useful improvements are usually:
- grouping repeated tool failures into one underlying incident
- distinguishing normal waiting from real blockage
- improving work-type classification
- adding weekly and monthly rollups
- letting a human annotate days like “new workflow,” “API outage,” or “hard migration”
- comparing friction before and after specific environment changes
- building a clean report mode from the beginning instead of redacting after the fact
The version-one tool should answer one question: where should we look more closely?
Once you can answer that, you have moved from vibe-based improvement to instrumented learning.