Research · Completed study
AI Reply Workflow Study
A controlled comparison of single-pass and staged AI replies to business inquiries.
Overview
I compared a one-call reply workflow with a seven-call workflow that checks the evidence before drafting. We examined factual support, completeness, next steps, usability, and resource use.
My role
I co-authored the study with Reeyan Thakrar. I built the staged workflow and review tools. Akshat Lakhiwal at the University of Georgia provided feedback.
What I worked on
- We tested 20 constructed inquiries across ten fictional business profiles, with three repetitions per workflow.
- We compared 120 responses across 480 logical calls, including the stages used to produce each reply.
Findings
- The staged workflow used 5.43 times as many input tokens.
- The staged replies contained fewer unsupported claims in this sample. The case-bootstrap interval for the difference included zero, and other quality measures involved tradeoffs.
Status
The comparison is complete. Curieux Review accepted the co-authored paper for a forthcoming issue.
The inquiry set was fictional and the comparison exploratory. The results do not establish performance on real client messages.
Links
Related writing
- When AI Briefings Confuse Free and Available Memory
TechChannel
- Before You Automate: A Trainer’s Test for Administrative AI
Personal Fitness Professional