All projects

Research · Completed study

AI Reply Workflow Study

A controlled comparison of single-pass and staged AI replies to business inquiries.

Overview

I compared a one-call reply workflow with a seven-call workflow that checks the evidence before drafting. We examined factual support, completeness, next steps, usability, and resource use.

My role

I co-authored the study with Reeyan Thakrar. I built the staged workflow and review tools. Akshat Lakhiwal at the University of Georgia provided feedback.

What I worked on

  • We tested 20 constructed inquiries across ten fictional business profiles, with three repetitions per workflow.
  • We compared 120 responses across 480 logical calls, including the stages used to produce each reply.

Findings

  • The staged workflow used 5.43 times as many input tokens.
  • The staged replies contained fewer unsupported claims in this sample. The case-bootstrap interval for the difference included zero, and other quality measures involved tradeoffs.

Status

The comparison is complete. Curieux Review accepted the co-authored paper for a forthcoming issue.

The inquiry set was fictional and the comparison exploratory. The results do not establish performance on real client messages.

Related writing