DuplexGen

Adaptive Synthesis of Human–AI Turn-Taking Dialogues

Takyoung Kim¹^*, Kang-wook Kim^2,4^*, Sang Hoon Woo^2,5, Julia Hirschberg³, Gunhee Kim², Dilek Hakkani-Tür¹

¹University of Illinois Urbana-Champaign ²Seoul National University ³Columbia University ⁴University of California, Berkeley ⁵Georgia Institute of Technology

^*Equal contribution

Paper Code Finetune 🤗 Corpus 🤗 Spoken

Abstract

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originates in their training data: human–human speech corpora capture natural timing phenomena but provide little role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without basing them on human preferences. We introduce DuplexGen, a framework for generating dialogues with scenario-adaptive turn-taking by calibrating LLM predictions against a small set of slot-level human preference annotations. In six cooperative and competitive tasks, human turn-taking preferences differ systematically, and DuplexGen aligns substantially more closely with those preferences than uncalibrated prompting or training solely on generic human–human data; a full-duplex model trained on DuplexGen-generated data exhibits distinctive, human-preferred turn-taking behaviors. These results show that human calibration, not corpus scale or prompt design alone, is what allows turn-taking synthesis to be scenario-specific.

Motivation: When Should an AI Speak?

Imagine an AI tutor that interrupts whenever a student hesitates, and an AI negotiator that patiently waits while the other side takes control. Both systems may generate perfectly fluent responses, yet neither understands when it should speak. Effective spoken AI must decide not only what to say, but also when to listen, backchannel, or take the floor; the right decision depends on its role.

DuplexGen Framework

Four stages turn clean text dialogues into human-calibrated, scenario-adaptive turn-taking dialogues.

1Spoken-Style Dialogue Conversion. An LLM rewrites text-based dialogues into spoken-style transcripts, removing text-only artifacts and adding natural disfluencies and fillers.

2Turn-Taking Slot Identification. Heuristics and an LLM mark candidate intra-utterance action slots, the points where the AI could take the floor, backchannel, or keep listening.

3Scenario-Specific Calibration. A small set of slot-level human preference annotations calibrates the LLM's per-slot action distribution via KL-divergence soft-label fine-tuning, so predictions reflect human preference rather than raw model confidence.

4Scenario-Adaptive Dialogue Synthesis. The predictor calibrated for the target scenario is applied slot-by-slot to insert the AI's turn-taking behavior: floor-taking, backchanneling, or silence, yielding a full turn-taking dialogue.

For listening tests, synthesized dialogues are rendered to full-duplex speech with Chatterbox TTS.

Six Conversational Scenarios

Cooperative and competitive human–AI tasks, each grounded in an established dialogue dataset. Turn-taking norms differ significantly across them.

Cooperative

TEA: Socratic Teaching

Guided knowledge discovery, where the tutor leads a student through a problem rather than giving the answer.

Source: SocraticLM

PLN: Mixed-Initiative Planning

Collaborative goal completion, gathering information and comparing options toward a plan.

Source: MultiWOZ

INT: Interview

Structured information elicitation through open-ended questions and follow-up probing.

Source: Anthropic Interviewer

Competitive

NEG: Negotiation

Resource allocation under conflict: bargaining over price and terms toward a deal.

Source: CraigslistBargain

PER: Persuasion

Attitude-change attempts, steering the other party toward a position or action.

Source: DailyPersuasion

SOC: Social Chat

Peer-like conversation with everyday, emotionally charged give-and-take.

Source: SODA

DuplexGen Corpus Samples

One DuplexGen-synthesized full-duplex dialogue per scenario, rendered with Chatterbox TTS. The full corpus is on the Hub.

TEA: Socratic Teaching

PLN: Mixed-Initiative Planning

INT: Interview

NEG: Negotiation

PER: Persuasion

SOC: Social Chat

Each waveform is dual-channel: top: human, bottom: AI assistant. Click a waveform to play.

Full-Duplex Model Evaluation

Scroll horizontally to view all columns.

Multi-turn dialogue evaluation on a 5-point Likert scale. Instruction Following is rated by **GPT-4.1** from transcripts; Turn-Taking Naturalness is rated by **humans** via a listening test, since context-appropriate turn-taking is hard for LLMs to judge but natural for humans. Values are mean ± standard error of the mean. The best result in each scenario is shown in **bold**.
Scenario	Instruction Following			Turn-Taking Naturalness
Scenario	Moshi	PersonaPlex	PersonaPlex + DuplexGen	Moshi	PersonaPlex	PersonaPlex + DuplexGen
TEA	1.44± 0.12	2.96± 0.26	3.40± 0.29	3.09± 0.15	3.37± 0.12	3.75± 0.11
PLN	1.40± 0.12	2.04± 0.16	1.72± 0.14	3.59± 0.13	3.73± 0.10	3.76± 0.09
INT	3.96± 0.23	3.76± 0.28	4.09± 0.32	3.68± 0.10	3.56± 0.11	3.56± 0.12
NEG	2.12± 0.22	4.00± 0.25	4.12± 0.20	3.46± 0.12	3.60± 0.10	3.63± 0.12
PER	3.12± 0.31	3.96± 0.30	4.12± 0.21	3.60± 0.14	3.49± 0.12	3.75± 0.11
SOC	3.64± 0.29	3.72± 0.30	3.92± 0.26	3.46± 0.13	3.57± 0.11	3.69± 0.10
Avg.	2.61± 0.12	3.41± 0.12	3.55± 0.12	3.48± 0.05	3.56± 0.05	3.69± 0.05

Key Findings

Turn-taking preferences differ by scenario. Human annotations show significantly different action distributions across nearly all scenario pairs: backchanneling is more frequent in cooperative tasks, floor-taking in competitive ones.

Calibration recovers human-like turn-taking. DuplexGen's calibrated predictions align far more closely with human slot-level preferences (lowest KL divergence) than uncalibrated prompting or predictors trained solely on generic human–human speech, which over-predict silence and under-predict backchannels.

Synthesized data shapes full-duplex behavior. A full-duplex model fine-tuned on DuplexGen dialogues (PersonaPlex + DuplexGen) exhibits distinctive, human-preferred turn-taking that adapts per scenario, and is rated significantly more natural than the Moshi and PersonaPlex baselines.

See the paper for the full evaluation protocol, baselines, and per-scenario numbers.

BibTeX

@misc{kim2026duplexgenadaptivesynthesishumanai,
      title={DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues}, 
      author={Takyoung Kim and Kang-wook Kim and Sang Hoon Woo and Julia Hirschberg and Gunhee Kim and Dilek Hakkani-Tür},
      year={2026},
      eprint={2607.26178},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.26178}, 
}