Listen to this audio :
Designing Vocal Performance
Then something changed.
Companies like ElevenLabs started asking,
"Can AI perform this sentence?"
How ElevenLabs explores Voice Performance
Traditional text-to-speech systems were designed to pronounce words correctly.
ElevenLabs explores a different direction: expressive speech generation. Alongside realistic voice synthesis, its newer models allow developers to guide how a line should be performed using natural-language instructions and expressive tags.
While ElevenLabs hasn't publicly described every internal mechanism, the user experience suggests a shift from reading text to performing speech.
Design Trade Off
Speed/Low Latency-
Highly expressive speech requires more reasoning about context and performance. ElevenLabs explicitly positions v3 as a quality-first model, distinct from its faster sibling models — Turbo and Flash prioritize speed and low latency for real-time assistants or call-center bots.
Less predictability-
An expressive model has more freedom. ElevenLabs encourages developers to constrain emotional behavior through system prompts and brand voice guidelines. A more deterministic model produces more consistent but often flatter—speech.
What This Means for Designers
For decades, designers have focused on writing the right words.
Voice AI introduces a new design material: Performance.
The same sentence can reassure someone, build urgency, express empathy, or create excitement—without changing a single word. The difference lies in how it's delivered.
As expressive voice models become more capable, designers will increasingly make decisions such as:
Should this response sound calm or energetic?
Where should the assistant pause?
Which word deserves emphasis?
When should the voice acknowledge emotion, and when should it stay neutral?
How expressive is appropriate for this product and this moment?
These aren't technical decisions. They're interaction design decisions.
Humans can naturally perform. Actors spend years learning it.
For years, Voice could pronounce words correctly and read sentences clearly.
But it couldn't perform them.
Voice AI is only beginning to do so.
You just heard the same sentence performed in different ways.
Nothing about the words changed. The meaning didn't come from the sentence.
It came from the voice delivery.
Yet each version told a different story.
So how does AI know how to perform?
Traditionally all the text was written in markup language, across major providers like Google, Amazon, Microsoft.
Eleven Labs trained their model Eleven v3 on real end to end speech. You can now give the model performance directions directly inside the script.
Before generating any audio, the model reads these directions and plans how the line should be delivered.
It's less like programming. More like directing an actor.
How can we design for Vocal Performance?
This requires optimizing several things simultaneously:
Emotional Intent:
Every line begins with an emotional objective.
Instead of requiring the developer to explicitly say "Speak happily"
the model tries to infer emotion from:
the conversation
punctuation
wording
previous dialogue
system instructions
Additionally developers can still guide delivery with prompts or expressive tags like
[laughs],[whispers], or[sighs]when they want finer control.
For example:
"I'm sorry.""I completely understand.""Let's fix this together."Each line has a different emotional intention.
The model performs each one differently instead of applying one emotion to the entire interaction.
Vocal Expression:
Emotion becomes believable through vocal expression.
Humans naturally say more than words-We laugh, whisper, hesitate, breathe.
These small sounds make speech feel alive. They transform speech from narration into performance.
Trade-off
More non-verbal behavior can increase realism, but too much makes the AI feel theatrical or distracting.Speaking Rhythm:
Imagine two people saying
"I understand."
One replies immediately.
The other pauses first.
"I... understand."
The words are identical. The feeling isn't.
Emotional Progression:
Real conversations don't stay emotionally constant.
A support interaction may begin with concern, move toward reassurance, and end with encouragement.
Even within a single sentence, the emotional delivery may shift as the meaning unfolds.
Voice Casting
Performance depends on the voice itself.
A calm narrator, an energetic presenter, and a child-like voice will perform the same script differently.
Choosing the right voice determines how believable every emotional cue will feel.
Imagine this.
You tell your friend about your recent holiday adventure only to hear a cold reply.
The first day, you'd smile.
By the tenth day, you'd roll your eyes.
The problem isn't what they say, its the tone and context.
Now imagine that mismatch isn't occasional. Imagine it's every single interaction, because the "person" you're talking to has exactly one personality, permanently, regardless of your mood, the situation, or what you actually need in that moment.
That's what talking to voice assistants has been like for years.
Why Does It Feel Wrong?
Why does it feel so specifically wrong when a voice's tone doesn't match what you need in the moment even when the actual information it gives you is correct?
That's because conversation isn't only about information.
It's also about how that information is delivered.
Think about the range of tone you naturally expect from different people in your life:
Doctor
Confident, reassuring
Close Friend
personal, warm
Colleague
Polite, Supportive
Customer support
Calm, Apologetic
Doctor
Confident, reassuring
Close Friend
personal, warm
Colleague
Polite, Supportive
Customer support
Calm, Apologetic
Designing Vocal Performance
You just heard the same sentence performed in different ways.
Nothing about the words changed. The meaning didn't come from the sentence.
It came from the voice delivery.
Yet each version told a different story.
Humans can naturally perform. Actors spend years learning it.
For years, Voice could pronounce words correctly and read sentences clearly.
But it couldn't perform them.
Voice AI is only beginning to do so.
Then something changed.
Companies like ElevenLabs started asking,
"Can AI perform this sentence?"
How ElevenLabs explores Voice Performance
Traditional text-to-speech systems were designed to pronounce words correctly.
ElevenLabs explores a different direction: expressive speech generation. Alongside realistic voice synthesis, its newer models allow developers to guide how a line should be performed using natural-language instructions and expressive tags.
While ElevenLabs hasn't publicly described every internal mechanism, the user experience suggests a shift from reading text to performing speech.
Filler Ratio
Take any simple question:
"What's the weather today?"
There are several possible replies:
"24°C."
"It's 24°C.".
"Sure! It's 24°C today."
"It looks like a lovely day today. It's currently 24°C with clear skies."
Lexical density:
Lexical density is simply how much of a sentence is actual information versus conversational language.
High lexical density:
"Timer set. Five minutes."
Low lexical density:
"Sure! I've gone ahead and set a timer for five minutes."
The answer is the same. The experience isn't.
Latency per Persona
Every personality style is implicitly also a latency trade-off. A warmer response takes longer to generate and almost certainly costs real milliseconds in generation and synthesis. Brief mode isn't just a tone choice - it's plausibly the fastest-responding mode by design
What This Means for Designers
2.Pitch variation (Prosody)
This is one of the biggest reasons voices feel alive.
Try saying
"Really."
in three ways.
😐 Really.
😮 Really?!
🙄 Really...
Same word. Three emotions.
Voice personality isn't something you add at the end of the design process. It's about designing the relationship users have with that character.
Good voice design isn't about making AI sound more human.
It's about reducing the gap between what a user expects to hear and how the assistant actually responds.
Because in conversation, people rarely remember the exact words.
They remember how those words made them feel.
Listen to this audio :
So how does AI know how to perform?
Traditionally all the text was written in markup language, across major providers like Google, Amazon, Microsoft.
Eleven Labs trained their model Eleven v3 on real end to end speech. You can now give the model performance directions directly inside the script.
Before generating any audio, the model reads these directions and plans how the line should be delivered.
It's less like programming. More like directing an actor.
How can we design for Vocal Performance?
This requires optimizing several things simultaneously:
Emotional Intent:
Every line begins with an emotional objective.
Instead of requiring the developer to explicitly say "Speak happily" , the model tries to infer emotion from:
the conversation
punctuation
wording
previous dialogue
system instructions
Additionally developers can still guide delivery with prompts or expressive tags like
[laughs],[whispers], or[sighs]when they want finer control.
For example:
"I'm sorry.""I completely understand.""Let's fix this together."Each line has a different emotional intention.
The model performs each one differently instead of applying one emotion to the entire interaction.
Vocal Expression:
Emotion becomes believable through vocal expression.
Humans naturally say more than words-We laugh, whisper, hesitate, breathe.
These small sounds make speech feel alive. They transform speech from narration into performance.
Trade-off
More non-verbal behavior can increase realism, but too much makes the AI feel theatrical or distracting.
3. Speaking Rhythm:
Imagine two people saying
"I understand."
One replies immediately.
The other pauses first.
"I... understand."
The words are identical. The feeling isn't.
Emotional Progression:
Real conversations don't stay emotionally constant.
A support interaction may begin with concern, move toward reassurance, and end with encouragement.
Even within a single sentence, the emotional delivery may shift as the meaning unfolds.
Voice Casting:
Performance depends on the voice itself.
A calm narrator, an energetic presenter, and a child-like voice will perform the same script differently.
Choosing the right voice determines how believable every emotional cue will feel.
Design Trade Offs
Speed/Low Latency-
Highly expressive speech requires more reasoning about context and performance. ElevenLabs explicitly positions v3 as a quality-first model, distinct from its faster sibling models — Turbo and Flash prioritize speed and low latency for real-time assistants or call-center bots.
Less predictability-
An expressive model has more freedom. ElevenLabs encourages developers to constrain emotional behavior through system prompts and brand voice guidelines. A more deterministic model produces more consistent but often flatter—speech.
What This Means for Designers
For decades, designers have focused on writing the right words.
Voice AI introduces a new design material: performance.
The same sentence can reassure someone, build urgency, express empathy, or create excitement—without changing a single word. The difference lies in how it's delivered.
As expressive voice models become more capable, designers will increasingly make decisions such as:
Should this response sound calm or energetic?
Where should the assistant pause?
Which word deserves emphasis?
When should the voice acknowledge emotion, and when should it stay neutral?
How expressive is appropriate for this product and this moment?
These aren't technical decisions. They're interaction design decisions.
.