Speech and voice
AI that transcribes calls and conversations, drafts notes and summaries, and can answer simple calls, with a person checking anything that matters.
What it is
Speech and voice AI turns spoken conversations into text, and text back into speech. The common uses are transcribing contact-centre calls, drafting notes from a consultation or meeting, summarising a call for the next adviser, and answering simple calls with a voice assistant that hands over to a person when it should.
A typical pipeline has three steps. Speech-to-text produces a transcript with timings and speakers, tuned with a custom vocabulary for product names, drug names or place names. A language model then summarises, drafts the note or pulls out actions. For live calls, text-to-speech, or one of the newer speech-to-speech models, speaks the reply. Each step can run on a recording after the call, or in real time during it.
In an enterprise, the hard parts are rarely the model. They are telling people they're being recorded and letting them say no, keeping recordings only as long as needed, integrating with the telephony and the record system, and making sure a person checks any note before it becomes part of a customer's or patient's record.
Use it when, and think twice when
Use it when
- People spend a lot of time writing up calls, meetings or consultations
- Calls follow common patterns, so summaries and routing can be checked
- You can tell people about recording and offer an alternative
- A person reviews notes before they're filed
Think twice when
- Callers are often distressed or vulnerable, and a voice bot would add friction
- Audio quality is poor, or your callers' accents and terms aren't in testing
- You'd use voice to identify people without treating it as biometric data
- Nobody will check the drafted notes before they're used
Reference architecture
01Channels
- Phone calls
- Consultations and meetings
- Voice notes
02Intake
- Amazon Connect, or your existing telephonyAzure Communication Services, or your existing telephonyYour existing telephony through a SIP linkTelephony and contact centre
- Amazon S3Blob StorageCeph, or IONOS S3 Object StorageRecording store
03Intelligence
- Amazon TranscribeAzure Speech in Foundry ToolsOpen-weight speech models (e.g. Whisper)Speech to text
- Amazon BedrockMicrosoft FoundryOpen-weight models served with vLLMSummarise and draft
- Amazon PollyAzure Speech in Foundry Tools (text to speech)An open-weight text-to-speech modelSpoken replies
04Orchestration
- AWS Step FunctionsLogic Apps or Durable FunctionsTemporalCall workflow
- Amazon EventBridge, SQSEvent Grid, Service BusApache Kafka or RabbitMQEvents and queues
05Systems of record
- CRM, case system or EPR
- Call recording archive
06People
- Person
Review and file the note - Person
Take over the call
- Person
- Cloud or platform service (AWSAzuresovereign / on-prem)
- AI agent
- A person decides
- Your existing systems
- Outside your organisation
Components, side by side
| Layer | AWS | Azure | Sovereign / on-prem |
|---|---|---|---|
| TelephonyReuse the contact-centre platform you have, if it can stream audio. | Amazon Connect, or your existing telephony | Azure Communication Services, or your existing telephony | Your existing telephony through a SIP link |
| Speech to textTest with your callers' accents, phone lines and terms. | Amazon Transcribe (en-GB), with a custom vocabulary | Azure Speech in Foundry Tools, with custom speech | Open-weight speech models (e.g. Whisper) on your own GPUs |
| RedactionPause recording for card payments, and mask identifiers before transcripts are stored. | Amazon Comprehend (PII detection) on the transcript | Azure Language in Foundry Tools (PII detection) | Open-source PII detection (e.g. Presidio) |
| Summaries and drafting | Amazon Bedrock | Microsoft Foundry | Open-weight models served with vLLM on your own GPUs, or IONOS AI Model Hub |
| Text to speech | Amazon Polly | Azure Speech in Foundry Tools | An open-weight text-to-speech model |
| Live voice conversationsNewer, and not offered in every region. Check UK availability before designing around it. | Amazon Nova Sonic (speech to speech), through Amazon Connect or Bedrock | Voice Live in Azure Speech in Foundry Tools | Open-weight speech and language models in a streaming pipeline |
| Workflow and events | AWS Step Functions, Amazon EventBridge, SQS | Logic Apps or Durable Functions, Event Grid, Service Bus | Temporal; Apache Kafka or RabbitMQ |
| Recording retentionDelete audio once the reviewed note is filed, unless you must keep it. | Amazon S3 with lifecycle rules | Blob Storage with lifecycle management | Ceph with lifecycle rules |
| Monitoring and audit | CloudWatch, CloudTrail | Azure Monitor, Log Analytics | Prometheus, Grafana, OpenTelemetry |
On a phone, the table shows the platform you pick. Sovereign / on-prem means open, portable components you can run in your own data centre or with a UK or European provider; IONOS is shown as one example, and others exist. What sovereign cloud means →
Operations
Keeping it reliable
AI fails quietly. These three habits make failure visible before customers see it.
Evaluation
Measure the errors that matter
Test transcription on real recordings from your own callers, with their accents, phone lines and background noise. An overall word error rate hides the errors that matter, so score names, numbers, drug names and amounts separately. Check summaries against what was actually said, looking for added facts.
Monitoring
Watch the edits
Track how much people change each drafted note before filing it, and which parts they change. Rising edits usually mean a new type of call or a gap in the vocabulary. For voice assistants, watch hand-over rates and abandoned calls.
Cost
Price it per minute
Speech services charge per audio minute, and summaries add model costs on top. Real time costs more than batch, so transcribe after the call unless a live use needs it. Report cost per hundred calls beside time saved per call.
Security and failure risks
Recording without proper notice
People aren't told they're being recorded or transcribed, or can't say no, and the recording has no clear lawful basis.
How the design avoids it
Clear notice at the start, a way to opt out, a DPIA, recording paused for card payments, and a defined retention period.
Voice spoofing
A cloned or recorded voice passes a voice-based identity check, or a caller impersonates a colleague.
How the design avoids it
Never rely on voice alone to identify anyone. Use step-up checks for anything sensitive, and treat voiceprints as biometric special category data.
Errors in the record
The transcript mishears a dose, an amount or a name, or the summary adds something nobody said, and it's filed as fact.
How the design avoids it
A person reviews and approves every note before it's saved, with the transcript beside it and key terms highlighted for checking.
Prompt injection by voice
A caller says something designed to make the voice assistant ignore its instructions or reveal data.
How the design avoids it
The transcript is treated as data, the assistant has narrow tools with platform-enforced permissions, and guardrails check what it says back.
Playbooks that use this pattern
All 12 playbooks using Speech and voice →Complaints handling
Every complaint caught, whatever channel it came through, answered fairly within the DISP deadlines, and traced back to its root cause. A handler writes and signs every final response.
- Retrieval (RAG)
- Agents with tools
- Speech and voice
- Human in the loop
Spotting vulnerable customers
Possible signs of vulnerability noticed the first time a customer mentions them, a trained person offered the right support, and outcomes checked so no group of customers is left worse off.
- Retrieval (RAG)
- Predictive ML
- Speech and voice
- Human in the loop
Ambient clinical notes
The consultation is captured, the note and the letter are drafted before the patient leaves, and the clinician checks, edits and signs every word.
- Speech and voice
- Agents with tools
- Human in the loop
- Event-driven integration
Patient enquiry assistant
Routine questions answered straight away, by phone or web chat, from content the trust has approved. Anything clinical or urgent goes to a person, with clear signposting to NHS 111 and 999.
- Retrieval (RAG)
- Speech and voice
- Agents with tools
- Human in the loop
Care notes and incident analysis
Better care notes in less time, often by voice, and early warning of patterns such as falls, weight loss, missed medicines and safeguarding signs, put in front of a manager who decides what to do.
- Speech and voice
- Predictive ML
- Human in the loop
- Retrieval (RAG)
Customer service agent
Order, delivery and returns questions answered in seconds, on chat, email and phone, from your real systems. Refunds within set limits handled end to end, and people dealing with every exception, complaint and vulnerable customer.
- Agents with tools
- Retrieval (RAG)
- Speech and voice
- Human in the loop
Field engineer assistant
Field engineers ask a question out loud and hear the answer from the approved procedure, manual or asset record, with its source, even with no signal. The permit, the safety rules and the engineer stay in charge of every safety decision.
- Retrieval (RAG)
- Speech and voice
- Human in the loop
- Event-driven integration
Helpline triage with safeguarding
Advisers spend their time with the caller, not the paperwork. Approved guidance is to hand, notes are drafted, and possible risk is flagged to the adviser and supervisor as it is said. People make every safeguarding decision.
- Speech and voice
- Retrieval (RAG)
- Human in the loop
- Event-driven integration
Sales: CRM assistant
Call notes, follow-ups and CRM updates drafted minutes after every conversation, for the salesperson to check and confirm. Contact rules are checked in code, and a person sends anything that isn't a fixed, approved template.
- Speech and voice
- Agents with tools
- Human in the loop
Patterns
All eight building blocks
Document AI
Read forms, letters and PDFs into structured data.
Retrieval (RAG)
Answer questions from your own documents, with sources.
Agents with tools
AI that takes small, controlled actions across your systems.
Human in the loop
Put people on the decisions that matter, with the evidence.
Predictive ML
Score, rank and forecast from your own historical data.
Event-driven integration
Connect AI to core systems safely, through events and queues.
Computer vision
Inspect photos, video and camera feeds for defects and damage.
Speech and voice
Turn calls and conversations into text, notes and actions.
Thinking about speech and voice in your organisation?
Talk through where it fits, what it would plug into, and which decisions should stay with people.
