Hinglish Voice AI: The Hardest Test and How to Judge It

Listen to a sales call in Delhi, Indore or Patna and you will hear sentences like “Sir, आपका loan approve हो गया है, बस एक document pending है।” Many callers don’t speak only Hindi or only English. They speak both, and they switch inside the same sentence.

For a voice AI, that mix is the hardest test there is. A Hinglish call exposes every weak point in the system: how it hears, what it writes before speaking, how it pronounces, and whether it sounds like someone from here. This post explains where things break and ends with a listening test you can run on any vendor’s demo, including ours.

Code-switching is not two languages taking turns

In Hinglish, English nouns sit inside Hindi grammar (“form fill कर दिया”), Hindi verbs attach to English words (“cancel करवाना है”), and an English phrase can sit in the middle of a Hindi sentence (“basically मैं यह पूछ रहा था”). A system that detects one language per sentence, or per call, gets this wrong all the time.

A voice agent has three parts, and Hinglish tests each one differently:

• Listening (speech recognition). It has to catch English words spoken with an Indian accent in the middle of Hindi. It also has to write them down in some script, and there is no single correct spelling. The authors of the Voice of India benchmark point out that strict word error rate penalises “non standardized spellings of code-mixed English origin words”, so they allowed several valid transcripts per clip. When you judge a vendor’s transcript, judge the meaning, not the spelling.
• Understanding and replying (the language model). It has to reply in the same mix the caller used, not switch to pure English or to formal Hindi.
• Speaking (text-to-speech). It has to pronounce the Hindi as Hindi and the English as Indian English, in one voice, without an odd jump in accent between the two.

Devanagari, romanised or mixed: why the script changes the voice

Most people never see the text a voice agent speaks from, but it shapes how the agent sounds. The same sentence can be written three ways:

Sarvam, which makes the Bulbul TTS models, is direct about this in its documentation. It says to “always use native script for Indic words”. Its list of known limitations says that “Romanised/transliterated Indic input significantly degrades output quality: this is the most common integration mistake”. Its advice for mixed text is to “write English words in English script, Hindi words in Devanagari” and to keep brand names in English. It also recommends ending Hindi sentences with । and English ones with a full stop.

That is one vendor’s guidance for its own engine, but the reasoning applies more widely. Romanised Hindi has no standard spelling (“hai”, “he”, “h”), so the engine has to guess. Devanagari spells out the sounds. English words written in Latin script are read as the English words they are.

Gender, honorifics and “aap”

Hindi verbs mark the speaker’s gender. A female voice should say “मैं check कर रही हूँ” and a male voice “मैं check कर रहा हूँ”. The language model writes the words and the voice engine speaks them, and they are separate systems. Unless the model is told which voice it is writing for, you get a woman’s voice saying “कर रहा हूँ”. Callers notice at once.

The caller’s gender matters too. Hindi forms addressed to another person often change with their gender (“आप आए थे” or “आप आई थीं”). An agent that doesn’t know who it is speaking to should not guess. Adding “ji” to the name, or rephrasing to avoid the gendered form, is safer.

On a business call, respect is not optional. The agent should always use “आप”, never “तुम”, with the matching plural verb forms (“आप क्या करते हैं?”). An agent that slips into “tum” even once sounds rude, whatever else it gets right.

Everyday phone Hindi, not textbook Hindi

Machine-translated Hindi tends towards formal, Sanskrit-heavy words that no one uses on the phone. On a sales or support call this sounds like a railway announcement, and callers stop listening.

The right mix depends on who you call. A caller in a small town and one in a metro office may use very different amounts of English. A good agent follows the caller’s lead: if the caller answers in pure Hindi, it uses fewer English words.

Haan, achha, hmm: backchannels and barge-in

Indian callers say “haan”, “achha”, “hmm” and “ji” all the time while the other person is talking. It means “I’m listening”, not “stop”. This is where many voice agents fail on real calls:

• An agent that stops every time it hears “haan” keeps breaking off mid-sentence and sounds broken.
• An agent that never stops can’t be interrupted when the caller really wants to say something.
• An agent that ignores the “haan” completely may miss that it was the answer to the question it just asked.

The right behaviour is a judgement call. Talk through a short acknowledgement, and count it as a yes if it answers a question. Stop and listen when the caller starts a real sentence (“नहीं नहीं, मैंने already…”).

Short answers are also where speech recognition is weakest. The Voice of India benchmark measured error rates by clip length. Averaged across languages, Amazon’s speech-to-text went from a 10.45% word error rate on clips over five seconds to 18.74% on clips under two seconds. Microsoft’s went from 10.90% to 18.59%. A soft one-word “haan” or “नहीं” on a phone line is exactly the clip most likely to be misheard, so test it on purpose.

Numbers, dates and phone numbers

Hinglish speakers mix number systems freely: “साढ़े तीन हज़ार”, “thirty-five hundred”, “3,500”, “ढाई lakh”, “twenty-five thousand”. Lakh and crore are normal. Things to check:

• Amounts should be spoken as amounts, in the caller’s language. ₹25,000 read as “two five zero zero zero” is a failure. Sarvam’s docs suggest commas in numbers over four digits and, for mixed text, setting the language code to “the language in which you want entities (e.g numbers) in speech”.
• Phone numbers should be read digit by digit, in groups, and read back once to confirm. A ten-digit number spoken as “nine hundred eighty-seven crore…” is useless.
• Dates and times such as “कल शाम”, “परसों” and “next Monday” have to become a specific date. “कल” means both yesterday and tomorrow in Hindi, and only the context tells you which.

Brand names, product names and people’s names

Your brand, your product names and your customers’ names come up on almost every call. Some acronyms are said as letters (“EMI”) and some as words (“NEET”). Names from Tamil Nadu, Punjab and Bengal follow different sound rules. A TTS engine will get some of these wrong. Sarvam’s docs recommend “a pronunciation dictionary to correct mispronounced names, brands, or domain-specific terms instead of relying on phonetic spelling workarounds”. Ask whether your vendor has one and who maintains it.

How to read vendor accuracy claims

Speech recognition accuracy is usually reported as word error rate (WER): the share of words the system got wrong. Lower is better. The number depends heavily on the test audio. The Voice of India benchmark, from researchers at IIT Madras and Josh Talks, uses 536 hours of unscripted two-person telephonic conversations from 36,691 speakers in 15 languages. On its Hindi set, WER ranged from 5 (Sarvam Audio) to 33.9 (GPT-4o Transcribe) across the 14 systems tested. Same audio, a nearly seven-fold difference.

The paper found two more things that matter to buyers:

• District-level results vary widely, from about 4% to 44% WER. Districts in the Hindi belt clustered below 10%, while linguistically diverse regions scored much worse. Ask where a vendor’s test audio came from.
• Poorer audio quality consistently pushed error rates up. Phone lines, speakerphones and noisy streets are your real conditions.

The authors also conclude that public benchmarks “can overestimate real world performance”. A good score on a clean public test set tells you little about your callers. The only test that counts is your own audio.

A 10-point listening test for any vendor’s demo

Call the demo line yourself, from an ordinary phone, not a laptop. Score each point 0 (fails), 1 (mostly) or 2 (natural). Run it twice: once as a friendly caller and once as a busy, distracted one.

1. 1

   Mixed sentence

   Say “मेरा form submit हो गया था, पर payment fail हो गया।” Does it understand both halves and reply in the same mix?

2. 2

   Script check

   Ask for the text log of what the agent said. Is the Hindi in Devanagari and the English in English? Romanised Hindi is a warning sign.

3. 3

   Gender agreement

   Listen for “कर रही हूँ” or “कर रहा हूँ” matching the voice. Switch to a voice of the other gender and test again.

4. 4

   Respect

   Does it always use “आप”? How does it address you when it doesn’t know your gender?

5. 5

   Register

   Does it sound like a person on the phone, or does it use textbook words like “कृपया”, “उपलब्ध” and “स्वीकृत”?

6. 6

   Backchannels and barge-in

   Say “haan… haan” while it talks. It should carry on. Then interrupt with a full sentence. It should stop and listen.

7. 7

   One-word answers

   Answer a yes/no question with only a soft “haan” or “nahi”. Did it catch it, and did it act on it?

8. 8

   Numbers and dates

   Give a phone number in mixed Hindi and English, an amount (“ढाई lakh”) and a time (“कल शाम सात बजे”). Does it read the number back digit by digit and get the date right?

9. 9

   Names and brands

   Ask about your own brand and products, and give three names from different states. Note every mispronunciation.

10. 10

    A bad line

    Call on speakerphone from a noisy street. When it can’t hear you, does it recover politely or apologise in a loop?

A score of 16 or more out of 20 on both runs means the agent is ready for a pilot with real leads. Anything below that needs fixing before your customers hear it. Whatever the score, listen to the recordings afterwards. Things you missed live are obvious on replay.