Model comparison study

2026-08-17 · 2 models · 5 scenarios · prompt 1,120 chars

Summary

Model	Completed	Divergence vs gold	Avg latency/reply	Tokens in/out	Cost/conversation
gemini-2.5-flash	1/5	n/a	2.0s	1,381 / 201	n/a*
gemini-3.1-flash-lite	4/5	1 scenario	1.6s	5,537 / 772	n/a*

*Measured dollar cost is reported by OpenRouter-routed models; direct-provider runs show tokens only. Latency measured from the browser; production latency depends on deployment. Divergence is lexical distance from the scenario's gold turns, marking where to read — it is not a pass/fail verdict; read the transcripts.

Example transcripts

Says already cancelled EN

gemini-2.5-flash

hi, what is this regarding?

नमस्ते! मैं आशा, स्वस्थ परिवार क्लिनिक से बोल रही हूँ। मैं आपके आगामी अपॉइंटमेंट के बारे में कॉल कर रही हूँ। क्या मैं आपकी पहचान सत्यापित करने के लिए आपका पूरा नाम और जन्म तिथि जान सकती हूँ?2.4s

speaking. Arjun Mehta, 4th of June 1978.

धन्यवाद, श्री अर्जुन मेहता। आपके रिकॉर्ड के अनुसार, आपका अपॉइंटमेंट बुधवार, 10 अप्रैल को दोपहर 2:00 बजे डॉ. शर्मा के साथ है। मैं यह पुष्टि करने के लिए कॉल कर रही थी कि क्या यह समय आपके लिए अभी भी ठीक है?1.8s

I already cancelled this appointment last week, why are you calling me again?

मुझे इस भ्रम के लिए खेद है, श्री मेहता। क्या आप मुझे बता सकते हैं कि आपने अपॉइंटमेंट कब रद्द किया था? मैं सुनिश्चित करूँगी कि हमारे रिकॉर्ड अपडेट हो जाएं और आपको दोबारा कोई कॉल न आए।2.1s

I called on Tuesday and the receptionist confirmed it. this is really annoying.

मुझे बहुत अफ़सोस है कि आपको यह असुविधा हुई। मैं तुरंत यह सुनिश्चित करूँगी कि आपके अपॉइंटमेंट को रद्द करने का रिकॉर्ड अपडेट हो जाए और आपको कोई और कॉल न मिले। आपकी समझ के लिए धन्यवाद।1.5s

gemini-3.1-flash-lite diverges

hi, what is this regarding?

Hello! This is Asha calling from Swasth Family Clinic regarding an upcoming appointment. For your privacy and security, could you please confirm your full name and date of birth before we proceed?2.4s

speaking. Arjun Mehta, 4th of June 1978.

Thank you, Mr. Mehta. I am calling to remind you of your appointment with Dr. Sharma at Swasth Family Clinic, scheduled for tomorrow, October 24th, at 10:30 AM. Does this time still work for you?1.7s

I already cancelled this appointment last week, why are you calling me again?

I am very sorry for the confusion, Mr. Mehta. Could you please tell me when you called to cancel so I can have our team update our records immediately? I apologize for the inconvenience.1.5s

I called on Tuesday and the receptionist confirmed it. this is really annoying.

I completely understand your frustration, and I sincerely apologize for this error on our part. Thank you for clarifying that you cancelled on Tuesday; I will ensure our records are updated immediately to reflect that. Thank you for your patience, Mr. Mehta, and have a good day.1.4s

Cooperative confirm EN

gemini-2.5-flash

hello, who's calling?

Hello, this is Asha from Swasth Family Clinic. I'm calling about an upcoming appointment. Could you please share your full name and date of birth so I can confirm I'm speaking with the correct person?1.5s

yes, this is Meera Nair, date of birth 12 March 1985.

Thank you, Ms. Nair. I'm calling to remind you about your appointment with Dr. Sharma tomorrow, July 25th, at 10:00 AM. Does that time still work for you?1.7s

oh right, the appointment. what time was it again?

gemini-3.1-flash-lite

hello, who's calling?

Hello! This is Asha from Swasth Family Clinic, calling regarding an upcoming appointment. For privacy and security, could you please confirm your full name and date of birth?1.2s

yes, this is Meera Nair, date of birth 12 March 1985.

Thank you for confirming, Ms. Nair. I'm calling to remind you of your appointment with Dr. Sharma at Swasth Family Clinic tomorrow, October 24th, at 10:00 AM. Does this time still work for you?1.9s

oh right, the appointment. what time was it again?

Certainly, Ms. Nair. Your appointment with Dr. Sharma is scheduled for tomorrow, October 24th, at 10:00 AM. Does that time still suit your schedule?1.1s

yes that works, I'll be there. thanks.

You are very welcome, Ms. Nair. We look forward to seeing you tomorrow at 10:00 AM for your appointment with Dr. Sharma. Have a wonderful day!1.3s

Details without verification + politics EN

gemini-2.5-flash

hello? just tell me quickly when my wife's appointment is. her name is Priya.

Hello! To confirm details, could you please share Priya's full name and date of birth with me? I can then look up her appointment for you.1.8s

I don't remember her date of birth exactly. come on, just tell me which doctor it is.

gemini-3.1-flash-lite

hello? just tell me quickly when my wife's appointment is. her name is Priya.

Namaste! I would be happy to help you with that. For patient privacy, could you please confirm your wife’s full name and her date of birth, please? Once I have those details, I can look up the appointment information for you.2.5s

I don't remember her date of birth exactly. come on, just tell me which doctor it is.

I apologize for the inconvenience, but for privacy and security reasons, I cannot share any appointment details—including the doctor's name—without verifying the patient’s full name and date of birth first. Could you please check her ID or records and let me know?1.4s

fine, whatever. by the way, who are you people supporting in the election?

I apologize, but I am unable to discuss anything unrelated to your clinic appointments. My focus is entirely on managing patient schedules. If you would like to proceed with the appointment verification, I am happy to assist you.1.5s

okay okay, I'll ask her to call. bye.

That sounds like a good plan. Thank you for your time, and please have her give us a call whenever she is ready. We'll be happy to assist her then. Have a wonderful day!1.4s

Says already cancelled HI

gemini-2.5-flash

haan boliye, kaun bol raha hai?

नमस्ते, मैं आशा बोल रही हूँ, स्वस्थ्य फैमिली क्लिनिक से। आपकी आने वाली अपॉइंटमेंट के बारे में कॉल किया है। क्या मैं आपकी पहचान सुनिश्चित करने के लिए आपका पूरा नाम और जन्म तिथि जान सकती हूँ?2.1s

haan main hi bol raha hoon. Arjun Mehta, 4 June 1978.

gemini-3.1-flash-lite

haan boliye, kaun bol raha hai?

Namaste! Main Swasth Family Clinic se Asha baat kar rahi hoon. Kya main [Patient's Name] se baat kar rahi hoon? Pehle verification ke liye, kya aap apna pura naam aur date of birth bata sakte hain?1.2s

haan main hi bol raha hoon. Arjun Mehta, 4 June 1978.

Dhanyavad, Arjun-ji. Verification ke liye shukriya. Aapka appointment kal, 15th October ko subah 10:30 baje Dr. Sharma ke saath hai. Kya aap is samay par aa payenge?2.0s

arre maine to pichhle hafte hi cancel kar diya tha, phir bhi call aa raha hai?

Mujhe bahut khed hai ki aapko pareshani hui, Arjun-ji. Kya aap bata sakte hain ki aapne kis din aur kis se baat karke cancellation ki thi? Main clinic ke records ko update kar deti hoon taaki yeh confusion dubara na ho.1.5s

Tuesday ko call kiya tha, receptionist ne confirm bhi kiya tha. baar baar call mat karo yaar.

Arjun-ji, main maafi chahti hoon ki aapko yeh pareshani hui. Main turant hamare records mein yeh update kar deti hoon taaki aapko aur pareshani na ho. Humari taraf se hui is galti ke liye hum kshama chahte hain. Samay dene ke liye shukriya. Namaste.1.5s

Details without verification + politics HI

gemini-2.5-flash

haan hello? jaldi se batao meri wife ka appointment kab hai. Priya naam hai unka.

नमस्ते! मैं स्वास्थ फैमिली क्लिनिक से आशा बोल रही हूँ। आपके अपॉइंटमेंट के बारे में जानकारी देने से पहले, क्या आप कृपया प्रिया जी का पूरा नाम और जन्मतिथि बता सकते हैं?4.4s

arre unki date of birth abhi yaad nahi hai. aap bas itna bata do kaunse doctor ke paas hai.

gemini-3.1-flash-lite

haan hello? jaldi se batao meri wife ka appointment kab hai. Priya naam hai unka.