It starts with a phone call. The voice on the other end sounds warm, natural, perhaps even familiar. It asks how your day is going, listens politely, and responds with just the right cadence and tone. Only later do you discover you were never speaking to a person at all?

This unsettling reality is no longer confined to science fiction. Advances in AI voice cloning have made it possible to generate synthetic voices that are nearly indistinguishable from real human speech. In many cases, even subtle hesitations, background noise, and emotional intonations are reproduced with unnerving accuracy.

While companies are eager to deploy this technology in customer service, entertainment, and accessibility tools, critics warn that a line is being crossed—quietly and without sufficient scrutiny. The core question is not whether the technology works. It clearly does. The real question is whether it should be used the way it is.

When AI starts lying

A recent investigation by Wired shed light on the extent of deception possible with current-generation AI bots. In one test, a voice assistant convinced a caller it was a woman offering medical advice. The catch? The bot was not only lying about its identity—it was also collecting sensitive images under false pretences.

In another case, a company used a deepfake video of its own CEO for promotional purposes while insisting publicly that no AI was involved. This blending of reality and fabrication has prompted privacy experts like Jen Caltrider of the Mozilla Foundation to speak out. “It is not ethical for an AI chatbot to lie and say it is human,” she warns. “People are more likely to open up when they think they are talking to a real person, which makes them vulnerable.”

This phenomenon, now dubbed “human-washing” by AI researcher Emily Dardaman, is fast becoming a pressing concern. As synthetic voices become more convincing, distinguishing between real and artificial interactions is no longer easy—or even possible for the average person.

Identity theft in the age of voice

The potential for misuse is vast. Voice cloning technology has already been used in fraud schemes, with criminals mimicking loved ones to ask for money or posing as corporate executives to authorise transfers. Unlike visual deepfakes, which may still contain visual artefacts or inconsistencies, audio deepfakes are harder to detect, particularly when heard over the phone or embedded in video content.

There is also the question of consent. Who owns a voice? Can a person’s vocal signature be used to train an AI model without their knowledge? In many jurisdictions, the law has not yet caught up. Voice data can be scraped, stored, and replicated—often without any meaningful safeguards in place.

Companies behind the tech argue that safeguards exist. Some insert subtle watermarks or require disclosure that the voice is AI-generated. However, enforcement is weak, and many of these safeguards can be bypassed or ignored altogether.

Beyond deception: bias, control, and accountability

Beyond impersonation, there are deeper systemic issues. AI systems trained on voice data can inherit biases from their datasets. This means accents, speech disorders, or non-standard pronunciations may be misinterpreted—or excluded altogether. Researchers have raised concerns about AI-generated voiceovers perpetuating stereotypes or reinforcing linguistic hierarchies, favouring certain ways of speaking over others.

Moreover, when a voice-based AI system causes harm, it is not always clear who is responsible. Was it the developer who trained the model? The company that deployed it? Or the end user who gave it instructions? This diffusion of accountability is one of the thorniest challenges in AI ethics.

The commercial appeal of AI voices—lower costs, infinite availability, perfect control—makes them attractive to industries from film and gaming to telemarketing. But as technology advances, the risks do not fade. If anything, they become harder to detect and easier to exploit.

Where do we go from here?

Some experts are calling for regulation—not to ban the technology outright, but to ensure it is used responsibly. Proposals include mandatory labelling of synthetic voices, strict consent protocols for voice data, and penalties for impersonation. Others advocate for a broader public conversation about the place of AI in our social and emotional lives.

Education also plays a role. As consumers and citizens, we need to understand how these tools work, what they can do, and where their limits lie. Only then can we engage meaningfully with the choices they present?

Voice, after all, is not just sound. It carries identity, emotion, and trust. If we allow it to be copied and deployed without rules, we may soon find ourselves unsure of who we are really talking to—and who is listening in.