🕑 Estimated reading time: 4 minutes  |  Last updated: January 21, 2025

AI helps attackers write convincing phishing messages in any language, clone voices, create deepfake videos and speed up research and coding. In early 2024, a finance employee at engineering firm Arup in Hong Kong transferred about US$25 million after a video call with deepfaked colleagues. At the same time, defenders use AI to detect anomalies, triage alerts and analyse malware, and AI applications themselves face new threats such as prompt injection. The best defences combine verification processes, phishing-resistant authentication and secure AI governance.

A computer circuit board with a brain on it

A phone call from your boss asking you to move money urgently used to be a simple request. Now you have to wonder whether the voice is real. Artificial intelligence has handed criminals powerful new tools, and defenders are racing to keep up.

Artificial intelligence has moved from research labs into everyday tools in just a few years. Large language models write emails and code; image, audio and video generators produce media that is increasingly hard to distinguish from reality. For cybersecurity, this is a double-edged sword. Criminals have new ways to deceive and attack, while defenders gain powerful tools to detect and respond. And the AI systems that organisations deploy create new vulnerabilities of their own.

How Attackers Are Using AI

Better phishing at scale

Generative AI can produce fluent, personalised messages in many languages, removing the poor grammar and awkward phrasing that once gave scams away. Attackers can feed in information scraped from social media and company websites to tailor spear-phishing emails to individual targets in seconds.

Voice cloning

With only a short audio sample from a video, podcast or voicemail, voice-cloning tools can imitate a person’s voice. Criminals use this in “CEO fraud” calls to finance staff and in scams that impersonate family members in distress asking for urgent money.

Deepfake video

Real-time and pre-recorded deepfake video can make it appear that a trusted person is speaking. The most striking case so far came to light in 2024: a finance employee at the Hong Kong office of British engineering firm Arup joined a video call with what appeared to be the company’s chief financial officer and other colleagues. All of them were deepfakes. Convinced, the employee made a series of transfers totalling around HK$200 million, roughly US$25 million.

Faster reconnaissance and coding

AI can summarise information about targets, explain unfamiliar systems and help write scripts. Security researchers and AI developers have reported attempts by criminal and state-linked groups to use AI chatbots for such tasks. Current evidence suggests AI mainly makes existing attackers more efficient rather than creating entirely new capabilities, but that may change as models improve.

Fake identities and documents

Generated profile photos, synthetic identity documents and AI-written CVs are used to open fraudulent accounts, bypass weak identity checks and even apply for remote jobs under false identities.

New Risks From AI Applications

Organisations deploying chatbots, AI assistants and AI-powered automation face new categories of vulnerability. The OWASP Top 10 for Large Language Model Applications highlights several, including:

Employees pasting confidential information into public AI tools is another common risk, as data may be stored or used in ways the organisation cannot control.

How Defenders Are Using AI

Defending Against Deepfake and Voice Fraud

  1. Never rely on voice or video alone for high-value requests. Verify through a separate, pre-agreed channel.
  2. Use multi-person approval for large or unusual payments, with mandatory call-backs to known numbers.
  3. Establish code words with family members and for executive requests in organisations.
  4. Train staff on deepfake scenarios, emphasising that urgency and secrecy are red flags regardless of how real a person seems.
  5. Limit public exposure of executives’ voice and video where practical, and monitor for impersonation.
  6. Strengthen identity verification for account recovery and onboarding with liveness checks and multiple signals.

Using AI Safely in Your Organisation

Verify unusual requests through a second channel

AI does not change the fundamentals of cybersecurity, but it raises the stakes. Seeing and hearing are no longer believing, so organisations must build verification into their processes rather than relying on instinct. At the same time, used wisely, AI can help defenders keep pace with threats that are growing in speed and sophistication.

Frequently Asked Questions

How can I tell if a video call is a deepfake?

Visual glitches, odd lighting or lip-sync problems can be clues, but deepfakes are improving. The safest approach is independent verification for any unusual request.

Is it safe to use ChatGPT or similar tools at work?

Only in line with your organisation’s policy. Avoid entering confidential or personal data into public tools unless approved.

Will AI replace cybersecurity jobs?

AI is more likely to change security roles than eliminate them, automating routine tasks while increasing demand for skilled analysts.

Leave a Reply

Your email address will not be published. Required fields are marked *