🕑 Estimated reading time: 4 minutes | Last updated: January 21, 2025
AI helps attackers write convincing phishing messages in any language, clone voices, create deepfake videos and speed up research and coding. In early 2024, a finance employee at engineering firm Arup in Hong Kong transferred about US$25 million after a video call with deepfaked colleagues. At the same time, defenders use AI to detect anomalies, triage alerts and analyse malware, and AI applications themselves face new threats such as prompt injection. The best defences combine verification processes, phishing-resistant authentication and secure AI governance.

A phone call from your boss asking you to move money urgently used to be a simple request. Now you have to wonder whether the voice is real. Artificial intelligence has handed criminals powerful new tools, and defenders are racing to keep up.
Artificial intelligence has moved from research labs into everyday tools in just a few years. Large language models write emails and code; image, audio and video generators produce media that is increasingly hard to distinguish from reality. For cybersecurity, this is a double-edged sword. Criminals have new ways to deceive and attack, while defenders gain powerful tools to detect and respond. And the AI systems that organisations deploy create new vulnerabilities of their own.
Table of Contents
How Attackers Are Using AI
Better phishing at scale
Generative AI can produce fluent, personalised messages in many languages, removing the poor grammar and awkward phrasing that once gave scams away. Attackers can feed in information scraped from social media and company websites to tailor spear-phishing emails to individual targets in seconds.
Voice cloning
With only a short audio sample from a video, podcast or voicemail, voice-cloning tools can imitate a person’s voice. Criminals use this in “CEO fraud” calls to finance staff and in scams that impersonate family members in distress asking for urgent money.
Deepfake video
Real-time and pre-recorded deepfake video can make it appear that a trusted person is speaking. The most striking case so far came to light in 2024: a finance employee at the Hong Kong office of British engineering firm Arup joined a video call with what appeared to be the company’s chief financial officer and other colleagues. All of them were deepfakes. Convinced, the employee made a series of transfers totalling around HK$200 million, roughly US$25 million.
Faster reconnaissance and coding
AI can summarise information about targets, explain unfamiliar systems and help write scripts. Security researchers and AI developers have reported attempts by criminal and state-linked groups to use AI chatbots for such tasks. Current evidence suggests AI mainly makes existing attackers more efficient rather than creating entirely new capabilities, but that may change as models improve.
Fake identities and documents
Generated profile photos, synthetic identity documents and AI-written CVs are used to open fraudulent accounts, bypass weak identity checks and even apply for remote jobs under false identities.
New Risks From AI Applications
Organisations deploying chatbots, AI assistants and AI-powered automation face new categories of vulnerability. The OWASP Top 10 for Large Language Model Applications highlights several, including:
- Prompt injection: attackers craft inputs, sometimes hidden in web pages, emails or documents the AI reads, that override the system’s instructions and make it reveal data or take unintended actions.
- Sensitive information disclosure: models revealing confidential data from their training or connected systems.
- Excessive agency: AI agents with broad permissions to send emails, make purchases or change systems, which an attacker could hijack.
- Insecure output handling: passing AI output directly into other systems without validation.
- Supply chain risks: compromised models, datasets or plugins.
Employees pasting confidential information into public AI tools is another common risk, as data may be stored or used in ways the organisation cannot control.
How Defenders Are Using AI
- Anomaly detection: machine learning has long been used to spot unusual login patterns, network traffic and transactions.
- Alert triage: AI assistants summarise alerts, correlate events and suggest next steps, helping overstretched security teams work faster.
- Phishing detection: models analyse language, sender behaviour and links to flag suspicious messages.
- Malware analysis and threat intelligence: AI helps reverse engineers understand code and summarise threat reports.
- Secure coding: AI tools can identify vulnerabilities in code, though their own suggestions must also be reviewed.
Defending Against Deepfake and Voice Fraud
- Never rely on voice or video alone for high-value requests. Verify through a separate, pre-agreed channel.
- Use multi-person approval for large or unusual payments, with mandatory call-backs to known numbers.
- Establish code words with family members and for executive requests in organisations.
- Train staff on deepfake scenarios, emphasising that urgency and secrecy are red flags regardless of how real a person seems.
- Limit public exposure of executives’ voice and video where practical, and monitor for impersonation.
- Strengthen identity verification for account recovery and onboarding with liveness checks and multiple signals.
Using AI Safely in Your Organisation
- Publish clear rules on which AI tools staff may use and what data may be shared with them.
- Prefer enterprise AI services with contractual data protections.
- Threat-model AI applications for prompt injection and data leakage before deployment.
- Give AI agents the minimum permissions they need and require human approval for sensitive actions.
- Log and monitor AI interactions for misuse.
- Keep up with regulation, such as the EU AI Act, which sets security requirements for certain AI systems.
Verify unusual requests through a second channel
AI does not change the fundamentals of cybersecurity, but it raises the stakes. Seeing and hearing are no longer believing, so organisations must build verification into their processes rather than relying on instinct. At the same time, used wisely, AI can help defenders keep pace with threats that are growing in speed and sophistication.
Frequently Asked Questions
How can I tell if a video call is a deepfake?
Visual glitches, odd lighting or lip-sync problems can be clues, but deepfakes are improving. The safest approach is independent verification for any unusual request.
Is it safe to use ChatGPT or similar tools at work?
Only in line with your organisation’s policy. Avoid entering confidential or personal data into public tools unless approved.
Will AI replace cybersecurity jobs?
AI is more likely to change security roles than eliminate them, automating routine tasks while increasing demand for skilled analysts.