English
Español
Cybersecurity8 min read
AI and Cybersecurity: the double-edged sword redefining the field.
AI as an offensive weapon
APT groups and cybercriminals are not waiting for the security sector to adopt AI. They are already using it. LLMs enable generating hyper-personalised phishing campaigns in any language at massive scale, without the spelling mistakes that previously betrayed fraudulent messages.
Audio and video deepfakes have already been used to impersonate executives in Business Email Compromise (BEC) attacks. In 2024, a Hong Kong financial firm transferred million believing they were in a real video call with their CFO.
LLM-specific threats
LLM01
Prompt Injection
An attacker manipulates the system instructions by inserting text in user input. The model ignores the original context and executes the attacker's instructions.
LLM02
Insecure Output Handling
The LLM generates malicious code, XSS or SQL injection that the application executes directly without validation. Every output must be treated as untrusted input.
LLM06
Sensitive Information Disclosure
Models trained on private data can "leak" sensitive information. Prompts can extract personal data, passwords, or business logic from the system context.
LLM08
Excessive Agency
When an LLM agent has too many permissions, an attacker can manipulate it to perform unauthorised actions: delete files, send emails, or execute transactions.
Defence: Input validation against Prompt
Injection (Python)
All user input must be validated before being sent to an LLM. Known injection patterns must be explicitly detected and rejected.
# LLM Prompt Injection Detection (Python)
INJECTION_PATTERNS = [
"ignore previous instructions",
"disregard your system prompt",
"you are now",
"jailbreak",
]
def validate_prompt(user_input: str) -> str:
lower = user_input.lower()
for pattern in INJECTION_PATTERNS:
if pattern in lower:
raise ValueError(f"Potential prompt injection: {pattern!r}")
sanitized = user_input[:2000].replace("<", "<").replace(">", ">")
return sanitized
def safe_llm_call(system_ctx: str, user_input: str) -> dict:
return {
"model": "gpt-4o",
"messages": [
{"role": "system", "content": system_ctx},
{"role": "user", "content": validate_prompt(user_input)}
],
"temperature": 0.3,
"max_tokens": 512
}
"AI will not replace security analysts. But analysts who use AI will replace those who don't."
Zenith — Primitive Labs
AI as a defensive tool
- 01
Anomaly detection — ML models that learn normal network behaviour and alert on deviations without predefined rules.
- 02
Malware analysis — Automatic classification of unknown samples via dynamic and static analysis with neural networks.
- 03
AI-powered SOAR — Automated incident response: alert correlation, prioritisation, and playbook execution without human intervention for known cases.
- 04
Phishing detection — Analysis of email headers, domain reputation, and content with classifiers trained to detect zero-day phishing.
- 05
Threat intelligence — LLMs to analyse CTI reports, correlate TTPs with MITRE ATT&CK, and generate detection rules automatically.
