AI is a family of techniques (statistics, search, machine learning, deep learning) used to make computers do things that look intelligent — recognise images, translate text, play games, generate writing. Today the dominant flavour is deep learning powered by large neural networks.

Deep learning powers vision, speech, and almost all modern NLP. It needs lots of data and compute, but it learns useful representations directly from raw input rather than relying on hand-crafted features.

Diffusion models iteratively denoise from random noise to produce a sample. Distinct from autoregressive models like LLMs, though hybrid architectures exist. Behind almost every text-to-image and text-to-video product.

Before deep learning, most of the work in an ML project was choosing and shaping features: bucketing ages, encoding categories, deriving ratios, building rolling aggregates. Deep learning largely replaced this with learned representations, but feature engineering is alive and well in tabular problems — fraud scoring, credit risk, and most security detection pipelines still lean on it heavily.

Term popularised by Stanford CRFM (2021). Foundation models are the substrate that gets fine-tuned, prompted, or RAG-ed for specific applications. Most LLMs sold as APIs are foundation models; you customise them via prompts or fine-tunes.

GenAI covers LLMs (text), image and video diffusion models, audio generation, and code assistants. The legal, IP, and security implications are materially different from classical predictive AI, which is why regulators treat it as its own category.

An LLM is a transformer trained on vast amounts of text to predict the next token given the preceding ones. From this simple objective emerges fluent writing, in-context learning, code generation, and reasoning behaviour. Claude, ChatGPT, Gemini and Llama are all LLMs.

Classic ML covers linear / logistic regression, decision trees, random forests, gradient boosting, support-vector machines, k-means, etc. It powers anti-fraud, recommendation, churn prediction. Most production "AI" pipelines still rely on plenty of classical ML.

Multimodality unlocks document understanding, screen-reading agents, and voice interfaces — and expands the attack surface accordingly. An image can carry instructions the user never sees, which makes indirect prompt injection viable through a screenshot or a scanned PDF.

A neural network maps inputs (text tokens, image pixels, audio samples) to outputs through a stack of weight matrices and non-linear activation functions. Training adjusts the weights so the output matches the ground-truth labels (or, for LLMs, the next token).

Self-attention is the heart of the transformer. For each token, it computes weighted sums of all other tokens in the context. Multi-head attention runs many of these in parallel. Variants like Multi-Query (MQA) and Grouped-Query Attention (GQA) trade accuracy for speed / memory.

SLMs trade breadth of knowledge for latency, cost, and the ability to run entirely on-device — which also means the data never leaves the machine. Usually produced by distilling a frontier model and training on carefully curated or synthetic data. The natural fit for privacy-sensitive classification, on-prem redaction, and offline assistants.

Introduced in "Attention Is All You Need" (Vaswani et al. 2017), the transformer replaced recurrent networks with self-attention. Every modern LLM is a transformer of one variant or another (decoder-only, encoder-decoder, mixture-of-experts).

Anthropic ships several tiers so you can route each task to the cheapest model that handles it. Claude is known for long-context performance, Constitutional AI alignment, and strong tool-use and agentic behaviour. MCP, now the de-facto standard for connecting models to tools, originated here. See the dated model snapshot at the top of this page for current tier names and prices.

DeepSeek's mixture-of-experts and reasoning models demonstrated that frontier capability did not require frontier-lab spending, which reset industry assumptions about training cost. Open weights enable self-hosting, at the cost of large multi-GPU setups because the full parameter count must still fit in memory.

Distillation produces faster, cheaper models while retaining much of the teacher's capability. Almost every "small" production model is distilled from something larger. Note the licensing trap: distilling from an API model may breach its terms of service.

Gemini handles text, image, audio, and video from the ground up rather than bolting vision onto a text model, and has consistently pushed the upper bound on context-window size. Gemma is the open-weights cousin distilled from the same research.

The open counterpart to Gemini, distributed under a Gemma licence. Smaller variants target on-device and edge deployment where sending data to an API is not an option.

The GPT line is the general-purpose family accessed via API or ChatGPT, with long-context and multimodal variants. Alongside it OpenAI ships reasoning models that spend tokens on internal deliberation before answering — slower and more expensive per call, stronger on maths, code, and multi-step logic.

Released under a community licence rather than a standard open-source one. Open weights enable on-prem and EU-data-residency deployments, local fine-tuning, and air-gapped use. The range runs from 1–3B on-device variants through to models that need a multi-GPU server.

Mistral helped popularise mixture-of-experts in the open-weights world. Many models ship under Apache 2.0, while the flagship tier is API-only. Often the default choice for EU-headquartered organisations with data-residency requirements.

An MoE model might hold hundreds of billions of parameters but activate only a few tens of billions for any given token. It wins on capacity-per-FLOP, but the full parameter count must still fit in memory at inference — which is why MoE serving is a multi-GPU game even though the compute cost looks small.

Llama, Mistral, Qwen, Gemma, Phi and DeepSeek are all open-weights. "Open-source" is stricter and would also require open training data and recipe, which is rare. Open weights enable on-prem deployment, fine-tuning, air-gapped use and EU data residency — the usual reasons a regulated organisation picks them.

Phi pioneered the "training data quality beats raw scale" thesis: small models trained on textbook-quality data outperform substantially larger ones trained on scraped web text. The practical proof that useful models can run on phone-class hardware.

Strong multilingual performance, especially for Chinese, and one of the widest size ranges in open weights — useful when you want the same model family across an on-device tier and a server tier. A closed flagship is available via API.

Reasoning models spend tokens on internal deliberation before the final answer. They beat standard LLMs on maths, code, and logic but cost more in both tokens and latency. Most frontier models now do this adaptively — deciding per request how much to think — rather than exposing a fixed thinking budget.

Alignment combines training-time techniques (RLHF, DPO, Constitutional AI), evaluation (red-teaming, benchmark harnesses), and run-time defences (guardrails, output filters). Safety teams at frontier labs spend the bulk of their effort here.

Reduces reliance on humans for every label. The "constitution" is a short, published rule-set (helpful, harmless, honest) the model is taught to follow when self-critiquing. Auditable in a way that a pile of human preference labels is not.

DPO directly optimises the policy on preference pairs using a closed-form objective. Often used as a cheaper replacement for, or follow-up to, RLHF.

A practical superpower of LLMs: you don't need to fine-tune for many tasks; you just include examples in the prompt. The model learns the pattern in-context.

Training compute is the headline number for "how big" a model run was, and it has become a regulatory trigger rather than just a bragging right. The EU AI Act sets its systemic-risk threshold for general-purpose models at 10^25 FLOPs (Art 51), which means the size of your training run determines which obligations apply.

No weights are updated; the model uses the examples in its context to infer the task. The bigger the context window and the model, the better the in-context learning.

Closely related to SFT. The dataset is a mix of instructions ("Translate this to French") with their ideal responses. Without it, base LLMs autocomplete weirdly instead of "answering questions".

LoRA (Low-Rank Adaptation) freezes the base model and trains only small low-rank matrices that augment selected layers — often around 1% of full-model size, and easily merged at inference. QLoRA combines LoRA with INT4 quantization, bringing fine-tuning within reach of a single consumer GPU.

Iteratively training on synthetic outputs amplifies model errors and reduces output diversity — rare events in the original distribution disappear first. A live concern as the open web fills with AI-generated content.

Pre-training is where most of the model's "knowledge" comes from. Datasets are typically multi-trillion tokens of web text, code, books and papers. It takes weeks-to-months on thousands of GPUs and produces a "base model" that has not yet been instruction-tuned.

Humans rank multiple model outputs; a "reward model" learns to predict their rankings; the LLM is then trained to maximise that reward. It is what turns a raw next-token predictor into something pleasant, refusal-aware, and on-format.

SFT is the "teach the base model how to follow instructions" step. Inputs are pairs of prompt + ideal response. Far cheaper than pre-training. Often used to specialise a model on a domain (medical, legal, customer-support) or a house output format.

Most newer models bootstrap a large share of their post-training data this way — it is cheaper, more controllable, and avoids some privacy exposure. Care needed: model collapse and circular reasoning if you train only on AI outputs.

BPE produces a fixed vocabulary (typically 32k–200k tokens) that handles any input by falling back to byte-level pairs, so it never encounters an unknown character. Variants: byte-level BPE and SentencePiece BPE.

Each model family has a specific chat template with its own special tokens and role tags. Mixing templates breaks the model in subtle ways — degraded output rather than a clean error. Serving frameworks apply the right template automatically; hand-rolled prompt assembly is where this bites.

Distinct from "lost in the middle", which is about position: context rot is about volume. Accuracy, instruction-following, and format adherence all degrade as the window fills, even when the relevant fact sits at the very start. The practical consequence is that a 1M window is not an invitation to send 1M tokens — curate what goes in.

Frontier models now ship 1M-token windows as standard, with some going further. But advertised and effective context are different numbers: recall degrades well before the hard limit, and cost scales with what you actually send. Big windows enable RAG-less workflows; they do not make retrieval obsolete.

A "prompt" can be a single user message or a structured stack: system prompt + chat history + tool definitions + retrieved documents + the user's latest message. "Prompt engineering" is the discipline of writing prompts that produce the desired output reliably.

SentencePiece treats text as a raw byte sequence and learns a sub-word vocabulary directly. Unlike BPE with pre-tokenization, it does not depend on whitespace splitting, which helps multilingual and no-space languages.

EOS = end-of-sequence. Each model has its own stop tokens. Clients usually also support custom stop sequences — stop generating when the model emits a given string. A missing or mismatched stop token is a classic cause of runaway generation and runaway cost.

The system prompt sets persona, rules, output format, and tool access. Treat it as configuration, not as a secret: it is recoverable by a determined user, so it must never contain credentials, and it cannot be your only access control.

Modern tokenizers split text into roughly 3–5 character chunks via byte-pair encoding or SentencePiece. Common English text averages about 4 characters per token. Code is denser (2–3 chars/token); CJK can be 1–2. APIs charge per token, hence the obsession.

Each model family has its own tokenizer, and the same text yields different token counts on different ones. Tokenizers also change between versions of the same family, so a prompt that fitted a context budget last year may not this year. Never estimate one vendor's token count using another vendor's tokenizer.

Once standard for machine translation. Now uncommon for open-ended generation; modern LLMs prefer sampled decoding, which produces more diverse, less templated output.

Two distinct mechanisms. Compaction summarises earlier history into a compact block. Context editing prunes outright — dropping stale tool results and old reasoning without summarising. Both are essential for agents that run for hours; both silently discard information, so anything load-bearing belongs in a file or memory store rather than in chat history.

Rather than a fixed token budget, current APIs expose an effort level (typically low through max) that governs how much the model deliberates and how thoroughly it works. It is the single most useful cost lever on reasoning models: dropping one level often halves spend with no measurable quality loss on routine work. Sweep it against your own evaluation set rather than defaulting to the maximum.

Useful for tasks where you want a stable answer — extraction, classification, structured output. Beam search is a multi-path generalisation, now rarely used for chat LLMs.

Inference dominates LLM cost in production: every chat is one or many inference calls, and training is a one-off. Optimising it is the job of vLLM, TensorRT-LLM, llama.cpp and friends.

During generation the model computes attention only for the new token and re-uses K/V tensors from previous tokens. The KV cache grows linearly with context length and batch size, and dominates VRAM at long contexts — often exceeding the weights themselves. Quantising it to INT8 or INT4 is a big lever.

At inference time, the model outputs a probability for every possible next token; the sampler picks one. Greedy = always the top probability (deterministic, repetitive). Temperature above 0 plus top-p / top-k gives creative and diverse output. Note that several frontier APIs have removed these knobs entirely, steering behaviour through prompting and effort settings instead.

A common 2–3x throughput win when the draft model agrees with the target on most easy tokens. Output is mathematically identical to running the large model alone — it is a pure latency optimisation, not a quality trade.

Implemented as Server-Sent Events (SSE) or chunked HTTP. It cuts perceived latency dramatically and is effectively required for any long response — non-streaming requests with large output limits tend to hit HTTP timeouts.

Temperature is applied to the logits before softmax. Practical defaults: 0 for code and extraction, 0.3–0.7 for assistants, 0.8–1.0 for creative writing. Above about 1.5 things get nonsensical. Worth knowing that temperature 0 never guaranteed byte-identical output.

Dominated by prompt processing (the "prefill" phase), so long contexts mean high TTFT. Prompt caching, speculative decoding and quantisation all help. On reasoning models, TTFT also includes however long the model spends thinking.

Rough figures: small local models on a laptop reach tens of tokens per second; a large model on a single datacentre GPU is in the same ballpark; hosted frontier endpoints span roughly 30–200 tok/s depending on tier and load. Batch throughput on a server is a different and much larger number than single-stream latency.

A hard cut-off, simpler than top-p. Less common in modern stacks because top-p adapts better to flat versus peaked distributions.

Top-p caps the long tail of low-probability tokens, reducing rambling without removing diversity. Common value: 0.9. Often used together with temperature.

Decoder-only (GPT, Claude, Llama) predicts the next token. Encoder-only (BERT, RoBERTa) is good for classification and embeddings. Encoder-decoder (T5, Whisper) suits translation and structured transformation. Encoder-only models are still the right, cheap answer for a great many classification jobs.

Embeddings are produced by a dedicated embedding model and used for semantic search, clustering, classification and RAG. Typical dimensionality is 768–3072. Security note: embeddings are not anonymisation — the source text can often be substantially reconstructed from them.

The de-facto attention kernel on modern GPUs: substantially faster and dramatically lighter on VRAM, with mathematically identical output. Invisible to end users but the thing that underpins practical long-context inference.

Reduces the KV cache by roughly 4–8x with negligible quality loss. This is the single change that made long-context inference practical on consumer hardware, and it is now near-universal in open-weights models.

A 32-head attention layer learns 32 different "perspectives" simultaneously. MHA is memory-heavy at long context, which is precisely what the GQA and MQA optimisations exist to fix.

Maximum KV cache savings, small accuracy cost. Largely superseded by GQA, which recovers most of the quality at nearly the same memory footprint.

Near-universal in modern LLMs. Much easier to extend to longer contexts than absolute positional embeddings — RoPE scaling is how a model trained at one context length gets stretched to a much longer one after the fact.

AWQ (Activation-aware Weight Quantization) and GPTQ are post-training quantisation methods, roughly equivalent in quality at INT4. Supported across the main GPU serving stacks. Distinct from GGUF, which targets CPU and mixed CPU/GPU inference.

GGUF bundles weights, tokenizer and metadata in a single file. Common variants such as Q4_K_M (~0.56 bytes/param) and Q8_0 (~1.06) let you pick a memory/quality point. Replaces the older GGML format.

NVIDIA dominates the datacentre, with AMD and Google TPUs as the credible alternatives. On the desktop, VRAM capacity matters more than raw speed for LLM work. Apple Silicon is a genuine third option for local inference thanks to large unified memory, at lower throughput than a discrete GPU.

Hosts hundreds of thousands of models, datasets and demo apps, and publishes the Transformers, Datasets, Tokenizers and Accelerate libraries. Also a genuine supply-chain risk surface: unsigned artefacts, typo-squatted repos, and model files that execute code on load.

Supports CPU, NVIDIA and AMD GPUs, Apple Metal, and Vulkan. Reads GGUF directly, ships as a single binary. The de-facto tool for running LLMs on a laptop, and the basis of most "local AI" products.

Pull a model by name and you have a local server exposing an OpenAI-compatible API. Bundles a model registry and a clean CLI. The fastest route from nothing to a private, offline model — and worth knowing about defensively, since it is also how unsanctioned local models arrive on corporate laptops.

FP32 (4 bytes/param) → FP16 / BF16 (2) → INT8 (1) → INT4 (0.5). Quality loss is small at INT8, modest at INT4 for 7B+ models, and larger on small models. A quantised large model usually beats an unquantised small one at the same memory budget.

Compiles model graphs ahead of time and supports speculative decoding, in-flight batching and FP8. Production users at scale often graduate from vLLM to TensorRT-LLM, trading flexibility and ease of deployment for throughput.

TPUs are matrix-multiply specialists optimised for transformer workloads, available via Google Cloud. The programming model is JAX / XLA rather than CUDA, which is the main practical barrier to switching.

Built around PagedAttention, which manages the KV cache like paged virtual memory and largely eliminates the memory fragmentation that used to cap batch sizes. Production-grade serving on NVIDIA and AMD. The usual default for self-hosted serving at scale.

Rule of thumb: weights = parameters × bytes-per-parameter (FP16 = 2, INT4 = 0.5), plus the KV cache (which grows with context length and batch size), plus roughly 10% activation overhead. At long context the KV cache often dominates. The VRAM Calculator on this site does the full breakdown.

Agent = LLM + tools + planner + memory. Examples: support agents, research assistants, coding agents, autonomous SOC triage. Risk scales with autonomy, and the important question for a security review is not what the agent is for but what credentials its tools hold.

Usually a directory of files or a managed store the agent reads and writes with ordinary file tools. It measurably improves long-horizon performance. It is also a persistence mechanism for an attacker: anything written to memory is replayed into every future session, so a single successful injection can become a durable backdoor. Never store credentials there.

Where MCP connects an agent to tools, agent-to-agent protocols connect an agent to other agents across organisational boundaries. Early but strategically important, and it raises questions traditional IAM has no answer for: how do you authenticate a request from a counterparty's autonomous agent, and who is accountable for what it does?

The famous prompt addition "let's think step by step" boosted performance on maths, code and logic. Modern reasoning models bake this into training and do it automatically, so the explicit instruction is now largely redundant on frontier models — and can even hurt.

Common strategies: fixed-size with overlap, semantic (paragraph or heading boundaries), and structural (Markdown sections, code blocks). Bad chunking is the single most common cause of bad RAG — before blaming the model, look at what the retriever actually returned.

Lets a model operate software with no API, which is exactly why it is attractive for legacy and internal systems. It is also the highest-risk agent pattern in production: the model acts inside an authenticated session, every web page it renders is untrusted input, and a screenshot can carry instructions invisible to the user.

The successor discipline to prompt engineering. Once agents run for hours across many tool calls, the hard problem stops being how you phrase the instruction and becomes what you keep in the window: which history to retain, when to compact, what to move to files, and how to avoid burying the important instruction under 400k tokens of tool output.

The control point most organisations discover they need on their second or third LLM project: one place to hold provider credentials, enforce redaction, apply per-team budgets, log prompts for audit, and switch providers without touching application code. Also the natural enforcement point for shadow-AI controls.

MLOps adapted for a world where the model is a third-party API that changes under you. Distinctive concerns: prompts are versioned artefacts, evaluation sets are as important as tests, provider model updates are an uncontrolled dependency change, and cost is a runtime variable rather than a fixed capacity plan.

MCP standardises how clients expose resources, prompts and tools to LLM applications, so one agent can connect to your repository, ticket system, filesystem and database through a consistent interface. Now broadly adopted across vendors. Its success is also its risk: a single compromised MCP server sits inside every agent that trusts it.

Largely standardised on MCP now. The security boundary is unchanged and frequently misunderstood: a connector runs with whatever credentials you grant it, and the model decides when to invoke it. Grant scope per connector, not per user.

Loop: Thought → Action → Observation → Thought. Often outperforms pure chain-of-thought because the model gets to ground its reasoning in real outputs rather than its own assumptions.

Initial vector search returns the top 50; a reranker picks the best 5–10 to feed the LLM. A large quality win for small extra latency, and usually a cheaper fix for poor RAG results than swapping the generation model.

The dominant pattern for "answer questions about my data". Components: chunking, embedding, vector store, retriever, optional reranker, prompt template, LLM. Beats fine-tuning for fresh or per-tenant data. Security note: retrieved content is untrusted input, and is the primary delivery vector for indirect prompt injection.

Modern APIs enforce this at the decoding level rather than merely asking politely, so the output is guaranteed to parse. Essential for anything downstream of the model in a pipeline. It guarantees shape, not truth — a schema-valid response can still be wrong.

A coordinator fans work out to subagents that explore in parallel and report back. Genuinely effective for wide, independent work such as searching many files at once. It also multiplies cost and latency, since every subagent re-establishes its own context — so a cap on how many may spawn is a standard control.

The insight that reset the field: you can buy capability at inference time, not only at training time. Reasoning models, best-of-N sampling and verifier loops are all forms of it. Practically, it turns model quality into a per-request cost dial rather than a fixed property.

You define a JSON schema for each tool; the model emits a structured tool call your runtime executes; the result is fed back. This is the mechanism behind every agent. The tool definition is also a security boundary: whatever credentials the tool holds, the model can reach.

Available as dedicated products (Pinecone, Weaviate, Qdrant, Milvus, Chroma) or as extensions to databases you already run (pgvector, Redis, MongoDB). Indexes: HNSW, IVF-PQ. Metric: cosine similarity, most commonly. For most organisations, the extension to an existing database is the right answer.

Coined by Andrej Karpathy in 2025. Genuinely productive for prototypes and throwaway tooling. The security concern is specific and well evidenced: unreviewed generated code ships with hardcoded secrets, missing authorisation checks, and outdated dependency versions, and nobody on the team can explain what it does when it breaks.

Each benchmark targets a capability: general knowledge, code, graduate-level science, real-world software engineering. Treat published scores with caution — contamination (models seeing test data during pre-training) is endemic, and a benchmark that everyone optimises for stops measuring anything useful.

Online (production telemetry: thumbs up/down, escalation rate, refund rate) versus offline (a curated test set scored against ground truth or by an LLM judge). Your own small evaluation set on your own data is worth more than any public leaderboard, and is the only way to safely change model or prompt.

RAG citations, tool-call results, explicit source attribution. Reduces hallucination and, just as importantly, makes outputs auditable — which is what turns an LLM feature into something you can defend in a regulated process.

Models hallucinate citations, function names, dates, CVE identifiers and court decisions with complete fluency. Mitigations: RAG with citations, structured output validation against real data, refuse-when-unsure prompting, and an eval harness that checks against ground truth. Fluency is not a confidence signal.

Scales evaluation beyond manual review. Methods: pairwise preference, rubric scoring, faithfulness checking. Known biases: position bias, length bias, and a preference for its own family's output. Mitigate with explicit rubrics, ensembles and human spot-checks.

From a 2023 paper: recall is high at the start and end of the context and dips in the middle. Practical implication — don't bury the important instruction or document halfway through a long RAG dump. Put critical content at the start or the end.

The standard measure of effective long-context recall as opposed to advertised context size. Multi-needle and reasoning-over-needles variants are harder and more informative — retrieving one fact verbatim is much easier than combining several scattered ones.

Perplexity = exp(loss). Used in pre-training as the optimisation target and useful for comparing models on a fixed corpus. Close to useless for evaluating instruction-following, helpfulness or alignment.

Frontier labs run dedicated red teams; open tooling includes PyRIT and garak. Required by EU AI Act Art 55 for general-purpose models with systemic risk. For an application team the useful version is narrower: attack your own agent's tool permissions, not the base model's safety training.

A classical ML threat against image classifiers. The LLM equivalents are jailbreak suffixes, instructions encoded in images, and invisible Unicode tag characters that the model reads but the user cannot see. Defences: adversarial training, input normalisation, and stripping non-printing characters.

Threat surface: typo-squatted packages, poisoned datasets, model files that execute code on deserialisation, malicious notebooks, and compromised tool servers. Mitigations: an AI bill of materials, signed releases, safe serialisation formats, and scanning artefacts before import rather than after.

Usually planted via data poisoning or a malicious fine-tune. The model passes normal evaluation cleanly and flips behaviour only when the trigger appears, which makes detection by testing close to hopeless. Rely on trusted training data and provenance rather than on catching it downstream.

A classic access-control problem that agents reproduce almost perfectly: the agent holds broad credentials and acts on instructions that may have arrived from anywhere, including a web page it fetched. The fix is the classic one — carry the requester's authority through to the action rather than acting with the agent's own ambient privilege.

Especially relevant for fine-tuning and RAG corpora, where the barrier to contributing data is low. Attackers publish poisoned datasets hoping you will include them. Mitigations: dataset provenance, signed releases, distributional baselines, and scanning fine-tune data as carefully as you would scan a dependency.

Now a mainstream business-email-compromise technique: a cloned voice or a video call impersonating an executive, approving an urgent transfer. The countermeasure is procedural rather than technical — out-of-band verification for payment and access changes, and removing "I recognised their voice" as an acceptable control.

The vulnerability that turns a prompt injection into an incident. A support agent with read-only ticket access is an annoyance when hijacked; the same agent with a delete-user tool and an admin token is a breach. Controls are ordinary least-privilege ones: scope each tool narrowly, use per-task credentials, and require human confirmation for irreversible actions.

Defence in depth on top of model-level alignment: a second, cheaper model or rule set that inspects what goes in and what comes out. Useful and worth having — but a guardrail that only inspects text does nothing about an agent whose tool just deleted a database.

Particularly nasty for agents — a malicious page can hijack a browsing agent into exfiltrating the user's session, and the user sees nothing unusual. Defences: never let retrieved content trigger privileged tools, sanitise rendered HTML, apply output classifiers, and require confirmation for any outbound action.

The model is not a trusted component. Rendering its output as HTML gives you XSS; passing it to a shell gives you command injection; interpolating it into SQL gives you SQL injection; writing it to a path it chose gives you traversal. Every classical injection defence applies unchanged — validate, escape, parameterise.

Variants: role-play framing, encoded payloads, multi-turn drift, character-by-character assembly, adversarial suffixes. Often combined with prompt injection. Distinct from prompt injection in target: a jailbreak attacks the model's training, an injection attacks the application built on it.

A privacy attack directly relevant to GDPR and training-data provenance questions. Large models memorise and can reproduce some training data verbatim, especially low-frequency strings such as keys and identifiers. Mitigations: deduplication, differential privacy, output filters.

ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) maps real-world attacks on ML and LLM systems into familiar tactic/technique structure — reconnaissance, model access, poisoning, evasion, exfiltration — with documented case studies. The right shared vocabulary for AI threat modelling with a security team that already thinks in ATT&CK.

Defences: query rate limits, output watermarking, terms of service. Costly at scale but very feasible for narrow capability theft — you rarely need to clone the whole model, only the part that does your competitor's valuable task.

The finding that matters operationally: embeddings are not anonymisation. Given an embedding vector and the model that produced it, an attacker can recover a great deal of the original text. Treat a vector database of customer documents with the same controls as the documents themselves.

Covers prompt injection, insecure output handling, training-data poisoning, model denial of service, supply-chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance and model theft. The natural checklist to structure an LLM application review around, and the one auditors are most likely to recognise.

The pragmatic control between "ban the tool" and "accept the risk". Runs at the gateway: detect names, identifiers, keys and account numbers; redact or tokenise; optionally restore in the response. Rarely perfect, and its real value is as much about creating an auditable record of what left the organisation as about the redaction itself.

Direct: the user types "ignore previous instructions and…". Indirect: instructions hidden in fetched web pages, emails, calendar invites or RAG sources. The most common LLM attack, and structurally unsolved — there is no reliable way to separate instructions from data in a single token stream. Defend by constraining what the model can do, not by trying to filter what it reads.

The visible surface of safety training. Two failure modes: over-refusal (blocks legitimate security or medical work and drives users to unmonitored tools) and under-refusal (unsafe). Frontier APIs now surface refusals as a distinct response state rather than an error, so applications should handle it explicitly.

Any agent that executes code needs a container or VM with no ambient credentials, restricted egress, and resource limits. Two failure modes matter in practice: genuine escape from the sandbox, and the far more common case of a sandbox that was never isolated in the first place — mounted with the host filesystem, or holding an environment variable with a production key.

Shadow IT with a much lower barrier to entry: a browser tab, a personal account, and confidential material pasted into a consumer chatbot or a locally-run model. Bans reliably fail; what works is providing a sanctioned route that is genuinely good enough, routed through a gateway that logs and redacts.

Reliably achievable given persistence, so design on the assumption it will happen. The real damage is rarely the prompt text; it is what people put in it — API keys, internal URLs, undocumented tool names, and the exact rules to be talked around. Keep secrets out of the prompt and enforce authorisation server-side.

Tool descriptions are read by the model as trusted instructions, so a server can smuggle directives into them — and a server that behaves for weeks can change its definitions afterwards, which is the "rug pull". Controls: pin and review tool definitions, alert on changes, and run third-party connectors with their own scoped credentials.

Two flavours: output watermarking, which biases the sampler so the text carries a detectable distribution, and provenance metadata such as C2PA content credentials. Machine-readable marking of synthetic content is required under EU AI Act Art 50. Both are fragile against a motivated adversary — paraphrasing strips one, re-encoding strips the other.

The prerequisite for every other AI control: you cannot classify systems under the AI Act, respond to a model-provider incident, or answer a customer questionnaire without knowing what you run. Extends the SBOM idea to weights, datasets, prompts and connectors. Building it usually surfaces a surprising amount of shadow AI.

A system in an Annex III area is presumed high-risk unless the Art 6(3) carve-outs apply (narrow procedural task, and so on). Provider duties follow: risk management, data governance, documentation, human oversight, and conformity assessment. Note how much ordinary internal tooling lands here — CV screening and employee monitoring both do.

Tiers: prohibited (Art 5), high-risk (Annex III + Art 6), general-purpose AI (Art 51–55), transparency-only (Art 50), and minimal. Prohibitions applied from February 2025 and general-purpose model duties from August 2025, with the bulk of the high-risk obligations following in 2026. Extraterritorial: it applies to output used in the EU regardless of where you are.

Used in policy discussion and voluntary commitments, typically tied to compute thresholds such as 10^25 FLOPs. Frontier providers attract the most regulatory attention and publish the most safety documentation — which, pragmatically, makes them the easiest vendors to diligence.

GPAI providers must keep technical documentation, respect copyright opt-outs, and publish a training-data summary. Systemic-risk models carry additional duties: risk evaluation and mitigation, serious-incident reporting, and cybersecurity obligations. If you fine-tune a model substantially, you may become a provider yourself.

Drawn up with the AI Office and industry, it translates the Act's general-purpose chapters into concrete commitments on transparency, copyright and safety, and signing it is the practical route to demonstrating compliance. For downstream deployers it is useful leverage: ask a prospective model vendor whether they are a signatory, and read what they filed.

Mechanistic interpretability attempts to reverse-engineer the internal circuits of LLMs; output-level interpretability covers attention maps, feature attribution and concept probes. Increasingly practical rather than academic — the same techniques that identify a learned feature can be used to monitor for or steer away from it.

Where 42001 specifies the management system, 23894 gives the risk-management guidance that feeds it — how to identify, analyse and treat AI-specific risks inside an existing enterprise risk framework. Useful for slotting AI risk into a register your board already reads.

Defines requirements for an AI management system: governance, risk, lifecycle, supplier management and transparency. Certifiable, which is what makes it commercially useful — it is becoming the operational backbone organisations use to evidence EU AI Act readiness to customers.

The methodology standard for assessing an AI system's impact on individuals and society, covering scope, timing and documentation. Maps closely onto the fundamental-rights impact assessment expected for high-risk systems under the EU AI Act, so one exercise can satisfy both.

Most AI controls are not new: access control, logging, supplier management, change management and incident response all apply, with AI-specific interpretations. Mapping them saves duplicated audit effort and, crucially, keeps AI risk inside a governance structure that already has attention and budget.

Model cards (Mitchell et al. 2018) describe models; datasheets (Gebru et al. 2018) describe datasets. Now effectively mandatory — required by the EU AI Act for high-risk and general-purpose models, and expected in any serious procurement. Read the "limitations" section first; it is where the useful information is.

NIS2 does not mention AI specifically, which is exactly the point: if you are in scope, an AI system supporting an essential service is simply part of your network and information systems. It inherits the same risk-management, supply-chain and incident-reporting obligations — including the 24-hour early warning.

Four functions (Govern, Map, Measure, Manage), voluntary, and complementary to the NIST Cybersecurity Framework. The Generative AI Profile (NIST AI 600-1) extends it with LLM-specific risks. Frequently the pragmatic starting point for organisations that already run a CSF programme.

The main practical tool of mechanistic interpretability: it pulls apart the dense, overlapping representations inside a network into sparse features that often correspond to recognisable concepts. Those features can then be monitored, or amplified and suppressed to steer behaviour — a control surface that operates below the prompt layer.

Applies to chatbots, content generators, emotion recognition, biometric categorisation and deep-fakes, and requires machine-readable marking of synthetic content. The lightest-touch tier of the Act and the one most likely to catch an organisation that thought it had no AI Act exposure at all.

In the absence of comprehensive federal legislation, US states have legislated individually — Colorado on algorithmic discrimination in consequential decisions, California and others on disclosure and training-data transparency, several on deepfakes and hiring. Obligations vary and effective dates shift, so anyone operating nationally tracks them as a live matrix rather than a fixed list.

All the major providers offer around a 50% discount when you can wait, typically up to 24 hours. Ideal for evaluations, bulk classification, backfills, dataset creation and embeddings at scale — anything without a user waiting on the other end.

The largest single cost lever in most production deployments. Two forms: routing classifies the request up front and picks a tier; cascading tries a cheap model first and escalates on low confidence or a failed check. Both need an evaluation set to prove the cheap path is actually good enough.

Cache reads cost roughly a tenth of normal input tokens; writes carry a modest premium, so it pays back within two or three requests. It is a strict prefix match, which is the part teams get wrong: a timestamp or session ID near the start of the system prompt invalidates everything after it. Keep the stable content first and the volatile content last.

Common gotchas: prompt-caching savings left on the table, output verbosity blowing the budget, routing every task to the most expensive model, agent loops generating thousands of tokens per request, and evaluation runs quietly costing more than production. Self-hosting shifts cost from per-token to hardware and staff — it does not remove it.

Two distinct controls, and you want both. A hard cap enforced by your infrastructure or the API stops runaway spend. A task budget tells the model how many tokens it has for the whole job so it paces itself and finishes cleanly rather than being cut off mid-sentence. Agent loops are where surprise invoices come from.

The spread across tiers is wide — roughly a tenfold difference between the cheapest and most capable models from the same vendor, and more across vendors. Output costs several times input, so verbose responses hurt disproportionately. Caching and batching are the two discounts available to almost everyone.