Self-hosted LLM deployment in the EU
Control prompts, retrieval data, model serving, logs, and support access with an architecture your team can inspect and operate. EU hosting is one control—not a shortcut to GDPR compliance.
Sovereignty is a control model, not a hosting label
A useful definition of sovereign AI is the documented ability to govern where data is processed, who can access it, which software and model versions run, how the system can be changed, and how it can be exited or recovered. Physical location matters, but so do corporate access, subprocessors, telemetry, licenses, encryption control, operator capability, and portability.
Legal status: the GDPR is binding law. EDPB, CNIL, and ENISA publications cited here are authoritative or expert guidance, not a compliance certificate. The architecture and delivery recommendations on this page are Yarify's engineering interpretation and are not legal advice.
Choose the least complex model that meets the control objective
“Cloud versus self-hosted” hides several workable options. Write down the prohibited data flows, required operator control, service level, workload shape, exit need, and team capability before selecting infrastructure.
| Deployment model | Control profile | Operational load | Good fit when |
|---|---|---|---|
| Shared model API | Provider controls model and most serving infrastructure; customer controls application and contract choices | Lowest | Data is permitted, provider terms and transfers are acceptable, and speed matters more than infrastructure control |
| Managed EU endpoint | Regional processing and managed operations, with control depending on provider architecture, support, and subprocessors | Low | EU processing location is required but the organization does not need to operate the model server |
| Dedicated EU tenant | Stronger isolation and configuration control while facilities or platform layers remain managed | Medium | Stable workload and stricter isolation justify reserved capacity without full platform ownership |
| Self-hosted in EU cloud | Organization controls serving stack, network, storage, keys, and deployment; cloud provider still supplies infrastructure | High | Custom controls, model portability, or restricted data paths justify an internal operating capability |
| On-premises or private site | Maximum physical and network control, with hardware lifecycle, power, cooling, and recovery owned locally | Highest | Isolation or local processing is mandatory and the organization can operate the complete platform |
An EU region does not prove that every administrator, support tool, backup, telemetry pipeline, or subprocessor stays in the EEA. Verify the actual processing chain and contract. GDPR Chapter V governs transfers of personal data to third countries or international organizations. GDPR official text.
Map every copy before drawing the cluster
The processing inventory is the architecture input. For each copy, record purpose, data categories, people, system, region, recipient, retention, deletion path, encryption, access, and controller or processor role.
Prompts and attachments
Raw user input, uploaded files, parsed text, temporary objects, browser or gateway caches, and rejected requests.
Retrieved knowledge
Source documents, chunks, embeddings, metadata, access-control attributes, search results, and quoted evidence.
Model serving
Tokenizer input, context and KV cache, generated output, batching buffers, safety filters, and temporary files.
Observability
Application logs, traces, prompt captures, metrics labels, evaluation samples, feedback, alerts, and crash reports.
Operations
Backups, snapshots, replicas, support bundles, administrator access, secrets, artifact registries, and vulnerability scans.
Improvement
Human corrections, accepted answers, preference data, fine-tuning sets, test cases, annotations, and model checkpoints.
GDPR control map for the deployment record
Purpose, lawfulness, and minimisation
Define each use and legal basis; restrict input and retained context to what is necessary. Article 5 principles apply even when the model runs locally.
Roles and supply chain
Identify controller, joint-controller, and processor roles; document Article 28 terms, subprocessors, support paths, and instructions.
Privacy by design
Apply Article 25 controls through defaults: narrow access, short retention, disabled training reuse, redacted traces, and separated environments.
Security
Select Article 32 measures from risk: access control, encryption, resilience, restore testing, logging, patching, incident response, and effectiveness testing.
Impact assessment
Assess whether Article 35 requires a DPIA, especially for likely high risk, special-category data, vulnerable people, systematic evaluation, or large-scale processing.
Rights and lifecycle
Make notice, access, correction, objection, restriction, erasure, retention expiry, backup handling, and model or index replacement operationally possible.
The EDPB says anonymity of an AI model must be assessed case by case and requires it to be very unlikely both that personal data can be extracted and that outputs can be linked to people whose data was used for training. It does not create a blanket exemption for open or locally hosted weights. EDPB Opinion 28/2024.
A reference architecture with separate trust boundaries
Keep identity, authorization, retrieval, inference, and operations separable. This makes policy enforceable, data flows inspectable, and individual components replaceable.
Identity and policy gateway
Authenticate the actor, attach tenant and purpose, enforce quotas and input policy, and create a correlation ID before model access.
Application orchestrator
Own workflow state, prompt assembly, tool schemas, deterministic rules, approval gates, retry policy, and safe failure behavior.
Authorized retrieval
Query permission-filtered indexes, preserve document and version provenance, cap context, and treat retrieved text as untrusted data.
Inference gateway
Route approved model versions, enforce context and output limits, isolate tenants where required, and expose consistent metrics and cancellation.
Model serving pool
Run pinned artifacts in restricted workloads with explicit accelerator allocation, network policy, health checks, rolling replacement, and no implicit internet egress.
Evidence and operations
Store minimized audit events, metrics, evaluation results, access changes, deployment records, incidents, backups, and deletion evidence under distinct retention rules.
Control plane
Keep model registry, signed or hashed artifacts, configuration, secrets, policy, deployment approval, vulnerability state, and rollback outside the model runtime. Record model source and license before distribution.
Data plane
Deny unnecessary outbound traffic, encrypt transport and stored data, keep secrets out of prompts, avoid content-rich logs by default, and make sensitive debug capture temporary and approved.
Secure RAG is an authorization problem first
Retrieval-augmented generation does not make an answer factual or permitted by default. The application must control which evidence can be retrieved, how it is used, and what the user can verify.
Ingest with provenance
Keep source system, owner, tenant, document ID, version, classification, permissions, validity dates, and deletion state with every chunk.
Filter before similarity
Resolve the authenticated actor and eligible document set before vector or keyword ranking. Never retrieve broadly and hide results afterward.
Treat content as hostile
Retrieved documents can contain prompt injection. Delimit evidence, prohibit instruction inheritance, restrict tools, and test malicious documents.
Return inspectable evidence
Show source, version, passage, and uncertainty for consequential claims; make abstention the correct result when evidence is missing or conflicting.
Deletion must propagate. Removing a source document should invalidate chunks, embeddings, caches, replicas, and evaluation copies according to the documented retention policy. An index without a deletion ledger is not governable.
Security covers the platform and the model lifecycle
Self-hosting transfers security work to the operator. Apply ordinary infrastructure controls and add controls for artifacts, datasets, prompts, retrieval, outputs, and model change.
Supply chain
Approve model and library sources, verify hashes or signatures, scan artifacts, record licenses, pin versions, and retain a reproducible build record.
Least privilege
Separate users, service accounts, tenants, environments, administrators, and backup operators. Re-check authorization at every downstream system.
Network and secrets
Use private endpoints where justified, deny default egress, rotate scoped credentials, keep secrets outside prompts and images, and monitor administrative paths.
Input and output
Limit size and type, scan attachments, encode downstream output, validate schemas, block unsafe tool arguments, and never execute generated code by default.
Model-specific tests
Test extraction, membership or memorization risk where relevant, poisoning, evasion, prompt injection, denial of service, unsafe output, and tenant leakage.
Operations
Patch hosts and runtimes, test restore and rollback, define incident ownership, monitor capacity and abuse, expire debug data, and rehearse model or index replacement.
ENISA recommends layering established ICT cybersecurity foundations with AI-specific and sector-specific practices across the lifecycle. Its framework is guidance, not certification. ENISA framework.
Capacity comes from workload evidence, not a GPU shopping list
Model architecture, precision, context length, KV cache, batching, serving engine, accelerator, and quality settings all affect throughput and memory. Benchmark a vertical slice before reserving production capacity.
Build a workload profile
Measure arrival rate, concurrency, prompt and retrieved-context distribution, output length, task classes, peak windows, streaming need, and availability target.
Benchmark accepted quality
Compare model and quantization candidates on the same representative tasks. Record task acceptance alongside first-token latency, output rate, queue time, memory, and errors.
Test failure and headroom
Load-test peaks, dependency slowdown, long contexts, cancellation, rolling deployment, accelerator loss, restore, and fallback. State the headroom and degradation policy explicitly.
Annual total-cost model
compute and accelerator capacity + idle and resilience headroom + storage and network + platform operations + security and compliance + observability + evaluation and model updates + facilities or provider support + expected incident and migration cost
Compare options using cost per accepted business task and the value of the control objective. Cost per generated token ignores retrieval, review, failed outputs, idle capacity, and staff.
Queue sanity check
For a stable system, average work in progress is approximately arrival rate multiplied by average time in the system. Use this Little's Law relationship to challenge concurrency assumptions, then validate them under the real serving stack. It does not replace a load test or account for burst shape by itself.
A bounded healthcare RAG example
“European healthcare” is not a sufficient scope. A defensible first use has a named user, permitted corpus, measurable answer, and explicit clinical boundary.
Use
Help authorized staff locate current approved procedures and assemble a cited administrative answer for review.
Data boundary
Use role-filtered institutional documents; exclude patient records unless the approved purpose specifically requires them.
Hard stop
Do not diagnose, prioritize treatment, alter a medical record, or conceal absent and conflicting evidence.
Evidence
Measure retrieval authorization, citation support, abstention, outdated-document rejection, harmful-answer rate, review time, and correction.
Health data is special-category data under GDPR Article 9. A lawful design must identify both the Article 6 basis and an applicable Article 9 condition where personal health data is processed; local hosting does not supply either. GDPR Articles 6 and 9.
Delivery gates for a production deployment
Each gate should leave a reviewable artifact. Do not buy the full cluster before workload quality, privacy, security, and operating assumptions survive a representative vertical slice.
Frame the control objective
Name the use, prohibited flows, data classes, roles, availability, latency, exit requirement, owner, current baseline, and go/no-go evidence.
Map data and obligations
Complete processing records, lawful-basis review, processor and transfer map, retention, rights workflow, security risk analysis, and DPIA decision.
Benchmark a vertical slice
Run representative tasks through identity, authorized retrieval, candidate serving stacks, evaluation, and observability; compare accepted quality and cost.
Build the operating path
Automate artifact verification, infrastructure, policy, secrets, deployment, rollback, backup, restore, patching, access review, monitoring, and incident evidence.
Validate before release
Perform privacy and security review, adversarial tests, load and recovery tests, human-workflow evaluation, documentation, training, and explicit acceptance.
Release and govern change
Use shadow or limited traffic, defined rollback thresholds, versioned evaluation, capacity alarms, incident ownership, periodic access and retention review, and an exit rehearsal.
The EU AI Act is a separate classification exercise
Self-hosting, open weights, and an EU data center do not decide whether an AI system is prohibited, high-risk, subject to transparency duties, or outside those categories. Classify the concrete system by intended purpose, actor role, deployment context, and the functions it performs.
Record the provider, deployer, importer, distributor, or other roles; intended users; affected people; inputs; outputs; human decision; and sector. Keep AI Act and GDPR work connected, but do not collapse one into the other. EU AI Act official text.
Self-hosted LLM FAQ
Does hosting an LLM in the EU make it GDPR compliant?
No. EU hosting can reduce transfer exposure and give an organization more control over storage, access, and retention, but GDPR compliance depends on the complete processing operation. That includes purpose and lawful basis, controller and processor roles, minimisation, transparency, data-subject rights, security, retention, DPIA requirements, subprocessors, support access, and any transfers outside the EEA.
Is a self-hosted open-weights model automatically private?
No. The application may still copy personal or confidential data into prompts, retrieval indexes, traces, backups, analytics, or support systems. The model itself may also require a documented assessment if its training involved personal data. Privacy depends on the model provenance and every component in the data path, not merely possession of the weights.
What is the most important security rule for self-hosted RAG?
Do not treat vector similarity as authorization. Authenticate the user, apply tenant and document permissions before retrieval, re-check access before returning evidence, preserve source provenance, and prevent retrieved text from overriding system instructions or invoking tools. A vector database must not become a parallel, weaker access-control system.
When is on-premises deployment better than an EU cloud?
On-premises can be appropriate where policy requires local isolation, network connectivity is constrained, latency to equipment matters, or the organization already operates suitable accelerators securely. An EU cloud or dedicated tenant is often easier when demand is variable and the team needs managed facilities, scaling, and hardware replacement. Decide from the control objective, workload profile, and operating capability—not a sovereignty label.
How should a team size infrastructure for an internal LLM?
Benchmark the intended model and serving stack with the organization’s real prompt lengths, retrieved context, output lengths, concurrency, and quality settings. Measure time to first token, output rate, end-to-end latency, queue time, memory use, failures, and accepted-task quality. Then add explicit availability and peak headroom. Public tokens-per-second claims are not a capacity plan.
Can a healthcare organization use self-hosted RAG safely?
It can support bounded uses such as locating approved procedures, drafting administrative correspondence, or assembling evidence for a qualified reviewer. Health data requires a lawful basis and Article 9 condition where applicable, strict purpose and access controls, logging and retention discipline, and often a DPIA. Clinical decisions should not be silently delegated to a general-purpose assistant.
Primary sources and status
Reviewed 2 September 2026. The GDPR and EU AI Act are binding regulations. EDPB and CNIL materials are data-protection authority guidance; ENISA provides cybersecurity guidance. Requirements and official guidance should be rechecked for the concrete organization, sector, and deployment date.
- Regulation (EU) 2016/679 — General Data Protection RegulationBinding EU legal text covering processing principles, privacy by design, processors, security, DPIAs, and international transfers
- EDPB Opinion 28/2024 on personal data in AI modelsAuthoritative supervisory guidance on when an AI model may be anonymous, legitimate-interest assessment, and unlawfully processed training data
- CNIL recommendations for GDPR-aware AI system developmentData-protection authority guidance on purpose, roles, legal basis, minimisation, retention, rights, security, and model-status assessment
- ENISA Multilayer Framework for Good Cybersecurity Practices for AIEU cybersecurity guidance combining baseline ICT controls, AI-specific controls, and sector-specific risk treatment
- Regulation (EU) 2024/1689 — EU AI ActBinding EU legal text whose obligations depend on the system, intended purpose, risk classification, and actor role—not hosting location