Private AI for security teams: what runs locally and what that buys you

Private AI means the models that power a product's AI features run inside your own environment, so the data they read never leaves it. For security tooling, that data is about as sensitive as it gets: privileged commands, session transcripts, credentials' metadata, and the audit trail itself.

Every security product now advertises AI. In regulated and sovereignty-constrained organisations, many of those features cannot be switched on, because they work by sending the content to a vendor's cloud. This article explains what to look for, and what to expect from models that run locally.

Why egress is the real question

Before asking what the AI can do, ask where it runs:

• Where does the content go? Is a session transcript or a command log sent to a third party to be summarised?
• Is it retained or used for training? Even if the answer is no, you are relying on a contract rather than a boundary.
• Does it still work when the network is closed? An isolated or air-gapped environment cannot call out. If an AI feature depends on the internet, it is unavailable exactly where assurance requirements are strictest.

A simple test follows: disable outbound connectivity and use every AI feature. Whatever still works is genuinely private.

What suits a local model

Models that run on your own hardware are smaller than the largest cloud models, but they are well suited to tasks that explain, draft and search, where a person reviews the output:

Task	Why it fits
Explaining why a session or login was scored risky	Summarising known factors in plain language
Drafting an access policy from a description	A person reviews and edits before it applies
Compiling a written access reason into a draft command scope	The approver reviews and can change it
Summarising audit events and answering questions about them	Retrieval over your own data
Drafting notifications and suggesting workflow improvements	Low risk, easy to review
Narrating system health in plain language	Turns metrics into a readable summary

These use cases share a property: the AI proposes and a person or a deterministic rule decides.

What should not depend on a model

Keep the following deterministic:

• Allow or block decisions on the command path. Pattern matching is fast, testable and explainable. A model's judgement is none of those. Use a model only for the grey zone, such as scoring an ambiguous command, never to overrule a clear rule.
• Authentication and authorisation. These are rules, not predictions.
• Evidence. Signed, hash-chained records should be generated from facts, not summarised by a model.

A useful rule: if you would be uncomfortable explaining to an auditor why the system did something, a model should not have made the call.

Expectations for models you run yourself

• Open-source models. Common choices are general-purpose models of around eight billion parameters for reasoning and summarising, a code-oriented model for parsing commands and policies, and a small embedding model for semantic search. You can inspect, pin and upgrade them on your own schedule.
• Hardware. They run on CPUs, and run much faster on GPUs. Size the deployment for your concurrency, and test response times with real workloads.
• Quality. They are less capable than the largest hosted models. Expect good summaries and drafts, and occasional mistakes. That is another reason the output needs a human or a rule behind it.
• No proprietary training. A product built on open-source models plus retrieval over your own data is not training anything on your data. That is a feature, not a limitation.
• Fallbacks. Where a model is unavailable, a good product falls back to rule-based behaviour, so automation continues while the AI is down.

Governance questions to ask a vendor

1. Which provider and models are used, and can the provider be pinned to local-only in production?
2. Is there a setting that prevents anyone from enabling a cloud provider by accident?
3. Are credentials, tokens and other sensitive identifiers removed before text is sent to a model?
4. Are AI requests logged with who asked, what for, and which model answered?
5. Are risk explanations traceable to the factors that produced them, rather than opaque verdicts?
6. Does every AI-suggested change need human approval before it takes effect?
7. What happens when the model is down?

A private-AI pilot

1. Disconnect outbound access from the deployment, at the firewall, not just in settings.
2. Exercise every AI feature: risk explanation, policy drafting, audit questions, access-reason compilation.
3. Record what still works and what does not. Anything that fails should fail visibly.
4. Watch the network for connection attempts to external addresses during the test.
5. Check output quality on your own data with the people who will use it.
6. Time it under realistic load.
7. Confirm the enforcement path does not wait on the model.

How ZenXPAM supports this

ZenXPAM's AI runs on local models inside your deployment, served through Ollama, and a local provider is enforced in production. It uses open-source models for general reasoning and behavioural analysis, for policy and command parsing, and for embeddings for semantic search, with no proprietary models needed. The features that use it include risk explanation, natural-language policy authoring, compiling an access reason into a draft command scope, audit summarisation and question answering, notification composition, workflow suggestions and health narration. The enforcement path is deterministic; the model is used only for grey-zone scoring. All of it keeps working with no outbound connectivity. See private AI with zero egress or book a demo.

FAQ

Is private AI as good as cloud AI? Not at everything. It is well suited to explaining, drafting and searching your own data, which is most of what security teams need from it, and it is available where cloud AI cannot be used.

Does private AI mean no data leaves at all? It means the AI features do not send content out. Check the rest of the product (telemetry, updates, support tooling) separately.

Can the AI block a command? In a well-designed product, no: blocking comes from deterministic rules. The model can help with ambiguous cases, and a person remains in control.

Does it need a GPU? It runs without one, but GPUs make responses much faster. Size it from a test, not a guess.

What about AI agents acting on our systems? Treat them as identities that need approval, a scope, monitoring and evidence, like any other privileged identity. That is a direction the industry is moving, and one to plan for now.

Related reading

• private ai session auditing pam risk scoring
• local llms vs cloud llms what enterprises trade off
• if ai is deciding which privileges are safe who is validating the model