AI governance
Every AI call in the company goes through one door.
The company already uses artificial intelligence. The question is whether it knows who calls, what is spent, what leaves the perimeter, and what happens when the vendor goes down. We are specialists in deploying that door, and it sits wherever you want it: in your cloud account, on your cluster or in your datacenter, hardened with the same security practice that sustains PCI-DSS and ISO 27001 environments.
The starting point
The company already uses AI.
It just does not know where.
This is not hypothetical, it is the picture every discovery finds: each application talking straight to its own vendor, each credential kept somewhere different, each usage landing on a different invoice. And halfway across, the empty rectangle: the place where someone could ask who called, what it cost and what was sent, and where today there is nothing.
Six origins inside the company calling four AI providers directly, each with its credential kept somewhere different and billed to a different account.
Halfway across there is only an empty dashed outline, in the exact place the gateway would sit. The calls cross that space without meeting any control.
-
Nobody knows what is being spent
The invoice arrives as one number. There is no way to say which team, product or experiment consumed what, so there is no way to cut what does not pay or defend what does.
-
The key is everywhere
API keys in repositories, in automations, in spreadsheets and on the laptop of whoever left. With no inventory, revoking one means first finding where it is used.
-
There is nothing to audit
When someone asks what the company sent to a third-party model, and that question does arrive, the answer has to exist. Without a single door, it does not.
-
Switching model means rewriting
Each application talks straight to its own vendor. A new model, a better price or a contractual requirement becomes an engineering project instead of a configuration change.
The single door
Applications stop talking to vendors.
They talk to policy.
One internal address receives every AI call in the company and decides what to do with it: whether the key is allowed, whether the budget permits it, whether the data may leave, which model serves it and what gets recorded. After that, it goes out. Before that, it may not.
Five applications calling a single gateway, which applies identity, budget, data policy, routing and logging before forwarding to three destination groups: cloud LLMs, open models, and agents and MCP servers.
One of the five calls is refused at the door for exceeding budget, before it leaves the company.
What the gateway delivers
Six controls that only exist
where there is a single door.
-
Cost
A budget that actually exists
Limits per key, per team and per project, applied on the call rather than in next month’s report. An overrun is refused, not discovered.
- Spend ceiling per key, team, project and period
- Token consumption attributed to whoever called
- Alert and automatic cut-off on reaching the limit
- Cost-based routing: the task goes to the tier it actually requires
- Response caching for whatever repeats
-
Access
A virtual key instead of the real one
The vendor credential never leaves the gateway. Each application gets a key of its own, with scope, expiry and immediate revocation.
- A virtual key per application, team or person
- An allowed-model list per key
- Expiry and rotation without stopping the application
- Immediate revocation, with no hunt for where the key was
- The vendor credential lives in exactly one place
-
Audit
A record of what was asked
Who called, which model, when, what it cost and which policy applied. It is the record the standard asks for, and that legal will ask for first.
- A trail per request, with author, model and cost
- Retention policy defined by content type
- Export to the observability tool the company already uses
- Evidence ready for an AI governance audit
-
Data
What leaves the perimeter, and what does not
Content rules applied before the call, not entrusted to the developer of each application. Sensitive data stops at policy, not at code review.
- Masking of personal data before sending
- Blocking by content pattern and by task type
- Choice of provider region, where one is offered
- Routing of sensitive tasks to a local model
-
Continuity
The application does not fall because the vendor did
Alternate routing, retry and distribution across providers. One model going down stops being the company’s incident.
- Automatic fallback across providers and regions
- Retry with backoff, without double billing
- Load distribution across keys and accounts
- Rate limit per key, so one team does not eat another’s quota
-
Independence
Switching model is configuration
A single interface across every provider. The application calls the gateway, and the decision of which model serves it lives in policy, not in code.
- A compatible interface across the whole application estate
- A new model enters without touching an application
- Self-hosted open models behind the same door
- Cost and outcome comparison between models on real traffic
What it looks like
Governance stops being a document
and becomes a screen.
Policy that lives in a slide deck does not survive the first quarter. After deployment, granting access, changing a limit and knowing who spent what are three screens, operated by the people who already look after the environment.
01Access
Every application with a key of its own.
The vendor credential stays in exactly one place, and each application, team or person gets a key of their own. Granting access stops being handing out a secret, and revoking it stops being a hunt for where it ended up.
- Creation, expiry and revocation without touching an application
- An allowed-model list per key
- Who used it, when, and what it cost, per key
| Key | Team | Last active | Spend / ceiling |
|---|---|---|---|
| atendimento-prodsk-…9f2a | serviceactive | 2 min ago | US$ 4,182/ 10,000 |
| copiloto-engsk-…tt09 | engineeringactive | 3 min ago | US$ 6,740/ 12,000 |
| agente-chamadosk-…pq77 | operationsactive | just now | US$ 3,301/ 8,000 |
| portal-websk-…7b01 | marketingactive | 5 min ago | US$ 1,905/ 5,000 |
| ci-avaliacaosk-…hh55 | qualityrate limited | 7 min ago | US$ 58/ 250 |
| laboratoriosk-…ww21 | researchexpired | 38 min ago | US$ 121/ 500 |
02Policy
The limit applies on the call, not in the report.
Spend ceilings, rate limits, permitted models and content rules live in the key policy. When the ceiling is reached the call is refused, rather than discovered at month end.
- A ceiling per key, team, project and environment
- Personal data masking and prompt-injection blocking
- Log retention defined by content type
Limits
production 6,000 · staging 2,000 · test 500
Guardrails
Allowed models
03Spend
The invoice stops arriving as one number.
Every request is attributed to whoever called, and consumption shows up per team, per key and per model. That is what lets you charge whoever uses it, cut what does not pay, and defend what does.
- Chargeback by area, with numbers finance will accept
- Cost compared between models on real traffic
- Export to the tool the company already uses
US$ 41,257
By team
By model
Illustrative screens from the panel that comes with the deployment. The names and figures are fictional.
Where it is used
The same door solves
six different problems.
- 01
Ending shadow AI
The first job is almost always finding what is already running. The gateway turns scattered use into visible use, and it is from there that policy stops being a document.
- 02
Chargeback by area
Each team sees its own consumption and answers for it. Without that, AI sits in the technology cost centre and never reaches the budget of whoever uses it.
- 03
Migration between providers
Switching vendor, or splitting traffic between two, becomes a policy adjustment. That is what lets you negotiate a contract from a position of strength.
- 04
Separate environments per stage
Development, staging and production with different quotas, models and policies. Testing stops spending the production budget.
- 05
Local models in the same place
An open model running in-house answers through the same interface as the external provider. A sensitive task changes destination without changing code.
- 06
The base for agents
An agent that executes needs limits, a record and the ability to be cut off. Putting agents in before the gateway is putting automation in without brakes.
Where it runs
Because it is open,
you choose where it sits.
AI governance should not require sending all of the company’s traffic to yet another intermediary’s service. Because the software is open, the topology is decided by whoever owns the data, and the four below are the ones we deploy.
-
The default for anyone already on cloud
In your own cloud account
Inside your VPC, with no public address, and egress passing through a controlled path. Neither the traffic nor the records cross third-party infrastructure, ours included.
-
For a mature platform
On the cluster you already have
An existing Kubernetes or ECS, under the same delivery, observability and alerting patterns as the rest of your platform. No new island for the team to learn to operate.
-
For a regulatory restriction on egress
In your datacenter
On-premise, with the open model running next to the door. The task that cannot leave the building does not leave, and is still served through the same interface as the rest.
-
For control without the headcount
Operated by haapit
On infrastructure we run, under a contracted SLA, with the NOC and the 24x7 SOC watching. The control exists from day one, without the company having to build a team for it.
Who deploys it
The software is open.
The deployment is the work.
LiteLLM is free, and anyone can bring a container of it up in an afternoon. What does not come in the container is the decision of where it sits, who may call, what gets masked, where the trail goes, how the secret is rotated, what happens when a provider goes down, and how all of that is proven to an auditor. That is where we are.
-
The company is not locked into us
An MIT licence, configuration versioned in the client’s own repository, and no proprietary haapit piece in the path. If one day they want to move the operation to another supplier, they can, and that is precisely why a client trusts putting all of the company’s AI behind it.
-
It sits wherever the company wants
Open source means the topology is decided by whoever owns the data. There is no obligation to send traffic to yet another intermediary’s service in order to get governance.
-
The security can be ours, not a catalogue’s
In a closed product, the available control is whatever the vendor chose to expose. With the code open, we apply our own practice: masking, hardening, segregation, our own guardrails, and the trail going to the SIEM the company already uses.
-
Whatever is missing, we write
The door has extension points before and after every call. A rule the company needs and that exists nowhere off the shelf goes in there, tested, versioned and supported as production code.
What we deliver
LiteLLM licence MIT
- Design of the topology, the perimeter and high availability
- Access, ceiling and retention policy, decided with the client before it becomes configuration
- Our own hardening, masking and guardrails
- Integration with the SIEM, the observability stack and the secret manager already in place
- Migration of the applications one at a time, with no downtime window
- Support under SLA, SOC on-call, and periodic policy review
model_list: - model_name: redacao-longa # the name the application knows litellm_params: model: bedrock/claude-sonnet aws_region_name: sa-east-1 # the data stays in Brazil - model_name: redacao-longa # same name, different provider litellm_params: model: azure/gpt-4o api_base: os.environ/AZURE_API_BASErouter_settings: routing_strategy: usage-based-routing-v2 num_retries: 2 # retry before giving upgeneral_settings: master_key: os.environ/LITELLM_MASTER_KEYAn illustrative excerpt. Both providers are published under the same internal name, so the application asks for redacao-longa and never knows which one answered. The variables are filled by the secret manager as the service starts, and are never in a file or an image.
Securing the door
Bringing the container up is easy.
This part is not.
The door comes to concentrate the company’s most sensitive AI credential and its most sensitive records. Built without discipline, it is not a new control: it is a new target. That is why who deploys it matters, and it is here that our security practice comes in.
-
Secrets and identity
The door concentrates the company’s most sensitive AI credential. If it leaks, everything leaks at once, so the secret never sits wherever is convenient.
- Vendor credentials in a secret manager, never in a repository or an image
- Injected only at runtime, with rotation that does not stop the service
- Least-privilege virtual keys: allowed model, ceiling and expiry
- Application authentication against the gateway, not just a static key
- Separate credentials per environment, with no production key in staging
-
Network and perimeter
A single door is also a single target. It is treated as an exposed critical service, with the same perimeter discipline as an environment under certification.
- No public address: reachable only from inside the perimeter
- Egress through a controlled path, with an allowed-destination list
- TLS terminated by us, with encryption in transit end to end
- Rate limiting and edge protection against abuse and quota exhaustion
- Provider region chosen by data requirement, not by their default
-
Data and records
The gateway’s log holds what the company asked the models. That is sensitive data by definition, and it is treated as such rather than as an application log.
- Personal data masked before sending and before writing
- Records encrypted at rest, with retention defined per content type
- Trail replicated to the SIEM, out of reach of whoever administers the door
- Consumption database separate from the application databases
- LGPD compliance handled in the design, not patched in afterwards
-
Operation and verification
A control nobody tests is not a control. The door enters the same verification and on-call regime as the environments we already run under contract.
- High availability across more than one zone, with failure testing
- Configuration versioned in a repository and reviewed against a hardening baseline
- Pentesting of the door itself, by our own offensive security team
- Guardrails before and after the call, against prompt injection and context leakage
- Monitoring in the 24x7 SOC, with alerts on overruns and on out-of-pattern behaviour
The yardstick is the one we use in certified environments: PCI-DSS, ISO 27001 and SOC, with QSA, CISSP and CISA in the same team that runs the operation. On the AI side, the design is checked against the OWASP Top 10 for LLMs and the NIST AI RMF.
Deployment examples
Four situations,
and the decision each one asks for.
These are deployment patterns, with no client names: what matters here is the architecture and policy decision in each case, which is the part no tool comes with.
- 01
Customer service with customer data
- The situation
- An assistant that has to read the customer history to answer, with national ID, address and payment data sitting in the middle of the context.
- The setup
- The door inside the client VPC, with the region pinned to Brazil and egress through a controlled path.
- The policy
- Personal data masked before sending, an economy model for classifying and triaging, and a frontier model only where the writing needs it.
- The result
- Personal data does not leave identifiable, and the bill drops because most of the volume never needed the expensive model.
- 02
A copilot for the engineering team
- The situation
- The whole team wanting a coding assistant, each with their own key, and proprietary code going outside with no control.
- The setup
- A shared door, with a virtual key per person and a self-hosted open model alongside.
- The policy
- A monthly ceiling per person, and routing of proprietary code to the local model, through the same interface.
- The result
- Consumption attributed per person and per team, and the code that could not leave stops leaving, without taking the tool away from anyone.
- 03
An agent that executes in production
- The situation
- An agent that opens and closes tickets, queries inventory and changes configuration, which is to say an agent that acts rather than only answers.
- The setup
- Its own least-privilege key, separate from the application keys, with a trail of its own.
- The policy
- Rate limiting, a post-call guardrail, and key revocation as the stop button.
- The result
- The agent gets brakes, and the brakes can be pulled in seconds by whoever is on call, with no deploy required.
- 04
Two providers at the same time
- The situation
- Dependence on a single vendor, a contract up for renewal, and no data of your own to back the negotiation.
- The setup
- Two providers behind the same door, with configurable traffic splitting and alternate routing between them.
- The policy
- Cost and accuracy compared on real traffic, and one provider going down served automatically by the other.
- The result
- The renewal is negotiated with numbers measured in-house, and one vendor being unavailable stops being an incident.
Governance
The standard asks for evidence.
This is where it is produced.
ISO/IEC 42001 asks for records of decisions, access control, risk management and monitoring of what the AI system does. Without a single door, each of those has to be proven application by application, and in practice it is not proven. The gateway is the layer where that evidence appears on its own, which is why it tends to be the first item on the adoption path rather than the last.
Read the paper on ISO/IEC 42001How we deploy it
Five steps, and the policy
is decided before the configuration.
- 01
Discovery
A survey of what already uses AI in the company, on which key, at what spend and under which policy. There is almost always more than the inventory shows.
- 02
Policy
Who may call what, with which ceiling, which retention, and which data never leaves. Policy is decided before it becomes configuration.
- 03
Deployment
A highly available gateway, in the cloud or inside the perimeter, with a virtual key per application and the vendor credential isolated.
- 04
Migration
Applications are pointed at the gateway one at a time, with no downtime window. The direct key is revoked at the end of each one.
- 05
Operation
A consumption panel, periodic review of limits and policy, and support under the same SLA as the rest of the operation.
Start with the discovery, at no cost.
A survey of what already uses AI in your company, on which key, at what spend and under which policy. It is the picture that precedes any governance decision, and it usually comes as a surprise.