The Missing Layer in Enterprise AI: A Governance Architecture for LLMs and Machine Learning

Most companies have systems for storing data, building AI models, and testing AI ideas. But fewer have a system for deciding which AI tools are safe enough to use with customers.

The chatbot that worked a little too well

Imagine a medium-sized insurance company that builds an AI assistant to help workers handle insurance claims in just six weeks. The workers love it. But then someone copies a customer's medical notes into the chatbot to get a quick summary. A month later, an employee responsible for checking that the company follows the rules asks a reasonable question: Where did those medical notes go? Who could see them? And which version of the AI model created the answer that was sent to the customer?

Nobody knows. The company does have records of what happened, but they are spread across three different systems. None of those systems recorded which model version was used. The project was supposedly approved, but that approval was just a presentation and someone agreeing during a meeting.

I have seen similar situations so many times that I no longer blame one particular team. This happens when a company builds systems for data, AI models, and applications but assumes that the rules and safety checks will somehow take care of themselves. They will not. Someone must design a proper system for managing AI responsibly.

Why the usual solution does not work

The usual response is to write a policy document and create a group of people to review AI projects. I understand why companies do this. Writing a policy feels like progress, and having a review group makes it seem like someone is responsible.

But the problem is how quickly things change. A group that meets once a month cannot properly review an AI system that changes every week. And a PDF document cannot stop an AI chatbot from accidentally sharing someone's phone number at 2 a.m. A written policy only explains what a company wants to happen. It becomes a real safety rule when the technology actually enforces it, records that it did so, and can show that record to someone checking it a year later.

There is also a reasonable argument against having too many rules. Strict controls can make it so difficult to experiment that teams stop trying new AI ideas. If a team needs approval from six different people just to test a prompt that summarizes text, it may stop asking permission. Workers might then use AI tools without telling the company. This is called shadow AI, and it can be even more dangerous. So the goal is not to create more paperwork. It is to put the right checks in the right places and make them automatic whenever possible.

Make governance part of the system, not just a committee

The biggest change in how I think about AI governance was to stop seeing it as a final approval step. Instead, I started seeing it as something that works alongside every part of the system, just like user accounts or computer networks. You would not let every team create its own login system. The same idea should apply to AI rules and safety checks.

Existing guidelines already support this approach. The NIST AI Risk Management Framework describes governing, identifying, measuring, and managing risks as things companies must keep doing. The EU AI Act sets different requirements depending on how risky an AI system is. In practice, this means companies must first know which AI systems they have and how risky each one is before they can follow the rules. Banks have used a similar approach for years under guidance about managing risks from models, such as SR 11-7 in the United States. These frameworks explain what companies should think about, but they do not tell companies exactly what technology to build. That is both useful and limiting. A clear technical design fills this gap.

Five parts and one shared record

Here is the structure I keep returning to. It is not the only way to organize AI governance. Your company might need four parts or six instead of five. The important thing is that every responsibility has a clear place and someone in charge of it.

The policy layer stores rules in a form that computers can understand. These include risk levels, rules about acceptable uses of AI, who can approve different activities, and which legal requirements connect to which safety checks. An AI model that makes decisions about customers' credit should face different checks from an internal tool that summarizes meetings. The policy layer is where those differences are defined.

The data layer keeps track of where information comes from. It records the history of data, whether people gave permission to use it, how sensitive it is, and how long it should be kept. For large language models, or LLMs, this also includes the documents stored in a search system that the AI uses to find information. Teams often forget this part. If the search system can access a file, the AI model might repeat information from that file.

The model layer is where AI models are registered and tested. Every model, reusable prompt, and specially trained version of a model gets a version number, a responsible owner, test results, and a short document explaining the model. Testing means checking more than whether the answers are correct. For models that give people scores, it includes checking for unfair bias. For LLMs, it includes checking whether answers are supported by reliable information, whether the AI correctly refuses certain requests, and whether it can resist harmful instructions hidden in its inputs. These attacks are called prompt injection.

The runtime layer is where safety rules are enforced while AI is being used. It includes checks on information going into and coming out of the model, limits on how often the system can be used, ways to send difficult cases to a human, and an emergency switch to turn the system off. The assurance layer checks what happens afterward. It watches for changes in model behavior, helps teams respond to problems, and sends reports to risk teams and company leaders.

All five layers share one record of evidence: logs that cannot be changed and that show decisions, responsible people, and the times when things happened. This may sound boring, but I would argue it is the most important part. If you cannot prove that your AI rules were followed, you are only claiming that they were.

How this works for a large language model

Let's return to the insurance company's claims assistant and see how the runtime layer would work.

First, someone sends a request to the AI. An input safety check looks for personal information and obvious attempts to trick the AI with harmful instructions. It also applies the rules for that person's job role. Next, the system searches for useful documents, but only among files the person already has permission to read. This is more important than many safety instructions written into prompts because it prevents the AI from seeing private information that it should not share. The AI then creates an answer. Another safety check looks at whether the answer is supported by the documents it found and whether it reveals sensitive information. If the AI is unsure or the decision could have serious consequences, the request goes to a human.

Every step is recorded in an audit log. This includes the user's prompt, the documents used, the model version, the results of safety checks, how long the request took, and the user's role. When the employee responsible for following the rules asks what happened, the company can answer in minutes instead of weeks.

But there are limits. AI safety checks do not always work. The OWASP list of risks for LLM applications exists because attacks that trick AI through instructions and problems involving leaked information have not been completely solved. No filter catches every problem. This design does not promise that nothing bad will happen. It promises that problems can be limited, noticed, and traced back to their source. That is a smaller but more realistic promise.

How this works for traditional machine learning

Traditional machine learning models need the same basic structure, but the checks happen at different stages. For example, a model that detects fraud or predicts customer demand goes through several steps during its life. Governance adds checks that must be passed before moving from one step to the next.

When data first enters the system, the company checks where it came from and whether it has permission to use it. Training must be repeatable, meaning someone else should be able to run the same process and get the same result. Testing checks how well the model works, whether it is unfairly biased, and whether it keeps working when conditions change. Approval requires a specific person to take responsibility. How carefully the model is reviewed depends on its risk level. When the model is released, the company first tests it with a small amount of real use, called a canary release, and has a plan to return to the previous version if something goes wrong. After release, the company keeps watching for drift, which means changes in the data or how the model behaves. If a measurement crosses a set limit, the model is trained again or replaced with an earlier version.

One stage that does not get enough attention is retirement. Old AI models often stay in use because nobody is responsible for deciding when to turn them off.

Who is responsible for each part?

A technical design without clear responsibility is just a picture on a wall. The approach I find most practical comes from a system of three levels of responsibility often used in financial services. It can also be made simple enough for smaller companies.

The teams building AI are responsible for their models and how they are used. They run tests and respond when warning systems find problems. A small governance team sets risk levels, reviews only the projects with the highest risks, and maintains shared safety tools. This means other teams can use those safety checks without building them themselves. Internal auditors and risk teams examine samples of the logs and test whether the safety checks actually work. They then report their findings to senior leaders.

Rules move from the governance team to the people building AI, while records showing what happened move back in the other direction. If teams must fill out a form every time they want to use a safety check, they may skip it. But if the safety check is already built into the platform, they will usually use it.

Where this approach can fail

There are two common ways this can go wrong. The first is governance that only looks good on paper: impressive policies but no real enforcement and no records. The second is the opposite problem: so many strict controls that teams find ways to avoid them.

I am also not certain that dividing governance into five layers is right for every company. A data team with only ten people probably does not need a separate governance department. It might just need one engineer responsible for keeping track of models and a regular habit of recording who approved each decision.

A practical plan for the first ninety days

I would not start by buying or building tools. I would start by making a complete list of the AI systems the company uses. You cannot manage systems you do not even know exist. Find every AI model and feature powered by an LLM, including those quietly used in spreadsheets and products bought from other companies. Give each one a risk level and name someone responsible for it.

Next, choose the system with the highest risk and make sure every step is recorded, from beginning to end. This should make it possible to trace any decision back to the exact model version and data source. Then add one automatic check to the process used to release models. For example, the system could block a model from being released if there is no record of its test results. Small, simple rules that are actually enforced are better than big plans that people can choose to ignore.

The missing layer worth building

AI in large companies usually does not fail because the models are not powerful enough. It fails in the gaps between different teams and systems. There is often a gap between the team that builds an AI tool and the team that must take responsibility when something goes wrong. There can also be a gap between what company policies say and what the technology actually does. Governance is the part that closes these gaps. It is much easier to design these protections before the first serious problem than to add them afterward.

The insurance company from the opening example eventually solved its problems. Its document search system was changed so people could only access information they were allowed to see. Its logs started recording which model version was used. And project approvals became proper written records instead of informal agreement in meetings. None of these changes was especially exciting. But together, they made it possible for the company to keep using its chatbot.

What does this kind of governance system look like in your company, and who is responsible for it? I would genuinely like to hear where you think this approach might not work.