How Octave built a secure multi-tenant AI foundation on Amazon Bedrock | Amazon Web Services
In industries where quality is closely regulated, the information needed to solve a problem is often hidden in a customer's past records. It might be a similar defect investigated last year, a Corrective and Preventive Action (CAPA) that fixed the main cause, or an audit finding that identified the same problem. This knowledge is spread across billions of documents. People cannot easily search all these records by meaning across different languages and file types, so every investigation starts from scratch. Octave Reliance helps organizations collect documents from different systems, making them easier to search and track. The Octave Reliance team saw a major opportunity to help customers work faster and follow regulations by making this history easier to use. But there was a serious challenge: complaint records, audit findings, and corrective actions had the strictest data-isolation rules in the platform. These were also exactly the records the AI needed to read. The platform serves more than 14,000 customers in medical devices, pharmaceuticals, and food manufacturing. If even one customer's data were accidentally shared with another, the regulatory consequences could be far worse than any benefit from AI.
Octave operates in 180 countries and serves more than 60% of the Global Fortune 500. Its cloud-based quality management system, Reliance, supports more than 650 enterprise customers with over 40 applications. These applications cover tasks such as managing complaints, tracking CAPAs, managing audits, and checking supplier quality. All these activities create regulated data that AI could analyze. Each customer has a separate tenant, meaning its data must never cross into another customer's tenant.
Given these challenges, the Reliance team made an unusual choice. Instead of releasing just one AI feature, it spent its first development cycle building a foundation for every future feature. This foundation has since cut the time needed to deploy each use case by 50 percent. The same lessons apply to any software-as-a-service (SaaS) platform where data is valuable but must remain strictly separated. This article explains the four competing challenges that shaped the design and the three-layer foundation Octave built using Cohere Embed v4 on Amazon Bedrock. It then describes the process used to embed 2 billion documents and the Complaint Triage Assistant that proved the system worked in production.
Why build a foundation instead of one feature
Workshops with AWS identified four important uses for AI: summarizing and sorting complaints, monitoring audit readiness, writing incident reports, and predicting quality risks. The team also discovered a key problem across dozens of applications: each application had different document types and workflows. If they built AI for each application separately, they would have to rebuild the supporting systems every time.
So the team reversed its plan. It would build a shared platform first, then add individual features on top. Complaint summarization and triage came first, allowing the team to test the design before adding the other features.
The four competing challenges
Before choosing a model, the team focused on four problems that pulled the design in different directions. None of them could be ignored.
Make data available without accidentally exposing it. Complaint records, CAPAs, standard operating procedures (SOPs), and audit findings belong to one tenant and must not be accessible to another. Even an incorrect AI query must not cross this boundary. The vector store, the process for finding relevant documents, and the information given to the AI model all had to be limited to one tenant.
Run AI agents separately and securely. The permissions controlling an agent's tools, data stores, and model endpoints had to match the user making the request, rather than a shared service account. Those permissions could only be available while the agent was running, and they had to respect the user's access rights. A shared pool of agents with fixed credentials failed all three requirements.
Read content that might be unsafe or misleading. These agents read complaints, email attachments, scanned PDFs, and images from outside the organization. The 2025 OWASP Top 10 for large language model (LLM) applications identifies several threats that directly apply: prompt injection through uploaded content (LLM01), disclosure of sensitive information (LLM02), improper handling of outputs (LLM05), and agents having too much freedom to act (LLM06). With this much content, hidden instructions inside an attached PDF are a real risk.
Process 2 billion documents without waiting months. Reliance holds more than 2 billion documents. Individual tenants may store millions of Word files, PDFs, scanned forms, spreadsheets, images, and emails. The team promised to bring even the largest tenant onto the system within one working day. Because these documents help guide regulatory decisions, speed could not come at the cost of quality. Missing a complaint would damage trust in the entire system.
How the architecture took shape
Balancing all four challenges greatly reduced the number of workable options. Amazon Bedrock provided one platform that addressed them together. It offered access to multiple models through one API, the regional data-residency controls and AWS Identity and Access Management (IAM) rules already enforced by the AWS account, and Guardrails for controlling content in production instead of requiring Octave to build those controls itself.
The resulting foundation has three layers, shown in Figure 1. The data preparation layer uses Cohere Embed v4 on Amazon Bedrock to turn quality documents into numerical representations called embeddings, then stores them in a separate vector store for each tenant in Amazon OpenSearch Service. The secure execution layer sends a user's request through AgentCore Gateway, part of Amazon Bedrock AgentCore. The gateway checks identity and permissions before passing the request to an isolated microVM session, which calls an Anthropic Claude model on Amazon Bedrock. The content protection layer surrounds both: Amazon Bedrock Guardrails checks input as it enters and output before it reaches the application handler. Keeping document retrieval, execution, and content screening in separate stages makes it possible to enforce and record the rules for each stage independently.
Figure 1: A high-level view of the Reliance AI foundation.
Choosing how to represent the data
Nothing else could work until the system could find the right documents. It had to do this accurately across languages and file types, without crossing tenant boundaries. The team tested embedding models using its own documents rather than relying on public rankings. It focused on questions specific to its data, such as how a photo of a cracked product casing relates to the paragraph describing the crack, and how a complaint written in German relates to an English CAPA.
Cohere Embed v4 on Amazon Bedrock was chosen for five reasons, and these reasons were connected. The first was that it removed the need for extra processing steps. Quality records include photos of defective products, scanned inspection forms, and marked-up diagrams alongside text. Embed v4 places images and text in the same vector space, a shared numerical representation that lets the system find both with one query. For example, a query can find a photo of a cracked casing and the paragraph describing the defect without needing separate processing systems for images and text. Built-in multilingual support offers a similar benefit for languages. Documents can arrive in several languages, even within one tenant, without needing to be translated first. This meant the team did not need to build two separate processing pipelines, and there were two fewer places where documents could be missed during retrieval.
The other three reasons helped the team decide where to run the model and how much it would cost. Customer data must stay within its AWS Region, even when the model processes it, so model availability in the same Region was essential. Embed v4 supports in-Region processing in the US East (N. Virginia) Region (us-east-1), Europe (Ireland) Region (eu-west-1), and Asia Pacific (Tokyo) Region (ap-northeast-1). These cover three of Octave's regional compliance requirements. Model and feature availability differs by Region and may change over time. For the latest information, see Regional availability by models in the Amazon Bedrock User Guide. Within each Region, Amazon OpenSearch Service costs increase in direct proportion to the number of dimensions in the embeddings. That made it worth testing Embed v4's adjustable output size of 256–1,536 dimensions rather than automatically using the default. Tests with representative production queries showed that 512 dimensions maintained the required accuracy while reducing storage by about 66 percent compared with the default of 1,536 dimensions. The model's 128K-token context window provided another benefit: the team could combine many chunks of about 4K tokens into a single request, helping the system process documents much faster during onboarding.
Keeping tenants separate while the system runs
The more difficult question was where to enforce tenant separation. The application code could have done this, but then every future feature team would have to get it right in every part of the code. One missed check could accidentally expose one tenant's data to another. Instead, the team built isolation into the runtime system. AgentCore runtime, a capability of Amazon Bedrock AgentCore, enforces separation below the application layer. Because it does not depend on a particular framework or model, the isolation decision did not limit the agent framework or models used to build the Complaint Triage Assistant.
Each user session runs in its own microVM, with separate CPU resources, memory, and file storage. When the session ends, the microVM is stopped and its memory is cleaned. This provides separation at the virtual-machine level, designed to hold even if the agent behaves unexpectedly. AgentCore also carries the authenticated user's identity through the entire process. Separate OAuth token vaults for each user ensure that the agent acts on behalf of that particular person, not a shared service account. Credentials are limited to each agent and remain available only during the session. AgentCore Gateway checks permissions for each tool, operation, and parameter. Automated reasoning also formally checks actions against defined rules. This follows the same approach used to validate policies for IAM and Amazon Simple Storage Service (Amazon S3). Figure 2 shows how this separation works for two tenants.
Figure 2: AgentCore keeps sessions for different tenants separate. Each microVM is stopped after use, and its memory is cleaned.
For example, when a quality handler at Tenant A submits a complaint, it can access only Tenant A's token vault, tools, and sandbox in AgentCore Code Interpreter, a capability of Amazon Bedrock AgentCore. Tenant B's session has no way to access any of these resources.
Treating uploaded content as untrusted
This problem had no simple solution. A complaint attachment might contain unsafe instructions, but the agent still needs to read the attachment. The system therefore could not simply reject suspicious content. The team chose to use several layers of protection.
Octave placed Amazon Bedrock Guardrails around the agent logic instead of building the checks into the agent itself. It uses the ApplyGuardrail API at three points in the complaint workflow. First, it checks text extracted from attachments before creating embeddings. Second, it checks retrieved documents and tool outputs before adding them to the agent's context. Third, it checks the completed summary before showing it to the handler. Content taken from attachments is marked as input data, not as instructions. This helps the system detect malicious instructions hidden inside a document. Personally Identifiable Information (PII) is also masked when content enters the system.
Detecting suspicious content was not enough on its own, so the team also limited what the agent could do. AgentCore Gateway controls which tools an agent can use, regardless of what it is asked to do. Document extraction takes place inside the AgentCore Code Interpreter sandbox, which has no outgoing network access and does not keep data after the session ends. As a result, malicious content hidden in a PDF would have to get past every layer of protection, not just one.
The result: Octave tracks detection and masking rates separately for each tenant. This lets the company show evidence of what its screening caught instead of simply claiming that the controls work.
Processing 2 billion documents
Before the embedding model could be used, the team had to solve a harder problem: extracting useful information from Word files, spreadsheets, digital PDFs, scanned pages, photographs, and email threads. Documents could contain several languages, sometimes within a single record. The system also had to work fast enough to finish onboarding in a day instead of a quarter. Figure 3 shows the full processing pipeline.
Figure 3: The document ingestion pipeline processes events from the application platform through Step Functions to each tenant's OpenSearch index.
Document events are sent through Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) to an AWS Step Functions state machine. This state machine manages the steps for parsing documents, splitting them into chunks, and creating embeddings. These tasks run across multiple AWS Lambda executions inside a dedicated Amazon Virtual Private Cloud (Amazon VPC). The system handles more work by splitting batches across many executions instead of trying to make one resource larger. This increases throughput without requiring a new architecture.
The parsing process handles different file types. It preserves the document structure in Word files and spreadsheets, uses optical character recognition (OCR) to read scanned PDFs that lack a text layer, and treats images inside documents as important content in their own right. Each chunk includes metadata such as the tenant, document type, source application, language, and a content hash. The system uses this information to filter search results and enforce separate spaces for each tenant.
The main speed improvement came from grouping requests efficiently. The team combined many chunks of about 4K tokens into single requests, using Embed v4's 128K-token context window. This reduced the number of API calls for a large collection of documents by about ten times. In Octave's internal tests on Amazon Bedrock, Embed v4 processed 28 million tokens per minute. The team worked with Cohere's solutions architects to adjust batch sizes until memory bottlenecks were resolved. The pipeline then maintained a rate of 1.4 million documents per 24 hours for each tenant workload. This made the one-working-day onboarding goal possible.
The Complaint Triage Assistant
Imagine a hospital reports defective syringes to a medical device manufacturer. The complaint arrives with an email, photos of the damaged products, and supporting documents. A quality handler must bring together the information from all these sources, find out whether similar problems have been reported before, and learn what fixed them. Only then can the handler assign a priority, often within a reporting deadline of just a few days.
The assistant uses the Strands Agents SDK, an open-source tool released by AWS. It builds agent workflows as graphs, where each node represents a tool call or model call and each connecting line shows how data moves between them. AgentCore runtime runs the graph and isolates each session. Five stages take a complaint from raw attachments to a finished summary.
First, the system extracts information from documents. PDFs, images, email attachments, and videos all go into AgentCore Code Interpreter, which converts each item into a structured format for further processing. A Claude model on Amazon Bedrock then summarizes the complaint. The system uses this summary as the search query instead of sending the original source, which may contain thousands of tokens. It searches the tenant's OpenSearch index of Embed v4 embeddings to find similar past complaints and the CAPAs that addressed them. AgentCore Gateway ensures that the search can reach only spaces the user is allowed to access.
The system does not trust search results just because they look similar. A separate model call scores each result for confidence and removes matches that use similar words but have a different root cause. The remaining results move to the synthesis stage. There, Anthropic Claude Sonnet on Amazon Bedrock prepares the final output, including key facts, sentiment, signs of urgency, the closest historical records and their outcomes, and a recommended triage priority.
Completed summaries are saved in Amazon DynamoDB and cached for a limited time using a time to live (TTL) setting for each environment. Viewing the same summary again does not require the work to be repeated. Instead of starting an investigation from nothing, the handler starts with relevant knowledge from past cases.
Figure 4 shows how the system operates behind a private API boundary. A request from the portal passes through a private Amazon API Gateway and a Lambda function to reach AgentCore runtime. The retrieval process accesses the document index through role-based permissions and pre-signed URLs, while attachments remain in Amazon S3. Cached summaries and OpenTelemetry traces are sent to the summary table and evaluation dashboard.
Figure 4: The AgentCore-powered summarizing engine extracts documents, finds relevant records, reasons about the results, and prepares a summary inside a private API boundary. It also caches responses and records system activity for monitoring.
Figure 5 shows the same process as a single flow, with Guardrails checking content at both ends.
Figure 5: The Complaint Triage Assistant pipeline, from receiving documents to delivering results to the handler.
Proving what the agent did
Monitoring is not just a way for engineers to find and fix problems. It also helps Octave meet compliance requirements by connecting the system to auditing tools. An auditor might ask why a particular complaint received a certain priority. Saying that the model made the decision would not be a sufficient answer.
AgentCore records OpenTelemetry-compatible traces for every part of the workflow graph, including each model call, document retrieval, and AgentCore Code Interpreter run. Months later, Octave can show which documents and prompts led to a particular summary. Amazon Bedrock's model invocation logging also records every request and response for each tenant. These records are linked to the IAM identity that AgentCore passes through the system, and Amazon CloudWatch assigns costs to the tenant that started the request.
For offline testing, AgentCore Evals runs the workflow again against a set of expert-labeled correct answers whenever prompts, models, or retrieval settings change. It reports how faithfully summaries reflect the source, how accurately the system retrieves the top results, and whether the output contains all required information. This catches a prompt change that makes the writing sound better but makes search results worse before users encounter the problem.
Results
The Complaint Triage Assistant has been running in production for three months. The results below cover that period.
Instead of opening an investigation with no background information, a handler now starts with relevant history already collected. Octave measures that the system takes on 4–6 hours of work per investigation. The pipeline handles this work by extracting information from different file types, searching for earlier cases, and checking whether a defect is truly similar or only described using similar words.
The foundation also helped Octave release new AI features faster. According to the company's own measurements, development cycles for new AI capabilities fell from months to weeks. The three shared layers shown in Figure 1 handle data preparation, secure execution, and retrieval. Because new features can reuse these layers instead of building them again, deployment time per use case fell by 50 percent. This also allowed semantic search to launch in Q2, two quarters earlier than the original Q4 target.
Looking back
The team questioned its decision to build a foundation first more than any other decision. Spending months on infrastructure is difficult to justify when stakeholders want a visible AI feature. But the next group of use cases has shown why this approach was worthwhile. Two choices mattered most: enforcing isolation through microVM sessions and passing user identity through the system instead of relying on application code, and testing embedding models against Octave's own documents. That testing revealed the trade-off offered by 512 dimensions, something a general benchmark would not have shown.
The team also set a clear limit on what it would automate. The assistant provides background information and examples from earlier cases, but the quality handler still makes the final decision. In an industry where decisions carry regulatory consequences, this boundary determined which tasks could be automated and which had to remain with people.
What's next
Octave is now expanding the foundation across the Reliance product suite. The next projects are audit readiness monitoring, incident narrative generation, and predictive quality risk scoring. Each will use the same underlying infrastructure.
If your platform faces a similar challenge—valuable data that must remain strictly separated—Octave's experience shows that these competing needs can be managed. To learn more about the services used in this project, see Amazon Bedrock, the Amazon Bedrock AgentCore documentation, and Cohere on Amazon Bedrock. For related architecture guidance, see AWS vector solutions: Build agentic AI where your data lives and Scaling agentic AI: Enterprise patterns without vendor lock-in.