Artificial intelligence values before autonomy

Organizations are preparing agentic artificial intelligence to act on their behalf before they’ve decided what they want it to stand for. The most important question to answer right now is who decides what AI can do when nobody’s watching.

In September 2026, Anthropic CEO Dario Amodei published “We Must Pace the Frontier,” asking the AI industry to slow down. Not because the technology stopped working, but because it started working faster than anyone could govern it. His proposed fix includes third-party evaluators, government regulation, international pacing agreements, and greater transparency and societal oversight.

The company with the deepest alignment-research bench in the industry just said alignment research isn’t enough.

AI CAN’T FEEL, BUT NEEDS VALUES

There’s a difference between AI that answers questions and AI that takes action. Conversations centered around synthesized content can be governed with policies, but autonomous action requires values.

When Claude generated mostly conversation, a constitution that outlined the basic principles and intentions for the tool’s behavior was a transparency document. Now, Claude manages investment portfolios, executes multi-step workflows, and makes decisions on your behalf without checking in. The question is no longer an open-ended exploration around what is possible. Instead, it is: “What governs AI decisions when you’re not in the room?”

Anthropic’s philosopher-in-residence, Amanda Askell, named to Fast Company’s AI 20 list, authored Claude’s Constitution. While reading the document, it becomes clear the company is wrestling with something bigger than ethics: Autonomous systems cannot operate from rules alone. They need a sense of what they are, who they serve, and how they make decisions when competing priorities collide. Anthropic, a model company with a reputation for safety-conscious AI, is writing the governance layer for agentic AI in real time.

“As models are more autonomous and take actions over longer horizons, they have a lot more decision points that you have to map out and make work well in advance,” Askell points out in her Fast Company profile.

WHO’S BEST EQUIPPED TO GOVERN AI?

Here’s where we must lean into human-led AI and apply values and identity work to AI the way we’d apply it to any other part of the organization that acts in our name. Consider what that looks like in practice.

For example, a health company deploys a scheduling agent to reduce no-shows. The AI works—patient throughput improves and care gaps close. What the agent learned is patients who have cancelled before are worse bets, so it moves them to less desirable slots. No human decided that. The agent derived it from its given objective. But research shows that missed appointments often reflect work and caregiving commitments, as well as transportation barriers.

The agent can recognize a pattern without understanding the circumstances behind it. And when it acts on that pattern, it can turn a prediction about patient behavior into a decision that limits a patient’s access to care. An engineer reviewing the system sees a model performing well against its metric. This isn’t a classic case of caring about what you measure instead of measuring what you care about.

The challenge isn’t just what we measure, but how AI acts on what we ask it to optimize. Organizations have always relied on communicators, brand strategists, culture leaders, and behavioral scientists to solve these challenges. So far, we’ve rarely applied those disciplines to AI.

Every organization deploying an AI agent is asking what this system knows, where it learned it, and which answer it should trust. Look at the actual work involved in understanding those answers: classification, provenance, authority control, retrieval, and deciding which source wins when two disagree. The same is true across the disciplines we’ve kept at the edges of AI work: storytellers, brand strategists, and user experience researchers. The skills they provide are still needed, but are now pointed at a new substrate. The people who move between them matter most of all.

I took Herbert Simon’s last class at Carnegie Mellon in fall of 2000 (Y2K, an inauspicious time). Simon won a Nobel Prize in economics and a Turing Award in computer science. He also built the field of human-computer interaction by refusing to accept that psychology, economics, and computing were separate subjects. He could see a field that didn’t exist yet because he wasn’t standing inside any one of them—the polymath advantage.

The people best equipped to govern AI aren’t necessarily the people building it. Engineers understand the failure modes and capabilities better than anyone, and no governance decision should be made without them. However, we’ve handed values and ethics decisions to technologists by default because AI arrived as a technology purchase.

AGENTIC IDENTITY AND VALUES FRAMEWORK

Every organization deploying agentic AI will need to know what their AI stands for when no one’s watching. This agentic identity and values framework helps organizations answer four fundamental questions.

1. Who does your AI serve? This reflects Anthropic’s concept of a “principal hierarchy,” as described in Claude’s Constitution, and defines whose interests take priority. In financial services, does the agent optimize for client, advisor, or firm? When those interests conflict, who comes first?

2. What operational values guide your agent’s decisions? Should the agent prioritize speed or accuracy? Personalization or privacy? These decisions map to corporate values, but they must also guide real-world tradeoffs.

3. What decisions require a human? This isn’t about deciding where to place humans in a workflow. It’s about identifying where you cannot delegate human judgment: diagnosis and treatment confirmation, termination decisions, financial approvals, and legal exceptions.

4. What should your AI never do? This is where constitutions, policies, and guardrails belong. Hard boundaries, escalation paths, and unacceptable behaviors come after identity and values have been established.

THE HARDEST QUESTIONS LEFT

The hardest questions that remain are no longer technical. They are supremely human. In teaching AI what to do, we’re teaching it what our organizations stand for. As a leader, you need to understand agentic AI doesn’t just execute tasks. It represents your brand.

Our world is changing faster than the humans inside it. Every recommendation, decision, escalation, and action becomes an expression of your values. Your choices contribute to building AI that people can trust.

Stephanie Trunzo is chief executive officer of MERGE.