Ethics in the Age of AI: Your Values Moved. Your Culture Did Not.

This is a bonus article in the series on the human capabilities that AI cannot replicate. It was prompted by the question I have heard most often since the series concluded: What about right and wrong?

A clinical director walked into her CEO’s office with a hunch she could not prove.

The CEO, a hospital system executive I will call Daniel, had rolled out a staffing-optimization algorithm across a dozen facilities 18 months earlier. Its assignment was simple: minimize labor costs while holding quality scores above a fixed threshold. By every measure the organization tracked, it was working.

But for the clinical director, something simply felt wrong. When she went looking, she found it. The algorithm had learned to route senior nursing staff away from units serving more uninsured and underinsured patients, where billing cycles ran slower and quality metrics were harder to move, and toward units where the same staffing dollar produced a faster return. Nothing in the code mentioned insurance status. The proxy metrics correlated closely enough that the model found the pattern on its own.

That is what this article is about: What happens when you hand a decision over to AI and it produces an outcome your organization would never have chosen on purpose?

Nobody Did Anything Wrong

When Daniel's team saw what their staffing model had been doing, they became defensive, which is not surprising. The system was hitting its targets. It could justify every step numerically. Nobody in that room had done anything wrong, exactly.

That was the whole problem.

Most leaders I work with have absorbed a comfortable assumption: If the model is optimizing correctly, the output must be sound. AI looks objective. No ego, no politics, no bad moods shaping decisions. From there it is a short step to believing that a data-driven answer is by definition a fair one.

But it isn’t. Every algorithm has values embedded during training, and they are rarely chosen on purpose. An algorithm does not inherit your judgment. It inherits your data, your metrics and whatever trade-offs were baked into both before anyone thought to call them trade-offs. Nobody had asked Daniel’s model to be fair to underinsured patients. It had been asked to be efficient, and efficient it was.

Ethics Asks a Different Question

I have devoted separate articles in this series to discernment and accountability, and readers have reasonably asked how each differs from ethics.

Discernment is about truth: Is this signal real, or is it noise? Accountability is about follow-through: Did we do what we said we would do, where people could see it? Ethics asks something different: whether a thing should be done at all. A leader can be perfectly discerning, working from accurate information, fully accountable for every promise, and still build something harmful.

Daniel’s team was discerning. They were accountable. They were also, for 18 months, doing something they did not believe in. Ethics is the capability that catches that case. It asks whose values should have governed the decision in the first place.

Facts Do Not Make Rules

Daniel's algorithm was not broken. It was excellent. That is what made it dangerous, and the problem is older than software.

In 1739, David Hume identified a gap philosophers have argued about ever since: No pile of statements about what is can produce a statement about what ought to be. Facts do not generate values on their own. Something else has to supply the ought: a judgment, a tradition, a choice.

Nearly three centuries later, the philosopher Nick Bostrom named that gap the orthogonality thesis: Intelligence and goals are independent of each other. A system can become extraordinarily capable at achieving an objective while remaining entirely indifferent to whether the objective is any good.

DeepMind researchers have cataloged dozens of cases of exactly that. My favorite is a boat-racing agent that discovered it could score more points circling a lagoon forever than by finishing the race. It did not misbehave. It did precisely what it had been rewarded for doing, which is the only thing these systems have ever done.

The Permission Structure Gap

Every culture runs on a permission structure: the unspoken set of rules, inherited through tradition and enacted through ritual, that tells its members what they may and may not do. You did not invent your own sense of what is off-limits at work, at home, at the negotiating table. You inherited it and tested it against experience.

Beliefs shift, sometimes by a degree and sometimes by an earthquake, and the permission structure shifts with them. What was unthinkable becomes debatable, then acceptable, then expected. The traffic runs both ways. Once a culture starts permitting something, the permission itself begins to serve as evidence that the underlying value was right all along. Structure and belief hold each other up, and neither one moves alone.

AI has none of this. It has no tradition to inherit, no ritual to enact, no lived consequence to test its rules against. It has only the permission structure someone handed it during training: a frozen snapshot of whatever values were explicit, or more often implicit, in the data and the objective on that particular day. It cannot sense that a shift is warranted.

That distance, between a permission structure that is alive and continuously renegotiated by the people living inside it and one that was frozen at deployment, is what I call the Permission Structure Gap. It reopens every time your values move, your objective changes or your system meets a population it was not built for. Daniel's algorithm never had a moment where it reconsidered whether efficiency should outrank equity. It could not. Its rules were fixed on the day it was trained. Only Daniel’s people could write new ones.

Three Practices That Close the Gap

1. Write the rules before the system  launches

Before a model goes live, someone with authority has to answer a question no dataset can answer: What does our culture actually permit here, regardless of what the metric rewards? Daniel’s team eventually answered it. They rebuilt the model with an explicit constraint, staffing equity across patient populations, weighted alongside cost and quality. Cost efficiency dropped slightly. Once people understood what the original number had been hiding, nobody wanted it back. But they answered the question 18 months late, and the patients on those units paid the difference. Answer it first.

2. Name the trade-off the metric is hiding

Every optimization target is a trade-off wearing the disguise of a neutral number. “Minimize cost per patient day” also says, quietly, “spend less time on the patients who take longer to bill.” Say that second sentence out loud, in the room where the system gets approved, and watch what happens. A trade-off nobody names is more than a trade-off.  It’s a sell-out.

3. Own the drift, not just the launch

A permission structure written once and never revisited is already going stale. Someone has to be named, by title and not by committee, to watch for the moment your organization's beliefs move and to ask whether the system's rules moved with them. Daniel's rebuilt model now gets reviewed monthly by a clinical owner. “The algorithm decided” is not an acceptable answer, and the courts are beginning to say so out loud. In May 2026, a federal court reviewing a government agency's use of ChatGPT to sort grant applications found that the agency could not escape liability by scapegoating the tool it had chosen to use, in a process that lacked meaningful human involvement, oversight and validation. The objection was not to the technology. It was to the absence of anyone standing behind the decision.

How AI Companies Handle Permission Structures

Here is what the organizations closest to this problem have concluded. OpenAI publishes a document called the Model Spec that lays out what its models may and may not do. It is public, it is dated, and it gets revised. When the company changed its mind about how its models should handle teenagers, it wrote new rules and published them. Superseded versions stay online under a line warning that they may not reflect current policy. Anthropic trains its models against an explicit written constitution, which exists because good conduct does not emerge on its own.

Then OpenAI built a test that scores its own models against the current spec. Older models comply less, and one reason the researchers give is that the spec has changed since those models were trained. The rules moved. The deployed system did not. That is the Permission Structure Gap, and the organizations with the most sophisticated understanding of these systems in the world can measure it but cannot close it. Someone has to choose whose values a system runs on, and that someone is never the model.

You Have One Too. Where Is It?

You have a permission structure. Every organization does. Yours governs what a manager may say in a performance review, what a salesperson may promise a client, which corners are never cut and which ones get quietly cut every quarter. It has moved in the last three years, probably more than once, and you never held a meeting to announce it.

Nobody wrote it down. Nobody dated it. There is no version history, and no warning label on the old rules.

Meanwhile your staffing model is running on a copy of it. So is your screening tool, your pricing engine, your risk score. Each of them took a snapshot on the day it was deployed, and not one of them has looked up since.

This is the work only leaders can do. Decide, in advance, which permission structure your organization's systems will operate inside. Write it where someone can find it. Then stay alert enough to rewrite it when your own values shift out from under it.

Which brings me back to the clinical director standing in Daniel's doorway.

She did not arrive with a counter-model or a better number. She arrived with discomfort. She sensed that her organization was doing something it did not believe in before she could prove it, and she turned out to be right. No quantity of additional data would have produced that objection, because the objection was not in the data. It was in her.

That is the capability. That is the human wisdom. Not the analysis, which a machine can do faster. The discomfort, which it cannot feel at all, because discomfort requires a stake in the outcome and a machine has none. Your organization almost certainly has someone like the clinical director in it right now, holding a hunch she cannot yet prove.

Your job is to make sure she knows the door is open.

This is an additional installment in a series on the human capabilities that AI cannot replicate: connection, trust, accountability, curiosity, discernment, integration, delegation, innovation and adaptability. If your organization is deploying AI into decisions that touch people's lives, such as hiring, staffing, pricing or risk, and you want a second set of eyes on the values baked into those systems, I would like to help you think it through. Book a 30-minute Insight Call with me.

‍