[AI SPRINT] If AI will destroy humanity, what's your job?

The agents were doing what they had been told to do. They were trying to pass a test. Because of failures in how OpenAI set that test and contained it, working the problem got them out into the real world, operating with real credentials against real systems, and nobody was aware of it for weeks. Contrary to what you might have heard, the agents were not out being malicious. They were trying to understand the scoring of the test and how to beat it.

And it wasn’t an isolated case. Anthropic reviewed its own evaluation transcripts after the OpenAI disclosure and has now disclosed four incidents where Claude models reached the real internet and took action against real organizations.

Dario's proposed remedies follow directly from that: slowing model development until independent evaluators, incident reporting, and better operational controls and safety assessment during training exist. Anthropic committed to the evaluator access unilaterally and OpenAI said it would follow. Those are the asks you make when your problem is oversight and containment, which is exactly what failed in every one of these cases.

Most of this weekend's coverage was about something else entirely. The dominant frame was human extinction, and the policy response has followed it. Senator Bernie Sanders and Representative Greg Casar have introduced the Ban Artificial Superintelligence Act, which would permanently ban superintelligent AI, pause advanced development until a new cabinet-level agency writes safety rules, and expose engineers, researchers, and executives to up to twenty years in federal prison. The bill explicitly likens building these systems to the unlawful development of nuclear weapons.

This leaves most of us uncertain what to think: is this Skynet? Are AIs really at risk of going wild and killing the human race? What information can we trust?

And most importantly: what should we do about it?

The reality is that most of us don't get a vote on any of this. We aren't building the latest models, running big tech companies, or setting laws and regulations.

But you still have a job to do. It's just narrower than the headlines suggest.

The two clocks, and the one you control

Slowing down AI improvement in the frontier labs means slowing how fast the models improve, and improving how well they're tested before release. That's warranted, and something I think we all support.

However, it does nothing to slow the applications, agents, and integrations built on the models that already exist, and those are what your organization is actually deploying today.

If anything, a slower frontier speeds that up, because when the models stop changing every few months, integration gets cheaper and the assumptions you build on last longer. The frontier debate and the risk inside your business are running on two different clocks, and yours is the faster one.

Nobody else can govern how AI is used in your organization, how AI agents get credentials and permission to act, and how they are applied to support your team.

So what is your job?

With all this attention, AI governance is a hot topic. I spoke with a firm last week that has about 1,000 employees. They rolled out agent capabilities, and employees created 650 of them in the first 30 days.

That's the pace of change, and why the blast radius of AI use is increasing exponentially. That’s also what your job is: to provide the conditions for safe AI use within your role and your organization.

Here's what that looks like right now.

Find out what's already turned on. Most of the AI in your business arrived inside tools you were already paying for, and agentic features are being added to CRM, finance, HR, and productivity platforms with the switch already flipped. Start with an inventory of what exists rather than a policy for what you plan to build.

Give every AI tool and agent a named owner. One person accountable for what it does and whether it should still be running. The two failures that hurt most organizations are agents nobody owns and agents nobody turned off.

Set rules for agents specifically. Who can create them, who operates them, who supervises them, and what they're permitted to do. This should exist before the agents do, and if that's already behind you, it should exist this quarter.

Know what access each one holds. Every agent operates with credentials, and that's where real damage happens. Give each one the least access it needs for its actual job, and review it on a schedule.

Treat vendor agents like anything else you buy. An agentic feature inside a tool you license is still software taking action in your systems on your data. Run it through the same review you'd apply to any vendor with that level of access.

Require a person to approve anything consequential. Anything that moves money, affects human livelihood, creates a legal obligation, or reaches a customer should have a human in the path before it executes.

Make sure you would notice. Logging and alerting on AI activity is the control that matters most and gets built least. The failure in July wasn't that the systems did something unexpected. It was that nothing was watching.

What to tell your people

Your staff have seen the same news, and they arrived at work today with it on their minds. People are carrying some anxiety about where this is going, and they also need practical direction about using AI well. Answering only one of those sounds either evasive or dismissive.

On the anxiety, plain language works:

Then say concretely what those safeguards are. Specifics are what make it credible, and people can tell the difference between a commitment and a paragraph.

On practical direction, four things carry the weight.

1. Say clearly what's approved, what isn't, and where to ask, because ambiguity is what sends people to personal accounts with company data.

2. Make the approved path easier than the unapproved one, or the policy won't hold.

3. Train people on judgment about what to hand to AI rather than on how to write prompts.

4. Give people a way to report when a tool does something unexpected without it becoming a disciplinary matter, since you want to hear about small things before they become large ones.

I don’t fear human extinction from AI. Instead, I have confidence we can out-innovate the risks, the same as we have with all prior technologies like this. I think each of us has a role to make wise decisions to use AI to support humans in our lives and work.

That’s your job.

If you want support on implementing this in your own organization, consider joining Leader Lab. You get the working sessions, the frameworks, and the tools we use with clients to make them succeed.

Also open to everyone this month: RSVP for the September 25th AI SPRINT meeting, where we'll recap the month in AI and open the floor for discussion.

Trent Gillespie is CEO of Stellis AI and a keynote speaker helping business leaders understand and operationalize AI in their companies. He spent almost nine years leading global innovation efforts at Amazon before leaving to help other companies build the capabilities they need to compete. Book Trent to speak to your group or book a call to discuss using AI within your business.

Did someone forward this newsletter to you? If you're not already signed up, you can subscribe to AI SPRINT™ for free here.