The question of who answers for an AI system is moving from the company that built it to the company that uses it. New York City is making the big AI labs testify under oath today while drafting rules that would require a human shutoff switch and independent testing before AI can be sold in the city. A Nvidia-backed startup is about to release an AI model whose inner workings are open to everyone, which puts both the freedom and the responsibility in the customer's hands. Security vendors are racing to attach a named human owner to every AI agent (a program that can carry out tasks on its own), and a peer-reviewed audit shows that regulatory clearance for medical AI says very little about patient outcomes. Whoever uses the system will be the one asked to prove it works and to shut it down.
Council Speaker Julie Menin has called all 51 members of the New York City Council together for a hearing today, Monday, October 5, which the Council called the first public testimony under oath from major AI companies about the dangers of AI technology.
Meta, OpenAI, Google, and Anthropic agreed to appear. The Council said that apart from Meta, the companies confirmed only after it explicitly threatened subpoenas (legal orders to appear). Google and Anthropic had declined by the September 25 deadline, and Anthropic confirmed hours before its subpoena was due. SpaceXAI did not respond and was subpoenaed on September 28, and the Council can ask a New York State court to enforce the order if the company does not comply. New York City Council Menin said, "Given the high stakes, these firms owe it to the public to come before the Council, answer our questions, and provide input on our proposed legislation under oath." The Next Web The bills on the table are proposals, and the outcome of the hearing was not known at publication. One Menin bill would bar selling or using an AI system in the city without independent testing, require a switch that lets a person shut down each system, and impose a $25,000 penalty per violation. Another would give whistleblowers a share of fines collected from AI companies, which the Council calls a first in the nation. A third would let people harmed by AI tools sue when the harm was foreseeable and the company lacked reasonable safeguards, and a fourth would require city contractors to report AI safety incidents within 24 hours. The Next Web The Council's own announcement lists the whistleblower program, the right to sue, and independent third-party testing among the measures under review. New York City Council
A city council is doing what the federal government has declined to do this week, and its bills are written for the companies that use AI, not only for the labs that build it. Independent testing, a shutoff switch, and a 24-hour incident report each fall on whoever puts the system to work. If your company sells to New York City agencies, employs New Yorkers, or serves them, check now whether you could produce an independent test report, show a human shutdown for each AI system in use, and file an incident report within a day. Most organizations cannot do all three today.
Axios reports that Nvidia-backed Reflection AI is preparing to release its first "open-weight" model soon, meaning one that anyone can download, use, and customize, with no release date given.
Sources expect other Western open-weight models this month. The model is expected to trail the most advanced U.S. systems at first while competing with the top Chinese open-weight models, with enough capability to help companies build their own low-cost AI systems. Axios says combining a lower-tier open model with high-quality company data can, in some situations, match the most expensive leading AI systems. Axios Reflection's stated goal is an "AI factory" that lets institutions assemble their own local AI systems from their own data, its models, and their own secure computing, and Axios says hedge funds and trading firms are already eager. Reflection recently signed deals with Nebius and SpaceX to rent Nvidia servers, without disclosed terms, and announced a "sovereign AI factory" partnership (a locally controlled AI setup) with Shinsegae Group in South Korea. The most powerful open models are almost all made in China, Axios notes, and Western banks and the Pentagon are balking at them over security. Open models can be harder to monitor and regulate than the three leading closed systems, though supporters say their transparency is also a security advantage, and open models make up only a small share of business use today. Axios
A credible Western open model gives you a second source of AI and a way to keep sensitive data inside your own walls. It also shifts responsibility. With a closed model, the vendor monitors, repairs, and answers to regulators. With an open one, your team takes over that work, which runs directly into the testing and shutdown expectations in New York's draft rules. Run a head-to-head test when the model ships: choose one task built on your private data, compare accuracy and total cost against your current AI service, and add in the staff and monitoring you would need.
Open models let a company take back control of its data and its AI agents, and Reflection's pitch is to sell that control back as infrastructure. Responsibility moves entirely to the company using the model, which avoids depending on a single AI vendor but creates hidden rule debt if the company's governance does not travel with the model. The sturdy position is to keep your policies, tests, and audit records in your own tracked systems so any model, open or closed, can be swapped in or out.
Ahead of SailPoint's Navigate conference, October 5 through 8 in Austin, SiliconANGLE frames AI agents as turning identity management (deciding who or what can access which systems) into a practical execution problem.
Krista Case, an analyst at theCUBE Research, said, "Enterprises need to know who or what is acting, whose authority it carries, what it can reach and whether that authority should still apply as the task changes." She added that agents "can be created and deployed faster than traditional access processes can discover, assign ownership and govern them," and cautioned against claims that a single platform will become the control center for AI agents, since the work spans identity, software infrastructure, applications, data, and monitoring. SiliconANGLE SailPoint CEO Mark McClain said, "The proliferation of AI agents is creating a new class of non-human identities, and each one represents a new attack surface." Analyst Zeus Kerravala pointed to a SailPoint trial at an unnamed Fortune 500 company that reportedly discovered more than 10,000 previously unknown AI agents, a claim from the vendor that has not been independently verified. SiliconANGLE discloses that theCUBE is a paid media partner for the event. SiliconANGLE
The number in the vendor's anecdote matters less than the pattern it describes: AI agents pile up faster than anyone's inventory, and the owner is the missing piece of information. Do the inventory yourself before buying anything. List every service account, access key, and automation tool that can act without a person clicking, record a named owner and a purpose for each, and set an expiration date. Then decide who removes an agent's access when its task ends. Case's warning is worth heeding: expect every vendor to claim to be the control center, and keep your own registry as the official record.
A review published August 19 in the journal PLOS Digital Health examined all 1,357 AI and machine-learning medical devices that the FDA had cleared or approved through December 5, 2025.
Only 34 (2.5%) were linked to registered forward-looking clinical trials, 12 (0.9%) had peer-reviewed publications, and only three (0.2%) evaluated patient-centered outcomes such as deaths, complications, or hospital readmissions. Radiology tools made up 1,059 of the devices (78%), and 62% of studies used observational designs with small, similar groups of patients. The authors, from the University of Toronto, MIT, Johns Hopkins, and other institutions, conclude that regulatory approval has outpaced clinical testing, creating "an ecosystem where innovation advances without accountability." PLOS Digital Health They propose staged evidence standards, including studies of at least 500 patients around the time of clearance and multi-site outcome trials of at least 2,000 patients afterward, advance registration of trials for higher-risk devices, and insurance payment tied to proven clinical benefit instead of regulatory status. Their closing line: "AI tools must be life-tested before they can be called life-saving." PLOS Digital Health
This is a healthcare finding with a general lesson: a certification tells you a product met a regulator's entry bar, not that it improves your results. The same gap exists anywhere a vendor cites a compliance badge as proof of performance. Make evidence of results a condition of buying. Ask for results measured in a setting like yours, agree on the measure before the pilot starts, and keep the contract right to walk away if the target is missed. That is also the habit New York's independent testing proposal would force on every company that uses AI.
Palmer's "The AI Slowdown Is a Sales Pitch" (September 27) argues that the major labs' calls to pace the frontier may serve their commercial interests as well as reflect real safety concerns, since certification and compliance rules could favor a few well-funded labs and raise costs for competitors.
Shelly Palmer He suggests judging any rule by whether obligations scale with demonstrated capability and risk, whether open and closed models are treated equally, and whether companies without frontier-lab budgets can comply, and he describes open models as a check on the leading labs' potential pricing advantage. Those tests apply directly to today's New York bills and to Reflection's arrival.
Responsibility for AI outcomes is landing on the company that uses the system.
Appoint one executive who owns the answer to three questions for every AI system in use: who tested it, who can shut it down, and who gets the call if it causes harm. Decide this quarter whether open models belong in your strategy, since they reduce dependence on a vendor and increase what your own organization must manage.
Testing, monitoring, and identity checks are becoming the products that sit between AI models and the businesses that run them.
Cities, states, and courts are setting requirements faster than Washington, and the requirements are written for the companies that use AI. Expect rising demand for independent testers, AI-agent registry tools, and incident-reporting services, and expect a split between closed leading systems for the hardest problems and open Western models for work built on private data.
Customers will read compliance badges more skeptically after the FDA audit and the New York hearing.
Support every performance claim in your marketing with a measured result and a named method, and be ready to show independent testing if a buyer or regulator asks. Prepare a plain statement of how a person can shut down your AI features, since that question is entering public hearings.
Build three capabilities that no vendor will provide for you: a registry of AI agents with named owners and expiration dates, a documented human shutdown procedure for each AI system in use, and an incident process that can file a report within 24 hours.
Staff the monitoring work that comes with any open model before you put one to use.
Open models change the cost structure from usage fees toward fixed infrastructure and staff, so model the full cost of ownership, including monitoring, testing, and security, before comparing it with an AI service you rent.
Treat per-violation penalties such as the proposed $25,000 as a floor, and plan for legal exposure created by the right to sue.
List every account and key that can act without a human, assign owners, and set expiration dates.
Build a testing setup that works with any AI model so you can check an open or closed model against your own data and swap it without rewriting your governance. Add a tested shutoff switch to each AI system in use, since it is the most concrete requirement in the New York proposals.
Ask management for the list of AI systems in use with the date of independent testing, the named human owner, and the tested shutdown procedure for each.
A hearing that compels sworn testimony shows how quickly an oversight question can become a legal one, and the board's record should show these questions were asked before anyone was subpoenaed.
The pattern across today's stories is that responsibility is being pushed down to the company that uses AI: cities draft duties for those who put AI to work, open models hand ownership to the customer, security tools assign a human to every agent, and an audit of medical AI shows that clearance does not stand in for proof. The assumption this breaks is that a reputable vendor or a regulatory badge carries the accountability for you. The choice this forces is whether your company will build its own testing, shutdown, and ownership records now, or be asked to produce them on someone else's timeline.
If a regulator, a customer, or a hearing room asked you tomorrow who tested your most important AI system and who could shut it off, would you have a name?
**Build the harness. Price the outcomes. Redesign the org.**