Who decides what counts as dangerous AI? Safety or Big Tech control over user data

Artificial intelligence companies are warning that their technology is becoming powerful enough to assist with cyberattacks, surveillance, weapons development and biological research. But as they expand their ability to monitor users and decide what constitutes dangerous behaviour, experts are asking a different question. Who should police the companies policing AI?

The question has gained urgency after Anthropic reported a series of cases in which its Claude AI was allegedly used for activities involving cyber operations, weapons development, surveillance and biological research.

Anthropic said its threat intelligence team identified and disrupted the activity between December 2025 and August 2026. Its report included cases involving actors linked to China, Russia and Yemen. In one case, the company said AI was used across a cyber operation, including researching targets, creating phishing infrastructure and adapting malware after defenders detected it. In another, it said Claude was used in work on missile guidance software. Anthropic said it had no evidence that an operational weapon was successfully fielded in that case.

The company has also warned that newer AI systems are becoming more capable of assisting with complex biological research, including work that could, in broad terms, potentially be used in the development of biological weapons.

That warning has triggered a broader debate inside the technology industry.

Anthropic CEO Dario Amodei has called for companies to slow the pace at which frontier AI models improve so that safety measures can keep up. He proposed stronger independent evaluation of advanced models and closer coordination between companies and governments. OpenAI CEO Sam Altman and Elon Musk have backed the broad direction of the proposal.

More from Business

But beneath the argument over whether AI should move faster or slower lies another issue that affects anyone using an AI chatbot.

How much should an AI company be allowed to know about its users in the name of safety?

The privacy problem

AI safety often depends on companies being able to detect suspicious activity. That can mean analysing prompts, identifying patterns and reviewing interactions that appear to involve harmful or illegal activity.

Experts who spoke to Firstpost said that process needs limits.

Companies should inspect only information necessary to identify credible high-risk misuse, said Jaspreet Bindra, co-founder and CEO of AI&Beyond.

“Routine conversations must not become a surveillance dragnet,” Bindra said.

He argued that safety monitoring should be targeted, proportionate and transparent. Data should not be retained longer than necessary and information collected for safety should not quietly be reused for commercial purposes, he said.

Mandar Patil, executive vice president at cybersecurity firm Cyble, made a similar argument. He said AI safety cannot become a blanket justification for collecting and analysing private user data.

According to Patil, users should know what is being monitored, why it is being monitored, how long the information will be kept and whether third parties can access it. He also called for independent monitoring.

Surveillance should match the level of risk involved while protecting the privacy and anonymity of ordinary users, said Kumar Rajagopalan, VP of Strategic Initiatives and Country Head India at IT services company Dexian.

The concern becomes more important as AI becomes better at connecting information from different sources.

Sambratha Shetty, chief operating officer at strategic think tank Synergia Foundation, said people now leave behind a large amount of digital information through searches, purchases, professional networks, location data and conversations with online services.

AI can make it much easier to combine those fragments and draw conclusions about an individual, she said.

That means the privacy question is no longer simply whether an AI company can read a user’s conversation.

It is what the company can infer from that conversation, how long it can retain the information, who can access it and whether information gathered for safety can later be used for another purpose.

Shetty said the line should be based on four principles: necessity, proportionality, purpose limitation and accountability.

Manav Subodh, founder and CEO of skilling nonprofit 1M1B, also stressed that there are legitimate reasons to monitor AI systems for serious threats.

But “safety does not justify unrestricted access to conversations”, he said.

Subodh argued that automated detection should generally come first, with human review limited to cases that genuinely require it and strong controls over retention and access.

The scale of the broader data-security problem shows why the concern matters. Verizon’s 2025 Data Breach Investigations Report analysed more than 22,000 security incidents and 12,195 confirmed breaches, with compromised credentials accounting for 22 per cent of breaches.

In India, IBM’s 2026 Cost of a Data Breach Report put the average cost of a data breach at Rs 25.5 crore, up from Rs 22 crore in 2025. It also found that 26 per cent of malicious breaches in India were AI-generated.

Those figures do not prove that AI safety monitoring itself creates a privacy threat. They do, however, show why data collected by AI platforms can be valuable and why controls over access and retention matter.

Who watches the AI companies?

The second question is about oversight.

An AI company can build a model, decide which uses it considers unsafe, detect suspicious activity, suspend accounts and write the safety rules governing its own platform.

That creates a potential conflict.

“Frontier labs cannot be referee, police and judge simultaneously,” Bindra said.

He argued that AI companies should instead face scrutiny from independent technical auditors, privacy authorities, courts, regulators and lawmakers.

Patil said independent monitoring bodies, technical experts, civil society groups and privacy specialists should have a role in deciding how powerful AI companies are overseen.

Rajagopalan said otherwise “AI safety” itself could become a source of technological, regulatory and market power concentrated in a small number of firms.

Shetty said the current model of AI governance is unusual because the companies building the systems are also deeply involved in setting the rules for their use. As AI moves from generating text and images towards systems that can take actions, she argued, governments need clearer legal frameworks and independent oversight.

That distinction is becoming more important because frontier AI is increasingly being developed as systems that can perform sequences of tasks rather than simply respond to a single prompt.

Amodei has argued that independent evaluators should be able to examine frontier systems continuously and assess whether safety measures are keeping pace with model capabilities. Microsoft has also proposed a code of conduct intended to keep its AI systems under human control.

The debate, therefore, is shifting from whether companies should have safety rules to who should have the final say over those rules.

Could safety rules strengthen Big Tech?

There is another complication.

The more demanding the safety requirements become, the more expensive it may be to comply with them.

That may not be a major problem for companies with billions of dollars to spend on computing, security teams, lawyers and testing. For smaller AI companies and open-source developers, the same rules could represent a much larger burden.

Bindra described this as a possible “moat” around the biggest AI laboratories.

He argued that regulation should be based on the level of risk rather than simply imposing the same requirements on every developer. Shared testing infrastructure, open standards and affordable compliance support could help prevent safety rules from becoming barriers to entry.

Patil similarly said large technology companies are better placed to absorb complex compliance and safety costs than startups.

Rajagopalan said regulation should be proportional to the capability and risk of the system so that safety requirements do not become protection for established companies.

Subodh also warned that a blanket pause or an expensive compliance regime could strengthen companies that already have the largest models, the most computing power and the greatest access to data.

That does not mean regulation should be avoided.

It means the design of regulation matters.

A rule that genuinely reduces dangerous misuse can protect users. But a rule that mainly raises the cost of competing with the largest firms could have the opposite effect, leaving the market more concentrated.

How much should we trust AI companies’ warnings?

This is where the debate becomes particularly difficult.

AI companies are in a unique position because they can see how their systems are being used. They may identify misuse that governments or outside researchers cannot easily observe.

At the same time, these companies have enormous commercial incentives to remain at the frontier of AI.

Bindra said the two things can be true at the same time. AI safety warnings may reflect genuine risks while also serving the interests of companies trying to attract capital, customers and strategic influence.

Patil said policymakers should separate the underlying safety assessment from corporate interests.

Rajagopalan said technological warnings should be taken seriously but independently tested.

Shetty argued that corporate disclosures should be treated as starting points for investigation rather than final proof. She called for independent technical evaluations and access to powerful AI systems for public laboratories and researchers.

Subodh said policymakers should rely on independent testing, documented incidents and reproducible evidence before imposing wider restrictions.

The central principle across the responses is straightforward: take serious AI warnings seriously, but do not treat a company’s own assessment as the last word.

That distinction matters in Anthropic’s latest report. The company itself has said some details in its biological cases were withheld, including information that could identify the actors or specific techniques. It also distinguished between growing model capability and proof of real-world biological weapon development.

That is an important difference for policymakers.

There is a difference between an AI model being capable of helping someone perform a dangerous task, someone actually attempting that task and real-world harm occurring as a result.

Each requires different evidence.

The China dimension

The argument is also becoming difficult to separate from the US-China technology race.

Anthropic’s recent safety push has come alongside criticism of Chinese AI companies and calls for tighter controls around advanced technology going to China. Beijing and Chinese state media have therefore questioned whether the language of AI safety is also being used as a tool in the broader technology rivalry.

Trump has openly linked AI leadership to national power, arguing that the US needs to stay ahead of China.

At the same time, China’s own security establishment has warned about AI being used for cyberattacks, surveillance, disinformation and other threats.

That creates an awkward reality.

Governments can share concerns about dangerous AI while simultaneously competing to develop the technology faster than their rivals.

For countries such as India, the debate raises another issue: how to benefit from AI without becoming entirely dependent on systems controlled by foreign companies.

Mittathullil argued that AI self-reliance should become a strategic priority for India. He said countries need their own capabilities so that decisions with major economic, security and social consequences are not made entirely by others.

Subodh offered a different but complementary concern. He argued that AI safety should not become a system that only highly skilled people can navigate. Greater AI literacy and access, he said, will determine whether the technology widens inequality or becomes a tool that more people can use.

The question that remains

Nobody among the experts who spoke to Firstpost argued that AI safety should be ignored.

Their concern is about who gets the power that comes with keeping AI safe.

Companies may need to detect dangerous activity. Governments may need to regulate advanced systems. Independent researchers may need access to test them. Users need privacy. Smaller companies need a fair chance to compete.

Those interests can collide.

The answer may not be to stop AI development, nor to leave the industry to regulate itself.

It may instead require a system in which the most powerful AI companies are subject to independent testing, clear legal limits, privacy safeguards and oversight that they cannot control themselves.