The hidden vulnerability of using large language models for propaganda

When a covert Russian influence operation was recently dismantled, the focus fell on the tools used—specifically, ChatGPT.

But a closer look at the operation reveals a more unsettling truth: the real vulnerability isn't the AI itself, but the human tendency to trust anything that looks like a credible institution.

The recent takedown of a network tied to the International Burke Institute (IBI) shows that the most dangerous use of large language models is not to generate vast quantities of fake news, but to create a thin veneer of legitimacy for a much older tactic: the fake think tank.

OpenAI banned a cluster of ChatGPT accounts originating in Russia that were used to promote the IBI, a self-described "expert community" based in Israel.

The operators used the AI to generate social media comments in English, posting them on platforms like Substack, Telegram, X, Facebook, and LinkedIn.

They even instructed the model to hide any linguistic clues of their Russian origin, using VPNs to bypass access restrictions.

While this might seem like a straightforward case of AI-generated propaganda, the operation's core was something far more sophisticated and dangerous.

Our analysis suggests that the real innovation here was not the volume of content, but the structure of the deception.

The IBI website, registered in February 2025, was not an AI-generated content farm. Instead, it was a carefully curated facade.

Of 36 articles sampled from the site, 34 were copied from real academic writings, often with false attribution.

One article on the China-Pakistan Economic Corridor was lifted from Cambridge University Press but attributed to a professor of South Asian politics.

Another on migration governance was taken from the Migration Policy Institute but credited to an Australian food science professor.

This was not lazy plagiarism; it was a deliberate strategy to manufacture authority.

The operators went further, creating a "sovereignty index" that cast Russia in a favorable light while denigrating Western countries.

Reports on France, the USA, Germany, and the EU were critical and polemic, using language that painted these nations as compromised or declining.

This is the first time an influence operation tied to Russia has gone to such elaborate lengths to build a fake institution, complete with purported experts, republished work, and a proprietary index.

The AI's role was limited to generating the promotional social media posts that pointed back to this credible-appearing website.

What does this mean for the average person or small business? The immediate impact of the operation was small—typical posts had low views, and IBI accounts had few subscribers.

The Telegram channels attracted more, with 10,000-20,000 followers each, placing the operation at the lower end of Category Three on the Brookings Breakout Scale.

But the infrastructure is the real threat. This operation was designed to be scaled over time, with assets that could be activated later.

The key takeaway is that AI did not create the lie; it made the lie look real.

For ordinary people, the lesson is to be skeptical of any new "expert community" or "think tank" that suddenly appears online. Check the authors—do they actually work there?

Look for multiple independent sources.

For small companies, the risk is that these fake institutions could be used to manufacture industry reports or endorsements that damage your reputation or manipulate your market.

The best defense is to verify the source of any data or analysis you use, and to be wary of content that seems too perfectly aligned with a single political or commercial agenda.

The vulnerability is not in the AI; it is in our own willingness to accept a well-dressed lie.

Credit: OpenAI (Disrupting a new covert influence campaign from Russia)