If there's one thing apparently on everyone's mind in this "ultimate election year" -- a year in which approximately half the world's population will cast a ballot -- it's the topic of generative AI, and its potential to disrupt, dissuade, and distort the fundamental building blocks of democracy. From India to Italy, from the UK to the U.S., politicians, media, and civil society have been warning about the possible impact of AI-generated "fake news" in swaying voters' opinions.
So far, the worst of these predictions have yet to materialize. In India's recent record-breaking electoral period, deepfake video and audio content proliferated, but it does not appear to have deceived many into changing their views. Meanwhile, in the EU, efforts by tech companies and lawmakers alike to "pre-bunk" election disinformation, whether generated by AI or not, ahead of the European Parliament elections seem to have paid off. There is evidence that some AI chatbots generated false EU election-related information, that far-right European parties used generative AI to create campaigning materials, and that in Rwanda, pro-government supporters are using ChatGPT to mass produce authentic-seeming propaganda. But at least for now, generative AI does not appear to be playing as large a role as, for instance, online targeted advertising, in influencing election outcomes.
Nevertheless, the year isn't over yet, and several high-stakes elections are still to come in France, the U.S., Africa, and elsewhere. With that in mind, this explainer piece takes a closer look at what exactly we mean when we talk about AI-generated disinformation, what the specific human rights risks are when it comes to elections, and what steps we can take to limit the impact on people's fundamental right to participate, fully and freely, in democratic processes.
The short answer is maybe. But before we go any further, it's important to remind ourselves that disinformation and misinformation are not the same thing. As per the working (although not universally accepted) definitions adopted by the UN special rapporteur on the promotion and protection of the right to freedom of opinion and expression, disinformation is understood as false information intentionally and deliberately disseminated in order to cause serious social harm, while misinformation describes the unknowing or inadvertent dissemination of false information. It's also worth remembering that neither disinformation nor misinformation is considered illegal, even if it may be potentially harmful, and no national or international laws should criminalize its dissemination.
The use of generative AI systems may exacerbate the challenges of both dis- and misinformation in two ways. Firstly, generative AI systems and applications are prone to randomly generating false information -- also known as "hallucinations" or "confabulations" (although this framing problematically suggests it is a bug rather than a feature). In reality, generative AI tools simply respond to the prompts they are given by producing plausible-sounding text, which, upon closer inspection, often turns out to be complete nonsense. They cannot be relied upon to generate factual information. Secondly, people can deliberately use generative AI systems to more easily, cheaply, and quickly create misleading content at large scale.
Despite this, more and more apps, including search engines, are incorporating generative AI to give the illusion of a knowledgeable "conversational interface," even though this has been shown to saturate results with inaccurate information. This is especially dangerous in an election context, when voters may seek information on political candidates and manifestos or voting requirements, dates, and locations. Some companies have recognized the risk of such unreliable, unpredictable information being produced around elections. Google, for instance, has restricted its Gemini system from providing election information, while Meta's AI chatbot only recently lifted its block on election queries in India, following the conclusion of elections there.
Generative AI tools make it easier, cheaper, and faster for people to produce disinformation and misinformation online. As mentioned above, you can use large-language models (LLMs) to produce a large amount of text that sounds like it could have been spoken or written by a human at little to no cost. You can also use generative AI tools to create realistic-looking and sounding images, videos, and audio (i.e. binary) files, without requiring much specialist technical knowledge or tools.
In some cases, generative AI tools can also increase the quality, and thus the persuasiveness, of online disinformation content compared with human-created false information. When it comes to text-based outputs, LLMs generate persuasive false information more cheaply than humans do, often citing sources that appear credible (even if they are made-up), referencing testimonials and statistics, and even acknowledging alternative explanations or contrary opinions. Humans can, of course, do the same if they want their disinformation to seem credible and convincing, but it takes more time, more effort, and more money.
On the other hand, generative AI tools don't always produce image, audio, and video outputs relatively effortlessly. Certainly, these binary outputs produced by generative AI are often cheaper to make than those made by humans; but when compared with high-quality human-made content, they aren't necessarily more convincing. Again, it comes back to lowering the cost, time, and skill barriers to entry for producing somewhat convincing content.
There's no solid evidence that the availability of generative AI tools is increasing demand for disinformation; the mere existence of AI-generated manipulated videos of U.S. President Joe Biden claiming to have encountered aliens does not mean that people want to watch this content. Nor can we say with definite certainty that AI-generated disinformation is significantly more persuasive than disinformation generated by humans. The fact that a given piece of disinformation exists is not the main issue; the real problems relate to how such disinformation is disseminated across large platforms. In fact, AI-generated disinformation is amplified in the same way that all disinformation is spread; via online targeting based on individuals' profiling, the abuse of people's data, opaque ad delivery techniques, and recommender systems optimized for engagement and profit -- all practices that are deeply problematic even before you add generative AI into the equation. These are broader issues that must be addressed regardless of how the disinformation in question is produced.
As mentioned above, the challenges related to the spread of disinformation go beyond the provenance of that content, but for now, many proposed measures seem focused on the "made by AI" component.
For instance, various possible measures fall under the category of "labeling," including suggesting the use of metadata with information on the date, time, and place the content was created, or embedding invisible watermarks within the content, which identify the AI model used to create it or reveal information about who prompted the output. Social media platforms X and Meta are rolling out visible labels for AI-generated content, intended to "communicate information credibility to users." These can either provide explicit information about content's credibility or can be contextual labels providing more information about a piece of content without making any explicit claim on its veracity or assessing its credibility.
Labeling isn't without its problems, however. When platforms label what is "false," this implies that what is left unlabeled must, by extension, be true and trustworthy, when this is not always the case. This risks undermining the legitimacy of, and thus people's trust in, proposed labeling systems; a situation that could be further aggravated if left solely in the hands of a few private companies dominating the market. While labeling reactively adds a marker to AI-generated content, allowing it to then be "read" as such, another, more preemptive approach is to identify such content by deploying tools that detect whether or not something is AI-generated. This is the thinking behind the development and use of AI content detection tools, which analyze linguistic and stylistic features to identify patterns consistent with AI-generated outputs, and which have been touted as a solution to prevent academic plagiarism.
But such tools have already been shown to be unreliable, frequently producing false positives, and lacking the ability to understand subtle nuances and human language, particularly in non-English language contexts. And with AI constantly evolving, such tools will always be playing catch-up, trying and failing to stay one step ahead.
Beyond efforts to identify it, there are numerous tools for applying content moderation to AI-generated content, from its creation to its dissemination. It's worth distinguishing between tools accessible to users only via a platform or an application programming interface (API), such as OpenAI's ChatGPT, and "open source" tools that can be downloaded, locally run, and modified by users, such as StabilityAI's Stable Diffusion. The former gives developers control over how exactly users can prompt the tool or what kind of outputs it can produce, while in the latter, any inbuilt content moderation safeguards can be removed or circumvented.
Broadly speaking, the moderation of AI-generated content starts during the training phase of the AI model's development, but can also happen at a number of other points, from training to end-user interaction. The moderation of end-user interactions happens when you input a prompt, which can be subject to automated content moderation. For example, if you explicitly ask ChatGPT to produce blunt disinformation about upcoming elections, you may get an automatic response declining to do so. Then, even if the prompt itself is allowed, the output may be moderated to ensure that an image produced contains no nudity or that a text output contains no racial slurs. OpenAI does this by "running both the prompt and completion through an ensemble of classification models aimed at detecting and preventing the output of harmful content," i.e. very similar automated decision-making processes to those widely used to moderate content generated by humans, which already over-restrict freedom of expression while simultaneously failing to detect illegal content.
Finally, company policies govern how people can create and disseminate AI-generated content, with generative AI tools' terms of use ostensibly prohibiting high-risk use cases, such as generating discriminatory content. However, a recent analysis of six major chatbots' rules against disinformation, which should govern user prompts, showed their policies to be vague, lacking any definition of what actually constitutes disinformation. And as we've previously analyzed, vaguely defined, open-ended policies lead to non-transparent practices, including excessive restrictions on freedom of expression. Some social media platforms are doing marginally better by gradually incorporating specific rules addressing AI-generated content.
As discussed above, the primary issue with AI-generated, credible-seeming disinformation is not the fact that it exists, but that it is being distributed and consumed. But focusing on generative AI in this context is somewhat of a red herring; this is a broader issue that goes beyond the provenance of the content, whether generated by AI or not. For now, it isn't clear what, if any, additional, new risks generative AI poses in the context of election disinformation that were not already present before generative AI came on the scene.
In the meantime, while the world panics over the as-yet-unproven unique impact of generative AI on elections, the very real and distinct societal harms caused by this technology are going unaddressed. Although it hasn't been the focus of this piece, we would be remiss not to mention the serious systemic risks posed by generative AI models used to create and disseminate non-consensual sexual imagery and child sexual abuse material, for instance, and the heightened online threats to the human rights, safety, and dignity of women, LGBTQ+ communities, and other racialized and marginalized groups.
To address these and other risks, policymakers must adequately enforce existing legal frameworks, such as the EU's Digital Services Act (DSA) and AI Act, which can tackle the dissemination of AI-generated disinformation at its roots. For example, enforcing the EU-wide ban on targeted advertising based on people's profiling is likely to be more effective than any content provenance or automated detection tools. It is also vital to encourage further research into the systemic risks that generative AI tools pose for societies and democratic discourse more broadly, in order to ensure that any future legislative efforts to tackle AI-generated disinformation safeguard, rather than undermine, fundamental human rights.