During the EU elections, Faktabaari in a partnership with NORDIS contractor Check First monitored the performance of recommendation algorithms on different platforms. Now the CrossOver monitoring network has expanded to other NORDIS countries as well.  The NORDIC fact-checkers are piloting a joint investigation and monitoring efforts on how the recommendation algorithms are serving information to people on the Nordics during the US elections. The Digital election watch is up and running! 

“The Tiktok Effect” -report 

During the EU elections, Faktabaari in a partnership with Check First monitored the performance of recommendation algorithms on different platforms. The EMIF-funded CrossOver Finland project focused in particular on Tiktok during the EU elections, as it has become an increasingly popular social media platform, especially among young people. Faktabaari and Check First had been working together since late 2023 to build a network of monitoring devices in Finland. Our objective was to analyze platform recommendation algorithms in an entirely independent manner.

In July 2024, Faktabaari and Check First released a report titled "The TikTok Effect: How the app shaped political discourse during Finnish European elections 2024." This report examines how TikTok organized and presented information on political topics in Finland in the lead-up to the EU elections.

The key findings of the report were as follows:

1. For users who are searching for information on the app, TikTok’s search suggestions contain toxic language towards women and minorities. They are also biased by being almost exclusively negative towards some politicians. TikTok’s suggestions have likely shaped what people searching for political information have seen and perpetuated negative stereotypes. The current system is also easy to manipulate for adversarial actors.
2. - Videos by Finnish MEP candidates about contentious issues in domestic politics were most likely to be promoted by TikTok’s recommendation system. 17-18% of videos about spending cuts and industrial action went viral, while less than 6% of videos discussing security or nature protection did. Contrary to public concerns, videos about immigration did not reach an exceptionally large audience.
3. TikTok applied its labels for election-related videos inconsistently. Only two-thirds of videos with the three most popular election-related hashtags had labels attached to them. The inconsistency of labelling limits TikTok’s ability to direct users to authoritative information sources. The presence of labels on videos from some candidates but not on others may change how the videos are perceived.

The full report can be accessed here. 

Collaboration with Nordis

These findings were immediately shared with colleagues from EDMO NORDIS, who received regular updates on the project's progress throughout the EU elections. Updates were shared during biweekly fact-checker meetings and at consortium-wide meetings.

During the EU elections, Faktabaari was the only NORDIS member to independently monitor platform recommendation algorithms using the methodology outlined above. However, encouraged by the success of Faktabaari’s results, EDMO NORDIS members validated a plan to expand the CrossOver project into Denmark, Norway, and Sweden. This expansion was incorporated into the NORDIS 2 project plan, resulting in three additional monitoring devices being deployed in these countries.

NORDIS fact-checkers envision a collaborative study examining algorithmic recommendations during the US elections in the Nordic countries. In October 2024, a joint meeting between fact-checkers and Check First determined the keywords that NORDIS will monitor during the US elections. 

NORDIS 2 members, along with stakeholders and the general public, have access now to the CrossOver Dashboard, where they can view the data collected through the monitoring network.

Comparing the findings with VLOP transparency reports

In March 2024, signatories of the Code of Practice on Disinformation (CoP) reported on their actions to protect the upcoming EU elections. Although a systematic evaluation of this report or the Digital Services Act (DSA) report against our own findings would have required additional resources, we made some interesting observations regarding TikTok’s activities during the EU elections.

In recent years, TikTok has implemented new policies aimed at protecting election integrity by directing users to authoritative information through labels on election-related posts. In its March 2024 CoP report, TikTok stated that it directs users to Election Centres via prompts on videos, LIVEs, election-related searches, and push notifications.  TikTok announced in February 2024:

“Videos related to the European elections will be labelled to direct people to the relevant Election Centre. As part of our broader election integrity efforts, we will also add reminders to hashtags to encourage people to follow our rules, verify facts, and report content they believe violates our Community Guidelines.”

However, during the EU elections, Faktabaari and Check First found that TikTok applied its labels for election-related content inconsistently. We gathered posts containing prominent Finnish election-related hashtags, including #euvaalit, #euvaalit2024, and #eurovaalit. From February 15, 2024 (when TikTok’s labeling policy was announced), to June 9, 2024 (the election’s conclusion), we downloaded 1,265 posts using these hashtags. A manual review of 100 messages from each hashtag confirmed that all posts were election-related. Within this dataset, only 66.2% of posts (838 out of 1,265) contained the election label.

In the Tiktok Effect report, Faktabaari and Check First made recommendations to TikTok regarding labeling, stating:

“TikTok's approach to labelling election-related posts could likely be improved with further investment, but the classification process for videos remains prone to error. To enhance its goal of directing users to authoritative sources, TikTok should consider experimenting with additional mechanisms beyond just labels. For example, the platform might add notes to users' timelines, including links to election-related information and reminders to vote.”

In addition to working closely with NORDIS, Faktabaari and Check First have held multiple meetings with EDMO in 2024, including participation in the EDMO Annual Conference on 26/05/2024 and an Ad-Hoc meeting with EDMO’s Policy Working Group on Recommenders and Data Access on 30/05/2024. During these events, insights and findings from the CrossOver Finland project were shared.

Now the NORDIS partners are gathering comparative data based on parameters established by fact-checkers, with a link to the Digital Information Literacy Work, as it contributes to raising awareness of algorithms and artificial intelligence within the information consumption of everyday citizens. We hope to further investigate and compare our findings to the platforms’ CoP and DSA reports to keep the platforms accountable from our side.

The Digital election watch is up and running!