Abusive posts online are less likely to be identified if they include emojis, new research suggests.
Some algorithms designed to track hate content, including a Google product, are not as effective when using these symbols.
Harmful posts can end up being lost altogether, while acceptable posts are wrongly tagged as offensive. according to the Oxford Internet Institute.
After England lost in the Euro 2020 final, Marcus Rashford, Bukayo Saka and Jadon Sancho received a torrent of racist abuse on social media, many of them featuring monkey emojis.
The start of the Premier League raises fears that more will follow unless social media companies can better filter this content.
Many of the systems in use today are trained on large text databases that rarely feature emojis. They may also have a hard time working when they come across symbols posted online.
Sky News Analysis showed that Instagram accounts that posted racist abuse that featured emojis were three times less likely to be shut down than those that posted hate messages that only contained text.
To help address this problem, the researchers created a database of nearly 4,000 sentences, most of which included emojis that were used offensively.
This database was used to train an artificial intelligence model to understand which messages were abusive and which were not.
By using humans to guide and modify the model, he was better able to learn the underlying patterns that indicate whether a post is obnoxious.
The researchers tested the model on abuse related to race, gender, gender identity, sexuality, religion, and disability.
They also examined different ways that emoji can be used offensively. This included describing groups with an emoji (a rainbow flag to represent gay people, for example) and adding hate text.
Perspective API, a Google-backed project that offers software designed to identify hate speech, was only 14% effective at recognizing hateful comments of this type in the database.
This tool is widely used and currently processes more than 500 million requests per day.
Sites like Instagram have come under fire for hosting abusive posts.
The researchers' model provided about a 30% improvement in correctly identifying hateful and non-hateful content, and up to 80% improvement over some types of emoji-based abuse.
However, even this technology will not be entirely effective. Many comments may only be obnoxious in particular contexts, for example, next to a photo of a black soccer player.
And problems with hateful images were highlighted in a recent report by the Woolf Institute, a research group that examines religious tolerance. They showed that even when using Google's SafeSearch feature, 36% of the images displayed in response to the search for "Jewish jokes" were anti-Semitic.
The evolution of language use makes this task even more difficult.
Research from the University of Sao Paulo showed that an algorithm rated Twitter accounts belonging to drag queens as more toxic than some white supremacist accounts.
That was because the technology failed to recognize that language used by someone about their own community could be more offensive if used by someone else.
Incorrect categorization of non-hateful content has significant downsides.
"False positives run the risk of silencing the voices of minority groups," said Hannah Rose Kirk, lead author of the Oxford research.
Solving the problem is made more difficult by the fact that social media companies tend to protect their software and data strictly, which means that the models they use are not available for scrutiny.
"More can be done to keep people safe online, particularly people from already underserved communities," added Ms. Kirk.
The Oxford researchers are sharing their database online, allowing other academics and companies to use it to improve their own models.
The Data and forensic analysis team is a multipurpose unit dedicated to providing transparent Sky News journalism. We collect, analyze and visualize data to tell data-driven stories. We combine traditional reporting skills with advanced analysis of satellite imagery, social media, and other open source information. Through multimedia storytelling, our goal is to better explain the world and at the same time show how our journalism is made.
