Your enforcement strategy should include clear processes for content removal, user sanctions, and appeal mechanisms. It’s also about educating users, providing support, and fostering a culture of safety within your community. Unlike more traditional careers, trust and safety professionals don’t have a set trajectory or shared disciplinary approach. Large companies may have an internal career path for their trust and safety employees but smaller companies often don’t. TSPA fills this gap by providing resources, mentoring, and programming to those who want to transition into trust and safety, as well as those who are currently in trust and safety and wanting to know how to grow professionally within this field. Visit our T&S Job Board to see which trust and safety jobs are hiring and the T&S Careers page to learn more about careers in trust and safety.
Risk mapping is essential for identifying these threats and should be customized to each platform’s specific needs. This is a foundational step in developing an effective safety policy that addresses particular vulnerabilities. Building and maintaining healthy online environments is now a required task of nearly all consumer-facing companies. This course prepares individuals to be effective practitioners of governance and, high-performing members and leaders of Trust & Safety teams. Our one-day intensive workshop provides a state-of-the-art overview on Trust & Safety and teaches best practices for building healthy, prosocial environments online.
- Until then, we’ll continue to monitor the industry’s progress in developing more comprehensive online safety metrics and the evolution of third-party tools to meet the nuanced requirements of Trust & Safety operations.
- Mitigate risks like bias, hallucination, and plagiarism throughout AI development; enhance accuracy with precise labeling; and lower monitoring costs for secure, scalable AI using AI training services and LLM operations.
- One of the most interesting findings was that different companies had unique “trigger events” that finally pushed their leadership to invest in Trust & Safety.
- Your enforcement strategy should include clear processes for content removal, user sanctions, and appeal mechanisms.
- It’s important to note that this is not about individual company policies or decisions, but rather about how to do and assess the work of trust and safety overall.
As digital platforms continue to evolve, so do the challenges of ensuring online spaces remain safe and secure. How do organizations implement content moderation practices that strike a balance between free expression and user safety? This is where Trust and Safety (T&S) teams come into play — a critical and growing field dedicated to protecting users, enforcing community guidelines and maintaining the integrity of digital services. The rapid adoption of Generative AI tools has introduced unprecedented risks for the Trust and Safety community. Offenders like child predators, terrorists, and racists now have easier access to create highly realistic, harmful content. Unlike established UGC platforms, new GenAI platforms face a complex threat landscape.
To effectively safeguard their users from harmful content, these companies must grapple with incredibly complex issues, from child sexual exploitation to misinformation, while also responding to new and rapidly evolving threats. Early trust and safety teams responded to common forms of online abuse (e.g., fraud, scams, phishing, spam, trolling, malware) which surfaced during the commercialization of the internet and the rapid growth of the web in the 1990s. Developing this trust and safety framework often starts with a company defining its core values and trust principles, which in turn leads to tailored trust and safety models, policy development, and enforcement operations.
According to research, games with less toxicity have a 16% higher player retention rate over toxic games, while gaming businesses with high player retention rates achieve revenue growth that is 50% higher than those with low retention rates. Academic research shows that toxic players drive away new players, but that experienced players are more resilient to deviant behavior. While the Stanford Internet Observatory coordinates this consortium, virtually all of the contributions come from volunteers across industry, academia, civil society, and government.
Long before the phrase became popularized, eBay was an early adopter of the term “trust and safety,” where “trust” referred to trust among eBay users and of the company itself, and “safety” referred to keeping platform users safe (Boyd, 2022). Transparency means clearly communicating a platform’s policies, practices, and decisions. This includes explaining how data is collected, used, and protected, and how content moderation and user behavior are managed. Transparency reports, which showcase efforts to improve user safety, have been around for a while.
Topics include social/behavioral approaches to online governance, leveraging procedural justice theory, and building effective Trust & Safety teams. With 66 percent of the world’s population using the internet, most of the planet has a stake in how digital services manage safety risks. At the same time, with so many new entrants joining this discussion, we risk forgetting the lessons learned from debates that have been raging since the internet’s inception.
Beyond compliance, governance should reflect your platform’s core values and mission, aiding consistent decision-making, especially around complex T&S policy decisions. When users see alignment between your policies, actions, and stated values, it strengthens their trust in your platform. If you are an organization, you can support TSPA and trust and safety professionals by sponsoring our events.
Our Address
To ensure trust and reliability, platforms and foundation models must embed safety measures across the AI lifecycle, covering development, deployment and ongoing management. This study draws from hundreds of responses and in-depth interviews with over 100 Trust & Safety professionals across various industries such as social media, dating, gaming, and marketplaces. Shining a light on their most pressing challenges, emerging threats, and the innovative ways these teams combat online harm. Sometimes, scammers will misspell names of trusted URLs or replace one of the letters with a number, like writing go0gle.com instead of google.com. In a rush, this is something you can easily miss, and with fake websites imitating real websites as well as they do, it’s pretty easy to give your personal information to the bad guys.
Positive Experiences
Companies deploying enterprise GenAI applications also face risks like internal data leaks and regulatory compliance issues. TSPA partners with those trust and safety teams to equip them with the expertise and skills they need to successfully navigate this incredibly high-stakes environment. This curriculum, written by volunteers with extensive experience in the field, is a key piece of that support. We hope this curriculum provides context on the complex considerations that influence how trust and safety professionals define acceptable behavior and content online and work to safeguard user safety and privacy.
In 2024, we updated our Misrepresentation Policy to address public-figure impersonation scams in Google Ads. These scams combine AI-generated photos, voiceovers, or videos (often called “deepfakes”) with fabricated news articles and social-media posts to promote fraudulent investment opportunities. If you work at a technology company and are looking to offer this training to your team of any size, please be in touch as we are excited about the possibility of providing this curriculum and training to practitioners. Innovation-driven governance, enabled by streamlined decision-making and a pioneering well-being program, helps our teams deliver tailored solutions that adapt to industry trends and meet evolving client needs. Have experienced deepfake attacks or scams aimed at stealing personal information or money. Have encountered fake news or articles online, while 38.8% have seen fraudulent product reviews.
These two tasks require different tools and efforts, with actor signals posing a bigger challenge to platforms. Through a blend of quantitative data and qualitative insights, the report seeks to equip Trust & Safety professionals with valuable knowledge and practical tools to help you navigate and address emerging threats and issues in the Online Safety space. Because they concern complex and technical issues, digital security threats can be much harder to get your head around. Even worse, digital threats are all around us, lurking in our inboxes, text messages, and on social media. Even with increasingly frequent jaw-dropping advances in technology, we can’t predict the future. But whatever digital safety risks lie over the horizon, we’ll be better situated to handle them if we learn from a few hard-won lessons from the past.
An all-important function that includes filtering, reviewing, and managing user-generated content. Content moderation requires balancing automated tools, which can quickly scan and flag inappropriate content, with human moderators, who handle more nuanced decisions, like cases involving hate speech or misinformation. Platforms must also continually update their moderation policies to reflect company values, social norms, novel abuse tactics, and evolving legal frameworks so that the content hosted on their sites promotes a safe and respectful environment. When they first get connected, new Internet users enter a complex digital environment, which can be difficult to navigate safely without prior experience. Helping newly connected users develop trust and safety skills is essential for building a safer, more trustworthy, and inclusive Internet.
Over time this Framework will evolve as they grow in maturity and can be assessed in more standard ways, such as in other disciplines like security and privacy. As discussed in the State of Trust and Safety 2024 report, maintaining user trust is crucial for platform longevity and engagement. By making Trust and Safety a core focus, organizations can create a reliable, supportive online environment that encourages community growth and secures their future in the rapidly evolving digital world. Looking forward, the digital landscape will continue to change, necessitating continuous adaptation to new threats and technologies. Trust and Safety strategies must evolve alongside these changes to ensure that online communities remain safe and trustworthy for all users.
While these companies and technologies drive innovation and growth and connect people in unprecedented ways, they also serve as vehicles for harmful and unwanted behaviors. To address such problems, companies have invested in efforts to prevent, mitigate, minimize, and—where possible and warranted—remove unwanted content or conduct. When people understand how and why decisions are made, they’re more likely to perceive them as fair, even if they don’t always agree with the outcomes. This could involve publishing regular transparency reports, explaining policy changes clearly, or providing detailed explanations when taking enforcement actions.
Senate passed the Kids Online Safety and Privacy Act (KOSPA) with overwhelming support, voting 91-3 in favor. This landmark legislation aims to protect children online by requiring platforms to limit addictive features, ban targeted ads to minors, restrict data collection, and enforce stricter default privacy and safety settings. KOSPA also addresses harmful content related to suicide and self-harm, eating disorders, and substance abuse. Although it still needs approval from the House, KOSPA represents a significant step forward in child online safety. These regulations and others create a complex legal framework that companies must navigate to maintain Trust and Safety on their platforms. Compliance not only helps avoid legal issues but also builds credibility and trust with users by showing a commitment to protecting their rights and personal data.
Definition Of Trust & Safety: Establishing Confidence And Security
Accountability means that both users and platforms are responsible for their actions, with clear and enforceable consequences for violations. This requires setting community guidelines or terms of service that users must follow and platforms must enforce. For platforms, accountability means responding to users and regulators when issues arise, whether through internal investigations, public apologies, or making systemic changes to prevent future problems. Ultimately, accountability strengthens trust by demonstrating that the company takes its responsibilities seriously and is committed to maintaining a safe and fair environment. From retail and social media, to government, travel and hospitality, diverse organizations are making investments in their Trust & Safety operations. In lectures and small group exercises, participants take an inter-disciplinary approach, acquiring tools and strategies from diverse fields that include law & policy development to data analytics, to behavioral science and human-centered design.
Our first report, based on internal assessments by 10 participating companies, was published in July 2022. We have since been working to enhance our process to support independent, third-party evaluation of company assessments. New technologies, like generative AI, can offer improved tools for T&S enforcement, while also potentially introducing new avenues for abuse. newlineBy staying informed and adaptable, you can stay abreast of technological advancements and proactively address emerging risks. Enforcement & Response is where your governance policies and safety-by-design principles are put into action. This aspect focuses on implementing operations and tools to enforce policies and respond promptly to violations.
T&S professionals work for large online services, small startups, and as volunteers; they practice on every continent, from Silicon Valley to Sydney, from Dublin to Dhaka, from Singapore to São Paolo. TSPA was founded in the wake of the 2018 Content Moderation at Scale (COMO) conference in Santa Clara, which was convened by one of TSPA’s founders and board members, Santa Clara University law professor Eric Goldman. During this conference, it became clear that there was a need for more community amongst people who do trust and safety work. Not only that, but organizations will need to prepare in light of the Trump administration’s expressed interest in reforming or potentially repealing Section 230 of the Communications Decency Act. Section 230, known as “the 26 words that created the internet,” is a statute that protects internet platforms from legal liability for content generated and posted by users. As the law continues to come under scrutiny from Congress, FCC Chairman Brendan Carr and others, internet platforms are considering what this means for their businesses and their responsibility for user safety.
A well-rounded Trust and safety strategy typically combines proactive and reactive approaches to provide comprehensive protection for users and the platform. Billions of people around the world learn, work, connect, and express themselves online. As internet platforms democratize access to information in an unprecedented way, they can also http://www.instagram.com/socialitylimited/ inadvertently facilitate the coordination of harmful activity and provide a venue for bad behavior.
The discipline of trust and safety was born in the early 2000s at companies like eBay and Yahoo, and then grew into the massive, global phenomenon of commercial content moderation with the rise of social media thereafter. Trust and safety professionals play a crucial role in maintaining a secure and reliable online environment. These professionals are responsible for developing and implementing trust and safety strategies, policies, and enforcement mechanisms. They analyze user behavior, monitor platform activities, respond to incidents, and collaborate with relevant stakeholders to enhance trust and safety practices. Additionally, they stay updated with industry trends and emerging risks, ensuring continuous improvement and adaptation of trust and safety measures.