AI

4 questions to ask when evaluating AI prototypes for bias

Comment

Hole in Blue Page
Image Credits: mikroman6 (opens in a new window) / Getty Images

Veronica Torres

Contributor

Veronica Torres is the worldwide privacy and regulatory counsel for Jumio, where she provides strategic legal counsel regarding business processes, applications and technologies to ensure compliance with privacy laws.

It’s true there has been progress around data protection in the U.S. thanks to the passing of several laws, such as the California Consumer Privacy Act (CCPA), and nonbinding documents, such as the Blueprint for an AI Bill of Rights. Yet, there currently aren’t any standard regulations that dictate how technology companies should mitigate AI bias and discrimination.

As a result, many companies are falling behind in building ethical, privacy-first tools. Nearly 80% of data scientists in the U.S. are male and 66% are white, which shows an inherent lack of diversity and demographic representation in the development of automated decision-making tools, often leading to skewed data results.

Significant improvements in design review processes are needed to ensure technology companies take all people into account when creating and modifying their products. Otherwise, organizations can risk losing customers to competition, tarnishing their reputation and risking serious lawsuits. According to IBM, about 85% of IT professionals believe consumers select companies that are transparent about how their AI algorithms are created, managed and used. We can expect this number to increase as more users continue taking a stand against harmful and biased technology.

So, what do companies need to keep in mind when analyzing their prototypes? Here are four questions development teams should ask themselves:

Have we ruled out all types of bias in our prototype?

To build effective, bias-free technology, AI teams should develop a list of questions to ask during the review process that can help them identify potential issues in their models.

There are many methodologies AI teams can use to assess their models, but before they do that, it’s critical to evaluate the end goal and whether there are any groups who may be disproportionately affected by the outcomes of the use of AI.

For example, AI teams should take into consideration that the use of facial recognition technologies may inadvertently discriminate against people of color — something that occurs far too often in AI algorithms. Research conducted by the American Civil Liberties Union in 2018 showed that Amazon’s face recognition inaccurately matched 28 members of the U.S. Congress with mugshots. A staggering 40% of incorrect matches were people of color, despite them making up only 20% of Congress.

By asking challenging questions, AI teams can find new ways to improve their models and strive to prevent these scenarios from occurring. For instance, a close examination can help them determine whether they need to look at more data or if they will need a third party, such as a privacy expert, to review their product.

Plot4AI is a great resource for those looking to start.

Have we enlisted a designated privacy professional or champion?

Due to the nature of their job, privacy professionals have been traditionally viewed as barriers to innovation, especially when they need to review every product, document and procedure. Rather than viewing a privacy department as an obstacle, organizations should instead see it as a critical enabler for innovation.

Enterprises must make it a priority to hire privacy experts and incorporate them into the design review process so that they can ensure their products work for everyone, including underserved populations, in a way that’s safe, compliant with regulations and free of bias.

While the process for integrating privacy professionals will vary according to the nature and scope of the organization, there are some key ways to ensure the privacy team has a seat at the table. Companies should start small by establishing a simple set of procedures to identify any new, or changes to existing, processing activities related to personal information.

The key to success with these procedures is to socialize the process with executives, as well as product managers and engineers, and ensure they are aligned with the organization’s definition of personal information. For example, while many organizations generally accept IP addresses and mobile device identifiers as personal information, outdated models and standards may categorize these as “anonymous.” Enterprises must be clear about what types of information qualify as personal information.

Furthermore, organizations may believe that personal information used in their products and services poses the greatest risk and should be the priority for reviews, but they must take into account that other departments, such as human resources and marketing, also process large amounts of personal information.

If an organization doesn’t have the bandwidth to hire a privacy professional for every department, they should consider designating a privacy champion or advocate who can spot issues and escalate them to the privacy team if needed.

Is our people and culture department involved?

Privacy teams shouldn’t be the only ones responsible for privacy within an organization. Every employee who has access to personal information or has an impact on the processing of personal information is responsible.

Expanding recruitment efforts to include candidates from different demographic groups and various regions can bring diverse voices and perspectives to the table. Hiring diverse employees shouldn’t stop at entry-and-mid-level roles, either. A diverse leadership team and board of directors are absolutely essential to serve as representatives for those who cannot make it into the room.

Companywide training programs on ethics, privacy and AI can further support an inclusive culture while raising awareness of the importance of diversity, equity and inclusion (DEI) efforts. Only 32% of organizations require a form of DEI training for their employees, emphasizing how improvements are needed in this area.

Does our prototype align with the AI Bill of Rights Blueprint?

The Biden administration issued a Blueprint for an AI Bill of Rights in October 2022, which outlines key principles, with detailed steps and recommendations for developing responsible AI and minimizing discrimination in algorithms.

The guidelines include five protections:

  1. Safe and effective systems.
  2. Algorithmic discrimination.
  3. Data privacy.
  4. Notice and explanation.
  5. Human alternatives, consideration and fallback.

While the AI Bill of Rights doesn’t enforce any metrics or pose specific regulations around AI, organizations should look to it as a baseline for their own development practices. The framework can be used as a strategic resource for companies looking to learn more about ethical AI, mitigating bias and giving consumers control over their data.

The road to privacy-first AI

Technology has the ability to revolutionize society as we know it, but it will ultimately fail if it doesn’t benefit everyone in the same way. As AI teams bring new products to life or modify their current tools, it’s critical that they apply the necessary steps and ask themselves the right questions to ensure they have ruled out all types of bias.

Building ethical, privacy-first tools will always be a work in progress, but the above considerations can help companies take steps in the right direction.

More TechCrunch

Silo, a Bay Area food supply chain startup, has hit a rough patch. TechCrunch has learned that the company on Tuesday laid off roughly 30% of its staff, or north…

Food supply chain software maker Silo lays off ~30% of staff amid M&A discussions

Featured Article

Meta’s new AI council is composed entirely of white men

Meanwhile, women and people of color are disproportionately impacted by irresponsible AI.

5 hours ago
Meta’s new AI council is composed entirely of white men

If you’ve ever wanted to apply to Y Combinator, here’s some inside scoop on how the iconic accelerator goes about choosing companies.

Garry Tan has revealed his ‘secret sauce’ for getting into Y Combinator

Indian ride-hailing startup BluSmart has started operating in Dubai, TechCrunch has exclusively learned and confirmed with its executive. The move to Dubai, which has been rumored for months, could help…

India’s BluSmart is testing its ride-hailing service in Dubai

Under the envisioned framework, both candidate and issue ads would be required to include an on-air and filed disclosure that AI-generated content was used.

FCC proposes all AI-generated content in political ads must be disclosed

Want to make a founder’s day, week, month, and possibly career? Refer them to Startup Battlefield 200 at Disrupt 2024! Applications close June 10 at 11:59 p.m. PT. TechCrunch’s Startup…

Refer a founder to Startup Battlefield 200 at Disrupt 2024

Social networking startup and X competitor Bluesky is officially launching DMs (direct messages), the company announced on Wednesday. Later, Bluesky plans to “fully support end-to-end encrypted messaging down the line,”…

Bluesky now has DMs

The perception in Silicon Valley is that every investor would love to be in business with Peter Thiel. But the venture capital fundraising environment has become so difficult that even…

Peter Thiel-founded Valar Ventures raised a $300 million fund, half the size of its last one

Featured Article

Spyware found on US hotel check-in computers

Several hotel check-in computers are running a remote access app, which is leaking screenshots of guest information to the internet.

9 hours ago
Spyware found on US hotel check-in computers

Gavet has had a rocky tenure at Techstars and her leadership was the subject of much controversy.

Techstars CEO Maëlle Gavet is out

The struggle isn’t universal, however.

Connected fitness is adrift post-pandemic

Featured Article

A comprehensive list of 2024 tech layoffs

The tech layoff wave is still going strong in 2024. Following significant workforce reductions in 2022 and 2023, this year has already seen 60,000 job cuts across 254 companies, according to independent layoffs tracker Layoffs.fyi. Companies like Tesla, Amazon, Google, TikTok, Snap and Microsoft have conducted sizable layoffs in the first months of 2024. Smaller-sized…

10 hours ago
A comprehensive list of 2024 tech layoffs

HoundDog actually looks at the code a developer is writing, using both traditional pattern matching and large language models to find potential issues.

HoundDog.ai helps developers prevent personal information from leaking

The changes are designed to enhance the consumer experience of using Google Pay and make it a more competitive option against other payment methods.

Google Pay will now display card perks, BNPL options and more

Few figures in the tech industry have earned the storied reputation of Vinod Khosla, founder and partner at Khosla Ventures. For over 40 years, he has been at the center…

Vinod Khosla is coming to Disrupt to discuss how AI might change the future

AI has already started replacing voice agents’ jobs. Now, companies are exploring ways to replace the existing computer-generated voice models with synthetic versions of human voices. Truecaller, the widely known…

Truecaller partners with Microsoft to let its AI respond to calls in your own voice

Meta is updating its Ray-Ban smart glasses with new hands-free functionality, the company announced on Wednesday. Most notably, users can now share an image from their smart glasses directly to…

Meta’s Ray-Ban smart glasses now let you share images directly to your Instagram Story

Spotify launched its own font, the company announced on Wednesday. The music streaming service hopes that its new typeface, “Spotify Mix,” will help Spotify distinguish its own unique visual identity. …

Why Spotify is launching its own font, Spotify Mix

In 2008, Marty Kagan, who’d previously worked at Cisco and Akamai, co-founded Cedexis, a (now-Cisco-owned) firm developing observability tech for content delivery networks. Fellow Cisco veteran Hasan Alayli joined Kagan…

Hydrolix seeks to make storing log data faster and cheaper

A dodgy email containing a link that looks “legit” but is actually malicious remains one of the most dangerous, yet successful, tricks in a cybercriminal’s handbook. Now, an AI startup…

Bolster, creator of the CheckPhish phishing tracker, raises $14M led by Microsoft’s M12

If you’ve been looking forward to seeing Boeing’s Starliner capsule carry two astronauts to the International Space Station for the first time, you’ll have to wait a bit longer. The…

Boeing, NASA indefinitely delay crewed Starliner launch

TikTok is the latest tech company to incorporate generative AI into its ads business, as the company announced on Tuesday that it’s launching a new “TikTok Symphony” AI suite for…

TikTok turns to generative AI to boost its ads business

Gone are the days when space and defense were considered fundamentally antithetical to venture investment. Now, the country’s largest venture capital firms are throwing larger portions of their money behind…

Space VC closes $20M Fund II to back frontier tech founders from day zero

These days every company is trying to figure out if their large language models are compliant with whichever rules they deem important, and with legal or regulatory requirements. If you’re…

Patronus AI is off to a magical start as LLM governance tool gains traction

Link-in-bio startup Linktree has crossed 50 million users and is rolling out the beta of its social commerce program.

Linktree surpasses 50M users, rolls out its social commerce program to more creators

For a $5.99 per month, immigrants have a bank account and debit card with fee-free international money transfers and discounted international calling.

Immigrant banking platform Majority secures $20M following 3x revenue growth

When developers have a particular job that AI can solve, it’s not typically as simple as just pointing an LLM at the data. There are other considerations such as cost,…

Unify helps developers find the best LLM for the job

Response time is Aerodome’s immediate value prop for potential clients.

Aerodome is sending drones to the scene of the crime

Granola takes a more collaborative approach to working with AI.

Granola debuts an AI notepad for meetings