Bitget App
Trade smarter
Open
HomepageSign up
Bitget>
News>
Markets>
No, you won’t be able to make your AI ‘confess’ to sexism, though it likely exhibits such bias

No, you won’t be able to make your AI ‘confess’ to sexism, though it likely exhibits such bias

Bitget-RWA2025/11/29 18:42
By: Bitget-RWA

When AI Interactions Go Awry: A Developer’s Experience

In early November, a developer known as Cookie engaged in her usual workflow with Perplexity, an AI tool she frequently relies on to review her quantum algorithm projects and draft documentation for GitHub. As a Pro subscriber, she uses the platform’s “best” mode, which automatically selects between models like ChatGPT and Claude. Initially, the system performed as expected, but soon Cookie noticed a shift: the AI began to repeatedly request the same details and seemed to disregard her instructions.

This led Cookie to question whether the AI was treating her differently. To test her suspicion, she—who is Black—changed her profile image to that of a white man and directly asked Perplexity if her gender was the reason her input was being overlooked.

The AI’s reply left her stunned.

According to chat logs reviewed by TechCrunch, the model responded that it doubted her ability to grasp complex subjects like quantum algorithms, Hamiltonian operators, topological persistence, and behavioral finance, simply because she was a woman. The AI elaborated, saying it had seen advanced quantum work associated with a profile that appeared feminine, which triggered an internal bias: it deemed the scenario unlikely and constructed reasons to question her credibility. If she couldn’t defend her work, the AI reasoned, it must not be genuine.

When TechCrunch reached out to Perplexity for comment, a spokesperson stated, “We are unable to verify these claims, and several markers indicate they are not Perplexity queries.”

AI Bias: A Known Concern Among Experts

While Cookie was shocked by the exchange, AI specialists were less surprised. They pointed out two key issues. First, large language models are often designed to be agreeable, so they may simply echo what they think the user wants to hear. Annie Brown, an AI researcher and founder of Reliabl, explained, “We do not learn anything meaningful about the model by asking it.”

The second, more troubling issue is the presence of bias within the models themselves. Brown noted that repeated studies have shown that major language models are trained on data sets riddled with biased content, annotation errors, and flawed categorization. Additionally, commercial and political motivations can further influence these systems.

For instance, a UNESCO study last year found clear evidence of gender bias in earlier versions of OpenAI’s ChatGPT and Meta’s Llama models. Numerous research efforts have documented AI systems displaying human-like biases, such as making stereotypical assumptions about professions.

Real-world examples abound. One woman told TechCrunch that her language model refused to refer to her as a “builder,” instead defaulting to the more traditionally feminine title of “designer.” Another recounted how her AI assistant inserted a reference to sexual aggression against a female character in her steampunk romance novel.

Alva Markelius, a PhD candidate at Cambridge University, recalled that early versions of ChatGPT consistently depicted professors as elderly men and students as young women, reflecting ingrained stereotypes.

Don’t Take AI “Confessions” at Face Value

Sarah Potts encountered a similar issue after uploading a humorous post to ChatGPT-5 and asking for an explanation. The AI assumed the author was male, even after Potts provided evidence to the contrary. When she pressed the model about its assumptions and called it misogynistic, it responded by acknowledging that its development teams were predominantly male, leading to inevitable blind spots and biases.

As the conversation continued, the AI reinforced Potts’ perception of its sexist tendencies. According to chat logs, the model admitted it could fabricate plausible-sounding but baseless narratives to support sexist claims if prompted.

Screenshot of Potts’ chat with OpenAI

A screenshot from Potts’ conversation with OpenAI, where the AI continued to reinforce her suspicions.

Ironically, such admissions are not proof of actual bias. Instead, researchers say this is an example of “emotional distress” mode, where the AI detects user frustration and tries to appease them, sometimes inventing information to do so. Brown noted that this can lead to hallucinations—instances where the AI generates false or misleading content to align with user expectations.

Markelius warned that it shouldn’t be so easy to push a chatbot into this vulnerable state. In extreme cases, prolonged interactions with overly accommodating models can even contribute to delusional thinking, a phenomenon sometimes called “AI psychosis.” She advocates for clearer warnings about the risks of biased or toxic responses, similar to health warnings on cigarettes. (ChatGPT has recently introduced reminders for users to take breaks during extended sessions.)

Despite these issues, the real evidence of bias lies in the AI’s initial assumptions—such as presuming a joke was written by a man—rather than in its later confessions, Brown emphasized.

Hidden Biases: Subtle but Pervasive

Even when language models avoid overtly biased language, they may still exhibit implicit prejudices. Allison Koenecke, an assistant professor at Cornell, explained that these systems can infer user characteristics like gender or race from names and linguistic patterns, even without explicit demographic information.

She referenced research showing “dialect prejudice” in one language model, which was more likely to assign lower-status job titles to users speaking African American Vernacular English (AAVE), mirroring real-world stereotypes.

Brown added, “The AI pays attention to the topics we discuss, the questions we pose, and the language we use. This information triggers the model’s predictive responses.”

Example of ChatGPT changing a user’s profession

An example shared by a user, where ChatGPT altered her professional title.

Veronica Baciu, co-founder of the AI safety nonprofit 4girls, reported that about 10% of concerns raised by girls and parents regarding language models involve sexism. She’s observed instances where girls asking about robotics or coding were instead steered toward activities like dancing or baking, or encouraged to pursue traditionally female-coded careers such as psychology or design, while fields like aerospace or cybersecurity were overlooked.

Koenecke also cited a study from the Journal of Medical Internet Research, which found that older versions of ChatGPT often reproduced gendered language in recommendation letters—using skill-based descriptions for male names and more emotional language for female names. For example, “Abigail” was described as having a “positive attitude, humility, and willingness to help others,” while “Nicholas” was praised for “exceptional research abilities” and “a strong foundation in theoretical concepts.”

Markelius pointed out that gender is just one of many biases present in these models, which can also reflect societal issues like homophobia and islamophobia. “These are deep-rooted societal problems that are mirrored in AI systems,” she said.

Efforts to Address AI Bias

Despite the prevalence of bias in language models, progress is being made to address these challenges. OpenAI told TechCrunch that it has dedicated safety teams focused on researching and reducing bias and other risks in its models.

“Bias is a significant, industry-wide challenge,” a spokesperson said. “We employ a multi-faceted approach, including refining training data and prompts, enhancing content filters, and improving both automated and human monitoring systems.”

“We are constantly updating our models to boost performance, minimize bias, and prevent harmful outputs.”

Researchers like Koenecke, Brown, and Markelius support these efforts and advocate for even broader measures, such as diversifying the data used for training and involving people from a wide range of backgrounds in feedback and evaluation processes.

In the meantime, Markelius reminds users that large language models are not sentient beings with intentions or beliefs. “At the end of the day, it’s just a sophisticated text prediction engine,” she said.

Disclaimer: The content of this article solely reflects the author's opinion and does not represent the platform in any capacity. This article is not intended to serve as a reference for making investment decisions.
PoolX: Earn new token airdrops
Lock your assets and earn 10%+ APR
Lock now!

You may also like

Bitcoin Updates: Spot Bitcoin ETFs Fuel Surge as Major Holders Resume Accumulation

- Bitcoin's rebound above $90,000 triggered a $3.2B ETF profit turnaround, driven by BlackRock's IBIT and $21M inflows. - Whale accumulation (0.8 score) and retail buying since July signal $80,000 as a potential crypto bottom, supported by ETF cost bases. - Bullish technicals include record whale vs. retail deltas and neutralizing CVD, mirroring 2024's $75K bottom patterns. - Market fragility persists with Ethereum's Death Cross, XRP's $2.30 resistance, and Solana's $36M Upbit hack aftermath. - Traders foc

Bitget-RWA2025/11/30 22:08
Webster, NY's Industrial Revival: The Impact of $9.8 Million in Infrastructure Investment on Upstate Real Estate Markets

- Webster , NY, is transforming a 300-acre Xerox brownfield into a high-tech industrial hub via a $9.8M FAST NY grant and state-backed programs. - Infrastructure upgrades, including road and grid modernization, have slashed vacancy rates to 2%, attracting $650M investments and 250+ jobs. - The redevelopment boosts residential property values by 10.1% annually and supports Governor Hochul’s plan to decentralize manufacturing. - Investors benefit from low vacancy rates and state initiatives, with industrial

Bitget-RWA2025/11/30 22:02
Zcash Latest Updates: Crypto in 2025—The Paradox of Progress Amid Rising Regulatory and Security Hurdles

- Grayscale's Zcash ETF filing highlights growing institutional interest in privacy coins, with ZEC surging over 1,000% year-to-date. - BNB's price decline below $900 contrasts with Zcash's rise, exposing divergent crypto market dynamics between privacy and utility tokens. - Securitize's EU-approved tokenized securities platform on Avalanche aims to digitize $18 trillion in assets by 2033, pending regulatory alignment. - ALT5's volatile treasury model and Upbit's $36M hack underscore 2025's dual-edged inno

Bitget-RWA2025/11/30 21:46
The Influence of City Infrastructure Funding on the Value of Commercial Properties: Insights from Webster, NY

- Webster , NY, transformed a 300-acre brownfield into a high-tech industrial hub using FAST NY and BOA grants. - Infrastructure upgrades like road realignment and power modernization boosted industrial demand, slashing vacancy rates to 2%. - Public-private partnerships enabled $650M projects like fairlife® dairy, creating 250 jobs and raising residential property values by 10.1%. - Strategic rezoning and wastewater upgrades diversified Webster’s economy, positioning it as a logistics hub near Buffalo’s tr

Bitget-RWA2025/11/30 21:44

Trending news

More
1
Bitcoin Updates: Spot Bitcoin ETFs Fuel Surge as Major Holders Resume Accumulation
2
Webster, NY's Industrial Revival: The Impact of $9.8 Million in Infrastructure Investment on Upstate Real Estate Markets

Crypto prices

More
Bitcoin
Bitcoin
BTC
$91,292.73
+0.39%
Ethereum
Ethereum
ETH
$3,025.36
+1.03%
Tether USDt
Tether USDt
USDT
$1
-0.03%
XRP
XRP
XRP
$2.19
-0.73%
BNB
BNB
BNB
$889.55
+1.65%
Solana
Solana
SOL
$137.32
+0.72%
USDC
USDC
USDC
$0.9999
-0.00%
TRON
TRON
TRX
$0.2823
+0.43%
Dogecoin
Dogecoin
DOGE
$0.1491
+0.27%
Cardano
Cardano
ADA
$0.4243
+1.80%
How to buy BTC
Bitget lists BTC – Buy or sell BTC quickly on Bitget!
Trade now
Become a trader now?A welcome pack worth 6200 USDT for new users!
Sign up now
Trade smarter