• Latest
  • All
  • How To
Web Crawler

News Sites Crackdown on AI Crawlers

September 20, 2023
instagram-edits

Instagram Adds Teleprompter to Edits App in Bid to Rival CapCut

June 4, 2025
Sora

Microsoft Brings Sora AI Video Generator to Bing App

June 4, 2025
google-ai-edge-gallery

Google Quietly Launches App to Run AI Models Locally on Android

June 4, 2025
adobe-photoshop-iphone

Adobe Launches Photoshop Beta App for Android

June 4, 2025
DHgate Tablet Cases deals
Material 3 Expressive

Material 3 Expressive Rolls Out in Gmail and Messages

June 4, 2025
Rose Njeri, the creator of Civic Email

Rose Njeri Charged With Cybercrime for Letting Kenyans Email Their MPs

June 3, 2025
Infinix Hot 60 Pro+ Leaked Design

Infinix Hot 60 Pro+ Could Be the World’s Thinnest Curved Display Phone

June 3, 2025
Infinix Note 50 Pro Review

Infinix Note 50 Pro Review: A Budget Phone Shouldn’t Be This Good

google-chrome

Three Billion Chrome Browser Users at Risk from Zero Day Hack

June 3, 2025
Jumia technologies

Jumia Gets a Tech Boost as AXIAN Telecom Joins the Table

June 3, 2025
whatsapp-iphone

Your Phone May No Longer Run WhatsApp After This Update

June 3, 2025
XChat

X Unveils XChat with Encrypted Messaging and Calls

June 3, 2025
Techweez | Tech News, Reviews, Deals, Tips and How To
  • News
  • Entertainment
  • Reviews
  • Features
  • Editorial
No Result
View All Result
Techweez | Tech News, Reviews, Deals, Tips and How To
  • News
  • Entertainment
  • Reviews
  • Features
  • Editorial
No Result
View All Result
Techweez | Tech News, Reviews, Deals, Tips and How To
No Result
View All Result

News Sites Crackdown on AI Crawlers

ElvyLewis Ndungu by ElvyLewis Ndungu
September 20, 2023
in News
Reading Time: 3 mins read
252
0
Web Crawler

In recent months, there has been an explosion of generative Artificial Intelligence (AI) as more companies continue to develop consumer-facing tools to help automate tasks, write documents, do research on various topics or even basic coding.

OpenAI’s ChatGPT took the world by storm following its release, having reached an estimated 100 million users just within two months after launch. Arguably, following its success more consumers and companies have continued to leverage generative AI models and tools in their daily tasks.

Since then, more generative AI tools have been released by other key tech players as well as smaller startups including Google’s conversational chatbot, Bard, Microsoft’s Bing chat, and OpenAI’s text-to-image generator, DALL·E 2.

However, the rise of large language models (LLMs) and generative AI have ushered new in challenges, bringing to light copyright issues. This has resulted in pushback from news sites, publishers, and intellectual property holders who see their data being collected by AI crawlers. With no clear regulatory rules controlling AI’s use of copyrighted material yet, some of the world`s largest news websites have taken matters into their own hands.

According to data presented by AltIndex.com, nearly one-third of the world’s top 50 news sites have blocked AI crawlers from accessing their content, and their number continues rising.

Notably, CNN, the New York Times, the Daily Mail, Reuters, and Bloomberg Have All Blocked At least One AI Crawler.

Crackdown on AI crawlers

AI companies send crawlers to collect data to train their models and provide information for chatbots.

However, as data is one of their core advantages, many of the world’s largest news websites have become extremely cautious, especially since there is generally no upside to handing over their data to AI crawlers, according to AltIndex.

Last month OpenAI launched its GPTBot crawler to collect data to enhance its language models. This escalated the situation further despite assurances that paywalled content would be excluded from websites. Several high-profile news sites, including CNN, Reuters, and the New York Times, blocked GPTBot.

According to a Kirwan Digital Marketing Agency survey, 28% of the top 50 news sites worldwide have blocked at least one AI crawler by the end of last month.

The study reveals that OpenAI’s GPTBot has been blocked 22% of the time across the top 50 news sites, with Bloomberg, Reuters, Business Insider, Washington Post, the New York Times, and CNN as the top names on this list.

CCBot has been blocked about half as often as the GPTBot, with a 10% share across the top 50 news sites. The survey further shows that ChatGPT had been blocked by the Washington Post only, the same as AnthropicAI being blocked by only one website, NewsNow.

Overall, the New York Times, Washington Post, Reuters, and UK’s NewsNow lead in blocking AI crawlers from accessing their content, with each news site blocking two AI bots.

Tags: AI
SendShare148Tweet92
ElvyLewis Ndungu

ElvyLewis Ndungu

Just a guy who loves to code and has a passion for storytelling | Bringing you the latest on all things tech. You can reach me via [email protected] or on Twitter.

Related Posts

Sora

Microsoft Brings Sora AI Video Generator to Bing App

June 4, 2025
google-ai-edge-gallery

Google Quietly Launches App to Run AI Models Locally on Android

June 4, 2025
ConnectedAfrica2025(Day4)-meta-foondamate

Connected Africa 2025 Day 4: FoondaMate and Meta Team Up to Bring AI to Classrooms

May 29, 2025
google-veo-3

Actors and Film Crews Are Worried About Veo 3 Taking Their Jobs

May 29, 2025
AI Africa policies database

New Platform Brings All African AI Policies Under One Database

May 28, 2025
Connected Africa Summit

Connected Africa Summit Calls for Unified Tech Vision

May 28, 2025

Latest

instagram-edits

Instagram Adds Teleprompter to Edits App in Bid to Rival CapCut

June 4, 2025
Sora

Microsoft Brings Sora AI Video Generator to Bing App

June 4, 2025
google-ai-edge-gallery

Google Quietly Launches App to Run AI Models Locally on Android

June 4, 2025
adobe-photoshop-iphone

Adobe Launches Photoshop Beta App for Android

June 4, 2025
Material 3 Expressive

Material 3 Expressive Rolls Out in Gmail and Messages

June 4, 2025
Rose Njeri, the creator of Civic Email

Rose Njeri Charged With Cybercrime for Letting Kenyans Email Their MPs

June 3, 2025

Best devices

budget smartwatches 2025

Best Budget Smartwatches To Buy in Kenya 2025

February 13, 2025

Best Infinix Smartphones To Buy in Kenya 2024

February 13, 2025

Best Laptops for Battery Life in 2024

August 21, 2024

Best “Battery Warrior” Smartphones To Buy in 2024

August 22, 2024

Instagram Adds Teleprompter to Edits App in Bid to Rival CapCut

June 4, 2025

Microsoft Brings Sora AI Video Generator to Bing App

June 4, 2025

Techweez is a fast growing influential source of technology news, reviews and analysis by leading tech geeks in the industry.

Follow Us

Editorials

Actors and Film Crews Are Worried About Veo 3 Taking Their Jobs

Samsung QLED TVs Now Officially Certified for Real Quantum Dot Technology

Trump’s Tariffs Will Be the End of Affordable Tech

5 Ways to Prep Your Tech for Resale

The Weaponization of PDFs: How Cybercriminals Are Exploiting a Trusted Format

Introducing A Brainbox Quiz: Techweez’s Monthly Trivia Night!

More News

Infinix Hot 60 Pro+ Could Be the World’s Thinnest Curved Display Phone

Infinix Note 50 Pro Review: A Budget Phone Shouldn’t Be This Good

Three Billion Chrome Browser Users at Risk from Zero Day Hack

Jumia Gets a Tech Boost as AXIAN Telecom Joins the Table

Your Phone May No Longer Run WhatsApp After This Update

X Unveils XChat with Encrypted Messaging and Calls

  • Terms Of Use
  • Techweez Brand
  • Privacy & Policy
  • Contact Us

© 2024 Techweez - Palahala Media Group may earn a commission when you buy through links on our sites.
A Palahala Media Group Brand. All rights reserved.
.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In

Add New Playlist

Techweez | Tech News, Reviews, Deals, Tips and How To
Crunchy Cookies 🍪 Ahead!

Hey there! Just a heads-up: we're big fans of cookies - both the digital and edible kind! 🍪 We use our cookies and some from third parties to ensure your browsing experience on our site is smooth sailing and secure.

 

But wait, there's more! We also use cookies to gather stats and insights on how you navigate our site. It's like getting a behind-the-scenes peek at your digital adventures!

 

Don't worry, you're in control. You can adjust your cookie settings anytime to suit your preferences. Feeling curious? Dive into our Privacy Policy for all the juicy details. Happy browsing! 🚀

Functional Always active
Listen, this legal stuff is about as exciting as watching paint dry. But it basically says we only use your stuff for what you asked us to do, and nobody else gets to peek!
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
It's those sneaky cookie crumbs websites leave behind to count visitors, like counting ants at a picnic! Totally harmless, just for fun facts. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
Hey there! Just letting you know we use some fancy gizmos to remember your preferences. This way, we can show you ads that are, well, not completely bananas.
Manage options Manage services Manage {vendor_count} vendors Read more about these purposes
Make cookies
{title} {title} {title}
Techweez | Tech News, Reviews, Deals, Tips and How To
Crunchy Cookies 🍪 Ahead!
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional Always active
Listen, this legal stuff is about as exciting as watching paint dry. But it basically says we only use your stuff for what you asked us to do, and nobody else gets to peek!
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
It's those sneaky cookie crumbs websites leave behind to count visitors, like counting ants at a picnic! Totally harmless, just for fun facts. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
Hey there! Just letting you know we use some fancy gizmos to remember your preferences. This way, we can show you ads that are, well, not completely bananas.
Manage options Manage services Manage {vendor_count} vendors Read more about these purposes
Make cookies
{title} {title} {title}
No Result
View All Result
  • News
  • Reviews
  • Features
  • Editorial
  • Automotive
  • Entertainment

© 2024 Techweez - Palahala Media Group may earn a commission when you buy through links on our sites.
A Palahala Media Group Brand. All rights reserved.
.