• Latest
  • All
  • How To
Web Crawler

News Sites Crackdown on AI Crawlers

September 20, 2023
How to Secure Your Phone and Yourself During Protests

How to Secure Your Phone (and Yourself) During Protests

June 25, 2025
xiaomi

Xiaomi June 26 Mega Launch: From EVs to Foldables

June 24, 2025
All-Screen-iPhone-2027-Apple

Apple Reportedly Working on an All-Screen iPhone

June 24, 2025
Unified Payment Interface

CBK’s New Fast Payment System Plan Could Rival M-Pesa

June 24, 2025
DHgate Tablet Cases deals
Starlink

Starlink Resumes Sign-Ups in Nairobi After 7-Month Freeze

June 24, 2025
ASUS ROG Flow Z13

ASUS ROG Flow Z13 Proves Portability and Power Can Coexist

June 24, 2025
Infinix InBook Air

Infinix InBook Air Review: Perfect Laptop for Students and Simple Workflows

spotify-mobile

Spotify Lossless HiFi Update Nears Reality After Years of Delay

June 23, 2025
whatsapp

WhatsApp Is Testing a New AI Tool To Help Users Write Texts

June 23, 2025
SHIF Status

How to Check Your SHIF Status in Kenya: A Complete Guide

June 23, 2025
youtube google ai

Google Finally Admits To Using YouTube Videos To Train Its AI Models

June 23, 2025
Microsoft AI for Good Lab tool GIRAFFE

How Microsoft’s AI for Good Lab Is Helping Save Giraffes from Extinction

June 20, 2025
Techweez | Tech News, Reviews, Deals, Tips and How To
  • News
  • Entertainment
  • Reviews
  • Features
  • Editorial
No Result
View All Result
Techweez | Tech News, Reviews, Deals, Tips and How To
  • News
  • Entertainment
  • Reviews
  • Features
  • Editorial
No Result
View All Result
Techweez | Tech News, Reviews, Deals, Tips and How To
No Result
View All Result

News Sites Crackdown on AI Crawlers

ElvyLewis Ndungu by ElvyLewis Ndungu
September 20, 2023
in News
Reading Time: 3 mins read
252
0
Web Crawler

In recent months, there has been an explosion of generative Artificial Intelligence (AI) as more companies continue to develop consumer-facing tools to help automate tasks, write documents, do research on various topics or even basic coding.

OpenAI’s ChatGPT took the world by storm following its release, having reached an estimated 100 million users just within two months after launch. Arguably, following its success more consumers and companies have continued to leverage generative AI models and tools in their daily tasks.

Since then, more generative AI tools have been released by other key tech players as well as smaller startups including Google’s conversational chatbot, Bard, Microsoft’s Bing chat, and OpenAI’s text-to-image generator, DALL·E 2.

However, the rise of large language models (LLMs) and generative AI have ushered new in challenges, bringing to light copyright issues. This has resulted in pushback from news sites, publishers, and intellectual property holders who see their data being collected by AI crawlers. With no clear regulatory rules controlling AI’s use of copyrighted material yet, some of the world`s largest news websites have taken matters into their own hands.

According to data presented by AltIndex.com, nearly one-third of the world’s top 50 news sites have blocked AI crawlers from accessing their content, and their number continues rising.

Notably, CNN, the New York Times, the Daily Mail, Reuters, and Bloomberg Have All Blocked At least One AI Crawler.

Crackdown on AI crawlers

AI companies send crawlers to collect data to train their models and provide information for chatbots.

However, as data is one of their core advantages, many of the world’s largest news websites have become extremely cautious, especially since there is generally no upside to handing over their data to AI crawlers, according to AltIndex.

Last month OpenAI launched its GPTBot crawler to collect data to enhance its language models. This escalated the situation further despite assurances that paywalled content would be excluded from websites. Several high-profile news sites, including CNN, Reuters, and the New York Times, blocked GPTBot.

According to a Kirwan Digital Marketing Agency survey, 28% of the top 50 news sites worldwide have blocked at least one AI crawler by the end of last month.

The study reveals that OpenAI’s GPTBot has been blocked 22% of the time across the top 50 news sites, with Bloomberg, Reuters, Business Insider, Washington Post, the New York Times, and CNN as the top names on this list.

CCBot has been blocked about half as often as the GPTBot, with a 10% share across the top 50 news sites. The survey further shows that ChatGPT had been blocked by the Washington Post only, the same as AnthropicAI being blocked by only one website, NewsNow.

Overall, the New York Times, Washington Post, Reuters, and UK’s NewsNow lead in blocking AI crawlers from accessing their content, with each news site blocking two AI bots.

Tags: AI
SendShare148Tweet92
ElvyLewis Ndungu

ElvyLewis Ndungu

Just a guy who loves to code and has a passion for storytelling | Bringing you the latest on all things tech. You can reach me via [email protected] or on Twitter.

Related Posts

whatsapp

WhatsApp Is Testing a New AI Tool To Help Users Write Texts

June 23, 2025
youtube google ai

Google Finally Admits To Using YouTube Videos To Train Its AI Models

June 23, 2025
Microsoft AI for Good Lab tool GIRAFFE

How Microsoft’s AI for Good Lab Is Helping Save Giraffes from Extinction

June 20, 2025
Aigov

U.S. Plans to Launch AI Hub for Government Agencies

June 16, 2025
Kenya-KICTANet-MindHYVE-ai-

Kenya Partners with US AI Firms to Co-Create National AI Policy with KICTANet

June 12, 2025
2025 Afrilabs Annual Gathering

AfriLabs Annual Event Returns to Nairobi With Big Plans for Tech Scene

June 10, 2025

Latest

How to Secure Your Phone and Yourself During Protests

How to Secure Your Phone (and Yourself) During Protests

June 25, 2025
xiaomi

Xiaomi June 26 Mega Launch: From EVs to Foldables

June 24, 2025
All-Screen-iPhone-2027-Apple

Apple Reportedly Working on an All-Screen iPhone

June 24, 2025
Unified Payment Interface

CBK’s New Fast Payment System Plan Could Rival M-Pesa

June 24, 2025
Starlink

Starlink Resumes Sign-Ups in Nairobi After 7-Month Freeze

June 24, 2025
ASUS ROG Flow Z13

ASUS ROG Flow Z13 Proves Portability and Power Can Coexist

June 24, 2025

Best devices

budget smartwatches 2025

Best Budget Smartwatches To Buy in Kenya 2025

February 13, 2025

Best Infinix Smartphones To Buy in Kenya 2024

February 13, 2025

Best Laptops for Battery Life in 2024

August 21, 2024

Best “Battery Warrior” Smartphones To Buy in 2024

August 22, 2024

How to Secure Your Phone (and Yourself) During Protests

June 25, 2025

Xiaomi June 26 Mega Launch: From EVs to Foldables

June 24, 2025

Techweez is a fast growing influential source of technology news, reviews and analysis by leading tech geeks in the industry.

Follow Us

Editorials

Abductions and Arrests! Kenyan Government’s Fear and Hate of X Users Makes No Sense

Actors and Film Crews Are Worried About Veo 3 Taking Their Jobs

Samsung QLED TVs Now Officially Certified for Real Quantum Dot Technology

Trump’s Tariffs Will Be the End of Affordable Tech

5 Ways to Prep Your Tech for Resale

The Weaponization of PDFs: How Cybercriminals Are Exploiting a Trusted Format

More News

Infinix InBook Air Review: Perfect Laptop for Students and Simple Workflows

Spotify Lossless HiFi Update Nears Reality After Years of Delay

WhatsApp Is Testing a New AI Tool To Help Users Write Texts

How to Check Your SHIF Status in Kenya: A Complete Guide

Google Finally Admits To Using YouTube Videos To Train Its AI Models

How Microsoft’s AI for Good Lab Is Helping Save Giraffes from Extinction

  • Terms Of Use
  • Techweez Brand
  • Privacy & Policy
  • Contact Us

© 2024 Techweez - Palahala Media Group may earn a commission when you buy through links on our sites.
A Palahala Media Group Brand. All rights reserved.
.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In

Add New Playlist

Techweez | Tech News, Reviews, Deals, Tips and How To
Crunchy Cookies 🍪 Ahead!

Hey there! Just a heads-up: we're big fans of cookies - both the digital and edible kind! 🍪 We use our cookies and some from third parties to ensure your browsing experience on our site is smooth sailing and secure.

 

But wait, there's more! We also use cookies to gather stats and insights on how you navigate our site. It's like getting a behind-the-scenes peek at your digital adventures!

 

Don't worry, you're in control. You can adjust your cookie settings anytime to suit your preferences. Feeling curious? Dive into our Privacy Policy for all the juicy details. Happy browsing! 🚀

Functional Always active
Listen, this legal stuff is about as exciting as watching paint dry. But it basically says we only use your stuff for what you asked us to do, and nobody else gets to peek!
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
It's those sneaky cookie crumbs websites leave behind to count visitors, like counting ants at a picnic! Totally harmless, just for fun facts. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
Hey there! Just letting you know we use some fancy gizmos to remember your preferences. This way, we can show you ads that are, well, not completely bananas.
Manage options Manage services Manage {vendor_count} vendors Read more about these purposes
Make cookies
{title} {title} {title}
Techweez | Tech News, Reviews, Deals, Tips and How To
Crunchy Cookies 🍪 Ahead!
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional Always active
Listen, this legal stuff is about as exciting as watching paint dry. But it basically says we only use your stuff for what you asked us to do, and nobody else gets to peek!
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
It's those sneaky cookie crumbs websites leave behind to count visitors, like counting ants at a picnic! Totally harmless, just for fun facts. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
Hey there! Just letting you know we use some fancy gizmos to remember your preferences. This way, we can show you ads that are, well, not completely bananas.
Manage options Manage services Manage {vendor_count} vendors Read more about these purposes
Make cookies
{title} {title} {title}
No Result
View All Result
  • News
  • Reviews
  • Features
  • Editorial
  • Automotive
  • Entertainment

© 2024 Techweez - Palahala Media Group may earn a commission when you buy through links on our sites.
A Palahala Media Group Brand. All rights reserved.
.