Companies that place any form of online content have little choice except to allow scraper robots to access their media: text, images, videos. This seems to be the rule accepted in 2025, and it is a situation that the vast majority of digital marketing experts accepted.
Answers to questions such as “Who offers the best price on the causes of women’s trousers?” It is likely that today an artificial intelligence such as Gemini is responded, positioned above the “traditional” list of the search engine “that the user can make his way manually. Instead of howl with the protest for any perceived injustice of the situation, companies are working hard to optimize their content for a better representation in the responses derived from the large artificial intelligence companies.
Blocking artificial intelligence robots
But not all companies choose to roll with fists and accept the “new normality”. In a recent interview with Business Insider, The CEO of Nextdoor, Nirav Tolia, says that not only its site, Nextdoor.com, Block Bots to scrape its content, has built the company even without optimizing its site for the search for Google. To date, he says: “We have never allowed us to be distributed by our content, we are not rubbed by any of the search engines”.
This is an extreme vision and a decision taken by Toolia on what describes as ideological reasons. As he admits, the refusal to allow AIS access to the content generated by the community on Nextdoor means that it is operational on him and his team to offer users the same level of experience as the customer that users of the search for chatgpt receive, but without using Gatgpt type technology.
But the Nurav Tolia’s approach is not extreme as the responses of other creators of content to the great artificial intelligence engines that scraper the Internet both for learning materials and for updated content to be produced in response to questions.
Based on artificial intelligence companies
At the moment there are several legal causes in which creators and artists have decided to sue and are taking artificial intelligence companies through the courts, supporting the violation of the copyright of their works. More recently than the dozens of existing cases, a group of artists filed against Stability Ai, Midjourney, Runway Ai and Deviantart, supporting the improper use of their works to train the generation models of images AI.
In the largest case of the last few months and the other side of the scale, Nbuniversal has filed against Midjourney, the AI ​​generation generation company, claiming that technology can create unauthorized images based on copyright images, such as those of Disney Pantheon and the world of Star Wars.
In various judicial and judicial battles of the past, the great artificial intelligence managers repeatedly stated that scraping the contents of others fall within the clause of “fair use” of copyright laws, which implies that the materials found online can be considered public domain, while those entities that collect the materials (AIS) do not offer the backup in the form of imitative creations.
The only difference between the legal cases brought by global multinational companies such as Nbuniversal and those smaller class actions issued by Life of Karla OrtizIt’s money. Disney and Lucasarts stop, potentially, to lose more than a person in pure monetary terms (even if a million dollars to the latter means more For the latter) and Hollywood studies have the funds to pay more lawyers longer.
But intellectually, and in the eyes of the law, there is little difference between the smaller class actions and any legal Juggernaut led by large Hollywood studies. It will probably take a decade or more for many legal cases to achieve the judgment and until then the artificial intelligence robots will continue their peaceful work.
Block, please?
The third path that some content creators choose to undertake is to ask artificial intelligence companies not to scrape their properties online. Experience has shown that companies such as Openi and Anthropic ignore the ten -year method of limiting automated external access to websites through a text line in a robot.txt file. Sitting on the root level of the websites, a typical robots.txt file resembles this:
User-agent: *
Disallow: /admin/User-agent: GPTBot
Disallow: / User-agent: Googlebot
Disallow: User-agent: *
Disallow:
Designed to be readable by man, the above example indicates to any web scraper which is ok to read the contents of the site (apart from the administrator section) for the indexing by search engines (Googlebot), but not ok for robots Ai (GPTBOT) to do the same. It is an “gentleman” agreement system that is based on good faith and can be ignored. In most cases, it is actually ignored.
So what can the protective companies of their intellectual property do? To be as Nextdoor, companies need significant technological resources to effectively block access for third parties and in general it is an ongoing process that must be constantly updated and optimized, in ways similar to computer security measures that play a game of cats and mouse with bad actors. In this case, however, the so -called bad actors are the evolving robots that read the web for content and defenders work for companies such as Nextdoor, erecting barriers and firewalls to prevent access.
Anyone who uses the web in these days are getting used to captcha checks, ranging from identification of the images to the Municipality “Click to show that you are not a robot”. These can theoretically be used to prevent robots from accessing a website, but they are so common and known that most sophisticated robots can find their way to get around them.
Block with extreme prejudice
Several thousand websites have distributed More technical tools To guide the Bot to the pass. Asking the device (whether it is a graphic web browser or the bucking scraping of the automated) to solve a puzzle written in the javascript programming language, visitors can be allowed or denied access (the typical scraping robots have no javascript functionality). As you might expect, most of the sites so protected Highly technical content written by technological experts; individuals and organizations with the knowledge of implementing this gateway.
Partner
The final option for content creators is really available only for content creators who attract a huge audience for their work, such as the New York Times, or owners of the Social Media site, Reddit. These are the entities with sufficient “muscles” of content to achieve private agreements with artificial intelligence companies. Based on these partner agreements, companies such as Google, Openai or X receive privileged access levels to the data repositories which in some cases represent decades of content.
In return, the publisher gets a commission, which becomes part of the way they can monetize their content. For news sites, these agreements are helping to fill the gap between paper and digital media, a gap that many publications have fought to cross over the past 20 years.
Adapt
Most marketing professionals have no choice but to undertake a new learning journey, review their old search engine optimization ideas and adopt new strategies to ensure that their messages are placed in front of the right people. There are, ironically, a good number of tools based on artificial intelligence there to help professionals adapt their content so that it can thrive at a time when AIS often proxy between creator and consumer.
By adapting to the changing paradigm, marketing experts will have to find new ways to create impact content. It is worth knowing how the AIS work and the different ways in which large companies have messages in environments generated by the AI. Armando themselves with a specialist knowledge of how llm and ais multimodal, companies can differentiate themselves from their competitors. If marketing experts work on the assumption sure that colleagues who work elsewhere have access to the same tools they make, using a wider view of their internal mechanisms will create marketing methods that are a cut above the rest.
Conclusions
The choices for content creators are blocks, adaptations, its or partners. For small companies that want to make an impression in the digital environment, the only real choice is to adapt. Blocking, suits or collaboration are out of most organizations or are too demanding to implement easily.
(Source of the image: “Let’s make a good team …” by and Yourdon is under license in CC by-NC-SA 2.0.)
See also: Meta, Tiktok faces a legal challenge for half a billion dollars in Brazil for minor protection
Attending 2025 Digital Marketing World Forum in London? Visit us at stand 238 or captures one of our creator’s economic language sessions.
Or you can Direct to the website to register And join us on 24-25 June in Olympia, London.
Discover more from Gautam Kalal
Subscribe to get the latest posts sent to your email.
Be First to Comment