In an effort to safeguard its data, X has introduced stringent terms of service, completely banning scraping and crawling without “prior written consent.” This marks a notable shift from the previous terms, which allowed crawling as per the instructions contained in the robots.txt file.
Crawling Restrictions and the Google Connection
Over the past few months, X has modified its robots.txt file, limiting access to crawling bots. Notably, it removed instructions for all crawling bots except Google. This move raises questions about the nature and terms of the agreement between Twitter and Google, which dates back to 2015 when Twitter allowed its tweets to appear in search results. Both companies have remained silent on this matter.
What the Robots.txt File Blocks
The updated robots.txt file now restricts crawlers from accessing information related to likes and retweets on specific messages. Additionally, search robots are barred from viewing user account likes, media files, and photos.
Elon Musk’s Opposition and Legal Actions
Elon Musk has been a vocal opponent of companies collecting X data for training AI models. In April, he threatened to sue Microsoft for allegedly using social network data for AI model training. In July, he initiated a lawsuit against several undisclosed companies on similar grounds.
Privacy Policy Changes
In recent developments, X modified its privacy policy to include the use of publicly available data for AI model training, notes NIX Solutions. Musk’s venture, xAI, which he founded in July, announced its intent to utilize public data, including X posts, for this purpose. The updated privacy policy also encompasses provisions regarding the collection of users’ biometric data, education, and work history.