Luxxle Web Crawler
Introduction
We use web crawlers to improve our index and to provide the best possible service to our users.
"Crawlers" are programs, also known as robots or spiders, that are used to scan websites by finding and following links from one webpage to another. Luxxle's primary crawler is called Luxxlebot, and this page contains information about what our crawlers do and how they interact on other websites.
"Crawlers" are programs, also known as robots or spiders, that are used to scan websites by finding and following links from one webpage to another. Luxxle's primary crawler is called Luxxlebot, and this page contains information about what our crawlers do and how they interact on other websites.
User-agent
When our crawlers visit a website, they identify themselves with various user-agents which contain information about the crawler and its specific version. Our crawlers are always identifiable in a website's referrer logs by the names Luxxlebot or Luxxlebot-news. They should be specified in robots.txt, the robots meta tags, and the X-Robots-Tag HTTP directives with these names.
The Luxxlebot user agent is as follows:
The Luxxlebot user agent is as follows:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Luxxlebot/1.01; +https://www.luxxle.net/bot) Chrome/116.0.5845.96 Safari/537.36
Crawler Verification
How to check to see if a web crawler from Luxxle is accessing your server:
- Do a DNS lookup on the IP address.
- Check to see if the domain name contains "luxxle.net".
- Do a forward lookup on the previous domain name.
- Check to see if the IP address matches the original one.
Questions?
If there was a problem when our crawlers visited your site, we're extremely sorry for any inconvenience.
Please contact us with any problem caused by the crawlers through our contact page.