Esprima-compatible implementation of the Mozilla JS Parser API
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
A specification compliant robots.txt parser with wildcard (*) matching support.
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.