ast-types118.4mMIT0.14.2Esprima-compatible implementation of the Mozilla JS Parser API
benjamnabout 2 years agospidermonkey, ast, abstract syntax tree, hierarchy isbot4.9mUnlicense5.1.28🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
omrilotanabout 2 months agospiders, bot, crawlers, googlebot robots-parser5.5mMIT3.0.1A specification compliant robots.txt parser with wildcard (*) matching support.
samclarkeover 2 years agospider, robots.txt, parser, user-agent simplecrawler214.8kBSD-2-Clause1.1.9Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
simplecrawlerabout 3 years agospider, simple, crawler, cache shift-parser135.4kApache-2.08.0.0ECMAScript parser that produces a Shift format AST
shapesecurityabout 3 years agoSpiderMonkey, spider, Shift, AST