crawler18kMIT2.0.2Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.
bda-research6 months agocrawler, javascript, spider, scraper @nodelib/fs.walk247.4mMIT3.0.1A library for efficiently walking a directory recursively
nodelib12 months agocrawler, NodeLib, fs, FileSystem fdir169.4mMIT6.5.0The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s
thecodrr3 months agocrawler, util, os, sys ua-parser-js76.3mAGPL-3.0-or-later2.0.6Detect Browser, Engine, OS, CPU, and Device type/model from User-Agent & Client Hints data. Supports browser & node.js environment
faisalman7 days agocrawler-detection, user-agent, client-hints, browser isbot9.8mUnlicense5.1.32🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
omrilotan17 days agocrawlers, bot, spiders, googlebot