robots-parser5.7mMIT3.0.1A specification compliant robots.txt parser with wildcard (*) matching support.
samclarkeIl y a over 2 yearsrobots.txt, parser, user-agent, scraper generate-robotstxt487.1kMIT8.0.3Awesome generator robots.txt
itgalaxyIl y a about 3 yearsrobotstxt, robots.txt, robots-txt, generate sitemapper134.1kMIT4.0.2Parser for XML Sitemaps to be used with Robots.txt and web crawlers
seantomburkeIl y a 2 monthsrobots.txt, parse, sitemap, xml express-robots-txt117.1kMIT1.0.0Express middleware to serve and generate robots.txt
modoscIl y a about 3 yearsrobots.txt, express, robots, robots