found 179 packages in 13ms

robots.txt4.8kNew BSD1.1.0

robots.txt middleware for express/connect to server up your robots.txt

tomgcoabout 3 years ago

A specification compliant robots.txt parser with wildcard (*) matching support.

samclarkeover 2 years agorobots.txt, parser, user-agent, scraper

Awesome generator robots.txt

itgalaxyabout 3 years agorobotstxt, robots.txt, generate, robots-txt

Gatsby plugin that automatically creates robots.txt for your site

mdreizinover 2 years agogatsby, gatsby-plugin

sitemapper135.8kMIT4.0.2

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

seantomburkeabout 2 months agorobots.txt, parse, sitemap, xml

Packages

Lightweight debugging utility for Node.js and the browser

Lodash modular utilities.

React is a JavaScript library for building user interfaces.

A querystring parser that supports nesting and arrays, with a depth limit

Higher-order functions and common patterns for asynchronous code

Babel compiler core.