NuGet Package RobotsTxt : A robots.txt parser for .NET

A robots.txt parser for .NET

Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.

See https://bitbucket.org/cagdas/robotstxt for usage examples.

Info

Version: 2014.02.19
Author(s): Çağdaş Tekin
Last Update: Wednesday, February 19, 2014
.NET Fiddle: Create the first Fiddle
Project Url: https://bitbucket.org/cagdas/robotstxt
NuGet Url: https://www.nuget.org/packages/RobotsTxt


Install
Install-Package RobotsTxt
dotnet add package RobotsTxt
paket add RobotsTxt
RobotsTxt Download (Unzip the "nupkg" after downloading)



Tags



STATS

must-have-score

1.3

avg-downloads-per-day

10

days-since-last-release

2239