Info
| Version: | 2.0.0 |
| Author(s): | Nicholas Bergesen |
| Last Update: | Wednesday, December 22, 2021 |
| .NET Fiddle: | Create the first Fiddle |
| NuGet Url: | https://www.nuget.org/packages/RobotsSharpParser |
Install
Install-Package RobotsSharpParser
dotnet add package RobotsSharpParser
paket add RobotsSharpParser
RobotsSharpParser Download (Unzip the "nupkg" after downloading)
Key Features:
- Parse robots.txt into Typed object.
- Lookup Allowed/Disallowed/Crawldelay based on User-Agent.
- Traverse sitemap in robots.txt for urls.
For More info see: https://github.com/nicholasbergesen/robotsSharp/master/README.md.