Top 20 NuGet robots Packages

Sitemap and robots dynamic generator core classes
Sitemap and robots dynamic generator with gadget and admin plugins for EPiserver
robots.txt middleware for .NET core
SimpleSitemap is a lite library that helps you create web sitemaps for collections or lists of items. These sitemaps follow the Sitemap Protocol. Both sitemapindex and urlset links are generated, based upon the data collection size and 'page size'. Examples of this could be a list of your users, ...
Robots.txt parsing and querying utility
The Robots Handler package provides editors with the ability to dynamically change the contents of a site's robots file. Instead of storing the contents of a robots file on the file system, an editor can specify its contents in an Umbraco content page. The property content is then served via a Http ...
IDeliverable.Seo is module for the Orchard CMS that allows site owners robust and granular control over SEO aspects. The module lets site owners control page titles, meta keywords and descriptions, robots.txt and sitemap.xml. Additionally, it allows 301/302 redirects to be configured for changed URL...
Configurable robots.txt handler supporting web.config based configuration. Icon supplied by Freepik (http://www.flaticon.com/authors/freepik)
Fornece todos os recursos necessários para o spidering de aplicações web.
AngleSharp IHtmlParser Extension for Mechanize.NET
With RoboDK API allows you to interact with industrial robots and RoboDK Software. RoboDK Software allows you to simulate and program industrial robots for any manufacturing operation, such as robot machining, inspection or pick and place applications. You can esily program and debug any industrial ...
Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.
Sitecore Multisite Http module, allows mulitsite handling of 404, & 500 errors, along with custom robots.txt generation
robots.text implementation for EPiServer