Top 20 NuGet crawler Packages

A ASP.NET Core Lib to create automatic searchs more easy

A minimal multithreaded web crawler

Crawler for main social networks

A Simple and Powerful Library to Deal with Web Robots Control Strategy.

AppStoresScraper is library for downloading app metadata from Steam, Google, Apple and Windows app stores

It helps you to use HAP in easier and meaningful way via Reflection. It works somehow like Entity-Framework. Go to wiki in github page for tutorial : https://github.com/parsalotfy/HtmlAgilityPack_Helper/wiki

A .Net Web Crawler

Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.

Simple web crawler. Discover website resources and sitemap. Supports .net framework, .net standard and .net core.

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

A fake agent random generater for cralwers.

Crawl and index your (static) asp.net website for searching using an MSBuild target. Simple search query interface. Uses Lucene.Net and NCrawler.

Library allows you to extract different data from text or web sources, analyze text content, extract different information and write your own data mining & extracting applications and services.

FSharp Web Crawler

The crawling/indexing component of the Sitecore Search Contrib project

The searching/querying component of the Sitecore Search Contrib project