Golang Web Scraper
Colly provides a clean interface to write any kind of crawler/scraper/spider
- Pholcus - Pholcus is a distributed, high concurrency and powerful web crawler software. Gospider - An flexible, modular and expansible Go concurrent Crawler (spider) framework. Ants-go - A distributed, restful crawler engine in golang.
- In order to keep this short, a web crawler is a bot that can browse the web so a search engine like google can index new websites and a web scraper is responsible of extract the data from that website.
With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
Scraping the Web in Golang with Colly and Goquery March 1, 2018. 9 minutes If told to write a web crawler, the tools at the top of my mind would be Python based: BeautifulSoup or Scrapy. However, the ecosystem for writing web scrapers and crawlers in Go is quite robust.
Features
- Clean API
- Fast (>1k request/sec on a single core)
- Manages request delays and maximum concurrency per domain
- Automatic cookie and session handling
- Sync/async/parallel scraping
- Distributed scraping
- Caching
- Automatic encoding of non-unicode responses
- Robots.txt support
- Google App Engine support
Batteries included
Go Web Crawler
Colly comes with all the tools you need for scraping. The war of mine for mac.
Open Source
Golang Web Scraper
Development of Colly is community driven and public. Need for speed underground mac os torrent.