Aranea is (going to be) a retro-themed search engine with its own web crawler and indexing system.
Written in go for several reasons:
- it’s fast. Although maybe not the fastest language it is a very nice option.
- It’s built for concurrency. A very important aspect of web scraping is being able to handle lots of urls at once.
- It compiles to a single binary. No special installations needed for end users, the executable includes all dependencies.
- It is a small language. Occam’s razor, less is more.
- It’s modern, maintained and has a lot of libraries.
to do:
-
simple site scraper using goquery
-
fetch site
-
extract description
-
extract title
-
extract links
- internal
- external
-