Today i wrote my first todo list as to simplify the building process and have a clear and descriptive plan showing exactly what I am to do and how i should do it. You can see it on the homepage. It will be updated as we progress throughout this project.
What I actually did today
I decided to go with Go (formerly Golang) as the programming language for the project.
The first step in building Aranea is to decide on a web scraper. If I were building with Python I would have gone with the commonly used BeautifulSoup, but as I’m writing in Go I had to do some research on what to use. I mostly came accross goquery, which is based on JavaScripts popular JQuery. I am not at all familiar with JQuery so if I’m gonna decide to use this I’ll have some learning to do. Goquery also lists several related projects out of whom gocolly seemed to be the most useful for me. It is a fast and lightweight web scraper and crawler that will allow you to easily scrape all new links found in the current page. This does seem to be an interesting option for Aranea.