Portfolio

Jun 16 2010

News Crawling Bot Development in Bude, Cornwall

Crawling News Bot built using C# and XMLRPC in Bude, Cornwall

The software agent works by crawling a list of websites, sent to the software agent periodically, which then parses the page and retrieves all the headlines and then submits it to the server, in this case is at www.thethirdeye.org.

The server then uses a small algorithm for populating the news based on a number of factors, which are:

- Date of when the headline was published.

- The website from which the article was posted on.

- How often the publishers website has occurred on the homepage of thethirdeye.

- The amount of associated Twitter discussions, Facebook Discussions.

- How often a headline has appeared on the internet.