Scraping Web Application and Bot using Machine Learning and Natural Language Processing

As an ongoing contract, my role involves developing a bespoke scraping infrastructure and UX that manages a bot, for scraping data across the internet to be analysed.

The system is built using some of the latest frameworks and technologies – Bootstrap, React, Tensorflow and Spacy. When a scrape is started, it initialises and executes a job for the bot. The bot will then scrape the data it needs whilst also performing various jobs such as getting the data ready to be stored in the system.

While its scraping, if any problems occur such as captchas, these will be dealt with automatically by the Bot. It will also log these events and send notifications for debugging purposes.

After the data has been scraped, the bot will then run a number of machine learning algorithms, such as classification and image recognition as well as identifying objects required for analysing. With every iteration the machine gets more intelligent. The machine is supervised in the form of controlling the data that it learns from.

Like what you see then, fancy a chat?

Email me Phone me

Testimonials

Dean Wronowski was recommended to me by a friend as I was looking for someone that could incorporate excellent design with all of the back end functionality of a good website. Dean was able to deliver on theses aspects above and beyond what I was expecting. He has a friendly and professional approach and delivers results on or before the requested deadline. No task seems to large for Dean as he leads the field in current and innovating website trends. A wizard of the website world!

Robin Ross - Sandymouth Surf School