Scraping Web Application and Bot using Machine Learning and Natural Language Processing

As an ongoing contract, my role involves developing a bespoke scraping infrastructure and UX that manages a bot, for scraping data across the internet to be analysed.

The system is built using some of the latest frameworks and technologies – Bootstrap, React, Tensorflow and Spacy. When a scrape is started, it initialises and executes a job for the bot. The bot will then scrape the data it needs whilst also performing various jobs such as getting the data ready to be stored in the system.

While its scraping, if any problems occur such as captchas, these will be dealt with automatically by the Bot. It will also log these events and send notifications for debugging purposes.

After the data has been scraped, the bot will then run a number of machine learning algorithms, such as classification and image recognition as well as identifying objects required for analysing. With every iteration the machine gets more intelligent. The machine is supervised in the form of controlling the data that it learns from.

Like what you see then, fancy a chat?

Email me Phone me

Testimonials

We have been working with Dean for several years and we could not be happier with his service and help. Being a small business without much time to sort out our website needs we really appreciate dealing with someone who has taken the time to understand what we do and how this relates to the crazy ideas we sometimes have regarding our website and how it it used. Dean has given good advice and when things need sorting out he is very prompt and the work gets done very quickly. We recommend Dean’s services to everyone.

Simon Hammond - Shoreline Extreme Sports