Scraping Web Application and Bot using Machine Learning and Natural Language Processing

As an ongoing contract, my role involves developing a bespoke scraping infrastructure and UX that manages a bot, for scraping data across the internet to be analysed.

The system is built using some of the latest frameworks and technologies – Bootstrap, React, Tensorflow and Spacy. When a scrape is started, it initialises and executes a job for the bot. The bot will then scrape the data it needs whilst also performing various jobs such as getting the data ready to be stored in the system.

While its scraping, if any problems occur such as captchas, these will be dealt with automatically by the Bot. It will also log these events and send notifications for debugging purposes.

After the data has been scraped, the bot will then run a number of machine learning algorithms, such as classification and image recognition as well as identifying objects required for analysing. With every iteration the machine gets more intelligent. The machine is supervised in the form of controlling the data that it learns from.

Like what you see then, fancy a chat?

Email me Phone me

Testimonials

You have indeed outstripped all my requests and I thank you very much…I may truly say that I have noticed the variety of your portfolio with a great interest. You are, as far as I am able to form an opinion, in an eminent sense, and in a degree with which few can compete, a most talented web designer. I should myself long ago have carried to you my custom as I am most impressed with the new web page you have created for me.
Dominik - DP Resolutions