With scraping data from the internet becoming more difficult and with new technologies being developed at a faster rate than ever before, I decided to build a new system (for personal use) called Prober, which designed to overcome the difficulties of scraping.
Prober scrapes websites (if robots.txt allow it to do so) without code or the use of xPaths. With websites constantly changing results in xPaths requiring changes, despite the fact that often, visually, nothing too drastic changes. Prober can predict visually where certain elements are placed on a page so that you don’t need need to update the xPaths. The system is trained, via visual annotation, on what an element looks like within a webpage; so doesn’t matter if a site’s layout slightly changes. Also It does not interact with the DOM at all which is what most scrapers rely on at the moment.
Prober is still in development but I can see the benefits and I’m loving it!
Locrafts is all about generating a buzz for local creativities using machine learning. It finds local gift makers and designers situated in the United Kingdom. Everything is pulled together, analysed and displayed in a single portal/website where the designers and their gifts are then displayed. The results can be filtered by the colours of the gift products and the location of the designers.
There are many factors behind Locrafts decision making process. Some of these include, analysing what you post on your website and what you share on your social media accounts (if api’s are permitted). Other factors are hidden for obvious reasons but the algorithm takes into account a lot of factors; it may decide against displaying some authors and their gifts on the platform. Therefore some of the gifts discovered may never reach the website in its attempt to reduce any spam gifts that could be posted and shared on the platform.
The system is built using cutting edge technologies: AWS is used for hosting the machine learning models, React is used for the front end interface and NodeJS is used for the backend api configuration.
More updates to come soon!
