At its core, Openthirdeye is not just another news aggregator. It’s a platform that leverages state-of-the-art AI technology to transcend conventional news curation methodologies. Our AI algorithm goes beyond mere keyword matching, diving deep into the intricacies of information quality and relevance. By considering a multifaceted spectrum of factors, Openthirdeye ensures that users receive a curated selection of news items that reflect the most pertinent and trending topics of the day.
The algorithm examines the relationships between various news items to surface a comprehensive perspective on a given topic. This feature provides users with a more holistic view of complex stories.
Openthirdeye firmly believes in transparency. Unlike many news platforms, we provide our users with complete visibility into the AI-driven curation process. For every article featured on our homepage, we offer a detailed breakdown of the factors that influenced its selection. This unique feature empowers users to critically analyse the credibility of the information they’re consuming, fostering a culture of informed consumption.
>> Visit Website
As part of a new contract I recently completed, I built a custom time entry management system for the company. The system was designed to manage the time from employees within the company so that time could be tracked against projects and wages could be calculated. Each time entry can be assigned against each project, and each user has a role that determines what they are able to do on the platform.
The system had features like user roles allowing admins to extract time entries in a CSV format and make changes to those entries if needed. The dashboard showed graphs and charts showing how many hours were spent on each project over time.
The system was built using the microservices architecture. This meant that each component of the application could be deployed separately and scaled independently. It also made it easier to maintain the codebase, as changes could be made to a single service without affecting any other part of the application.
It was built using React, Material UI, Kubernetes and RDS database.
As apart of one of my contracts, I was tasked with developing a bespoke website scraping system allowing the internal team to extract reviews and attributes from websites. The system was designed to be highly scalable, allowing the team to scale up the selenium workers that were needed to scrape a given website. The system also included a dashboard allowing the team to view the status of each scrape, as well as monitor the project status.
During production, it was able to scrape over 100k comments in less than a few hours, and allowed the team to scale up as needed to scrape large websites quickly and efficiently. When prompted with Bot detection technologies, the system was able to overcome these protections from websites by using a combination of Selenium and PhantomJS in order to scrape the website, combined with IP switching.
The system was able to overcome the following website protections:
-CAPTCHA
-Captcha2K (XEvil)
-ReCaptcha v1 and v2
-Multilayer Captcha
A desktop application was also developed using Electron where websites that provided significant Bot protection, humans were able to extract the reviews from websites with the click of the mouse . The desktop application was able to work as an extension of the web browser as well (in the chrome browser). It allowed users to extract reviews, manage their projects and monitor the status of their project.
The team needed a way of managing the data once it was extracted, so I developed some middleware so that any data was passed into their internal pipeline.
The system was built using Laravel, VueJS, Python, Selenium and Electron. It was hosted on Kubernetes infrastructure, with Redis used as a cache layer.
As a software developer in Cornwall, with an ever-growing client base, it was time for me to build a software solution that could help to automate my business internally. The requirement for the system was that it had to be integrated seamlessly with various API’s and automatically process the data by triggering certain actions.
At scheduled times, the systems connect to various API’s to retrieve and download the latest transactions, processing them using a custom rule-based system. This results in the various expenses being generated and assigned to certain projects and budgets automatically. This has already saved a large percentage of time.
The sending and receiving of emails, are also processed into the system and assigned to clients. A process is then triggered that analyses and profiles the sentiment and tone of the messages. This allows the system to generate a profile of whether the client is positive or neutral with their communications, combined with information such as the duration of time it takes for a client to pay their invoices.
This system is already saving me large amounts of time in various areas, from client management to managing invoices and projects. If you are looking for a software development in Cornwall, then feel free to contact me to discuss your software idea. Click here to find out how to contact me.
Mark from Bude Tourist Information Centre approached me to develop a bespoke business system to automate the booking process for the Canal Suite meeting room.
My starting point was to document a clear understanding of this projects requirements, which included a visual solution for the application’s branding and the user interface. This was then reviewed by Mark, with subsequent adjustments made based on his feedback.
It was then simply a matter of breaking the project down into weekly sprints, with the purpose of focusing on features that would benefit the users of the system. As soon as each new system feature was developed it was deployed to the live system, allowing users to benefit from the latest features as soon as possible
Overall the project involved setting up the system’s infrastructure, which consisted of:
- Code repositories
- A Development Environment
- The Development of an API Backend built using Laravel and VueJS for the Frontend Development
- Production Server Configuration – using a redis database for the queue system and a mysql database for the data storage
- Clear deployment processes with version control
Built to help automate and monitor online communities mentioning certain keywords in realtime, a bespoke system called Radar, uses the latest technologies in machine learning, natural language processing, laravel combined with VueJS.
The system communicates with a series of Radar scanners (raspberry pi) which load selected websites in a browser, which is controlled via a bespoke plugin that periodically checks for the given keywords.
If a match is found, an alert is sent back to the system. From here, the system will perform some cleaning tasks; such as encoding, filtering out rude words and discarding posts containing duplicates or retweets. The next task is to broadcast the new alert to users logged in. If e-mail alerts have been selected then an email will be sent, notifying the user of the new alert. Similarly, the alert will be sent to the chosen API if the option is selected.
A number of scheduled tasks also run periodically; one of the priority tasks involves performing predictions on whether the alert is interesting based on whether previous alerts of a similar ilk have been marked as interesting. Another task performs sentiment analysis which rates the emotion of the alert to detect whether it is negative or positive. One of the last tasks checks the system performs, is to define whether any of the new alerts are similar to any of the last 100 alerts – this is useful for detecting trends.
* There are a number of tasks which haven’t been mentioned to protect the intelligence of the system
If you’re looking for bespoke systems to automate your business processes, then please feel free to use my contact page here.
As apart of one of my contracts, I was tasked with developing a bespoke project management system allowing internal and external clients to view and update the status of a project at any given time.
The system was built using the very latest technologies at the time of writing this article. These were: React, which is an open-source framework, built by Facebook, for rendering the interface and managing the components; Redux which is another open-source framework for managing the the state of the application and allowing actions to be distributed throughout the system, Socket.IO , another open-source framework that creates and manages a persistent connection between a client and server, yielding real-time interactions between the user and the system.
Another requirement of the system was that it had to interact with another system (also built by me) for executing an internal process. This requirement was achieved by allowing both systems to talk to each other using APIs.
Furthermore, Bitbucket was used throughout the project, so that a log of each iteration with the system could be recorded and the repository could be cloned for use by additional developers as the system expands. Lastly, Webpack was used so that resources and assets could be compressed and various other scripts executed upon compiling for live production.
If your looking for a bespoke software solution then feel free to contact me with your requirements by going to this page.
The Falcon Hotel approached me to develop an interface for a virtual concierge in their reception area.
The system works by updating itself every night from various sources. The entire ‘Bude Activities’ section is retrieved from the Bude Tourist Information website every night. If any pictures are added or removed, these changes will be automatically updated on the virtual concierge. The same is true of any other content modifications.
Any events that are added to The Falcon Hotel website, are also automatically transferred to the Virtual Concierge every night.
If your looking for a bespoke software solution then feel free to contact me with your requirements by going to this page.
As an ongoing contract, my role involves developing a bespoke scraping infrastructure and UX that manages a bot, for scraping data across the internet to be analysed.
The system is built using some of the latest frameworks and technologies – Bootstrap, React, Tensorflow and Spacy. When a scrape is started, it initialises and executes a job for the bot. The bot will then scrape the data it needs whilst also performing various jobs such as getting the data ready to be stored in the system.
While its scraping, if any problems occur such as captchas, these will be dealt with automatically by the Bot. It will also log these events and send notifications for debugging purposes.
After the data has been scraped, the bot will then run a number of machine learning algorithms, such as classification and image recognition as well as identifying objects required for analysing. With every iteration the machine gets more intelligent. The machine is supervised in the form of controlling the data that it learns from.
Developing automatic website scrapers can sometimes be challenging due to how modern websites are built. Some of these challenges are the continuous layout changes, captchas, bot patterns, paginations, being anonymous, logins and websites that populate new content by continuously scrolling.
I was tasked with developing a Google Chrome Extension that would help with speeding up manual scraping. The biggest benefit of developing a plugin is that the user can load a website as if they are browsing it normally which reduces all the challenges automated scraping is faced with.
The plugin was built in such a way that it allows the user to click anywhere in the page and automatically extract the text. The text is then passed through to a system which is then translated, cleaned and mapped to certain fields in the system. A summary is also visible showing the scraping progress.
