1. Home
  2. Usecases Catalog
  3. AI Modeling and Evaluation

Artificial Intelligence - Modelling and Evaluation

Overview

There are many types of AI learning and there have been numerous attempts to classify techniques in major genres. The list is an ever evolving one and can be derived from different taxonomy attempts depending on the view one has looking at the different algorithms. During the Modelling and Evaluation phase of an AI pipeline the Machine Learning Engineer explores a taxonomy of AI cores  that can be used when tackling different AI business cases.

Supervised learning

Any type of learning that involves historical knowledge of the underlying target variables that map business objectives can be regarded as supervised learning. What this means is that before training the AI cores the machine learning engineer has  a dataset that models the way historically the system has behaved and can effectively ask the AI cores to optimise decision performance based on that prior knowledge.

List of AI cores that can apply in this learning genre includes:

  • Neural Networks
  • Deep Neural Networks and their derivatives
  • Decision Trees and all their derivatives
  • Statistical techniques

Unsupervised learning

In unsupervised learning the machine learning engineer, has  no historical knowledge available concerning the underlying target variable that maps business objectives. What is available, is a dataset that historically models the general behaviour of the system within an ecosystem. What the ML engineer is asking the cores to do in this case is to discover patterns within the data that can be used to optimise decision performance in the future. It is often the case that unsupervised learning can be followed by early supervised learning, a genre that could be called semi-supervised learning. The process can, under circumstances, evolve to a solid supervised system.

List of AI cores that can apply in this learning genre includes:

  • Clustering techniques
  • Deep Neural Networks and their derivatives
  • Statistical techniques

Reinforcement learning

This type of learning is based on control theory and tries to imitate how we as humans learn. The AI system is left to explore a digital twin of the problem space and is only provided with positive or negative feedback depending on its decisions. The AI gradually calibrates itself so that its decisions maximise the rewards it collects. Techniques in this genre include:

  • Q learning
  • Deep reinforcement learning

Ensemble structures

A collection of AI cores working together and contributing their decisions to solve a problem is called an ensemble structure. These structures can be wrapped under a collective algorithm such as xgboost or can be proprietary hybrid structures working together under a governance structure that can be as simple as a voting system to another AI overseeing all AI cores and grading their contributions.

Training – Validating – Testing

Although this is a step that precedes the core selection it often is implemented very close to the core, so we include the step in this section. Essentially the team needs to decide on a training, testing, and validating strategy structure. Such a process involves segregating the data into appropriate subsets and deciding on machine learning metrics to use to evaluate the AI cores. Often some of the data is completely left out of sight to prevent team biasing during AI core implementations. Cross Validation is a robust approach to the problem of overfitting.

AI Core optimisations

One of the most important steps of the modelling and evaluation phase is optimising the AI cores. Such a process can be very delicate and time-consuming involving tuning of multiple hyperparameters that can affect AI core performance. Some of those hyperparameters include:

  • Learning rate
  • Epochs
  • Momentum
  • Regularisations
  • Network structures

There are a few common strategies that a Machine Learning Engineer can deploy during hyperparameter optimisations within the modelling and evaluation phase including:

  • Grid search
  • Random search
  • Bayesian approaches
  • Gradient-based approaches
  • Evolutionary approaches