What Is Machine Learning?
Machine learning is the ability of an application to identify patterns in the data and predict future events by using these patterns. All data, regardless of type, will have a pattern. Machine learning happens when we train the machine to find and use these patterns to give us some useful predictions.
Machine learning (ML) can be explained as automating and improving the learning process of computers based on their experiences without being programmed.
The process starts with feeding the machine (computer) data and then implementing machine learning models using the data and different algorithms. The choice of algorithm depends on what type of data we have and what kind of task we are trying to automate.
How Things Work in Reality
When it comes to online shopping, there is a lot of data to sift through, including millions of users with an unlimited range of interests with respect to brands, colors, price ranges, and more. In practice, buyers tend to only search for a few products at a time.
Researchers, data scientists, and machine learning experts build models that the machine can then use to automatically complete tasks on its own. Traditionally, advertisements were primarily seen in newspapers and magazines; today, technology has allowed us to implement online buyer-specific advertisements, which is an efficient method for targeting the most receptive audience.
How Does ML Work?
Gathering Data: The process starts with collecting data in the form of text files, excel files, images, or audio data.
Processing Data: Sometimes the data collected is in a raw form and needs to be rectified (i.e., the data must be made understandable to the machine).
Building Learning Models: Learning models need to be built with suitable algorithms and techniques to train the machine.
Testing Models: Model testing is done to measure its effectiveness and to look for any room for improvement.
Implementation: The machine is then given a set of data with certain patterns. Using a machine learning algorithm, the machine will be able to find the patterns in the data and generate a model to identify these patterns in a new set of data.
As shown in the below illustration, if your application supplies new data to this model, the model will process the data, recognize patterns, and compare those patterns with known patterns. This model will then return a probability of the desired output.
QA & Testing for Machine Intelligence
The main difference between intelligent machines and traditional IT systems is that it is extremely hard to predict the output generated by AI systems. In traditional IT, a tester can use defined rules to predict the exact outcome and resulting status of the system and can compare this with the actual outcome.
Intelligent machines have machine learning capabilities that will result in different outcomes if the same function is called at different moments in time. This demands new testing skills.
One thing a tester can do is to define the tolerances for the outcome that sets boundaries between correct and incorrect outcomes.
Test Objects in Machine Learning Solutions
Machine learning solutions contain many components, just like classical software solutions. However, the components are different-especially when it comes to the component's role and properties. Here is an overview and a brief description of the test objects in a machine learning solution:
Dataset: The dataset contains all data that is available for the machine learning solution. Typically, the dataset includes a company's historical data and needs to be processed to be viable.
Training Data: The training data is a subset of the dataset. This data is used to train the model in the development process.
Test Data: The test data is another subset of the dataset. It is used to verify that the model works as intended.
Model: The model consists of the algorithms that the AI uses to interpret the data and learn from it.
Training (Phase): The training phase is the process in which the algorithm learns from data and makes predictions.
Inference (Phase): After the model is trained, it can make inferences based on the input data.
Source Code: A machine learning solution has much fewer lines of code than classical solutions. Nevertheless, there is source code that can have errors and therefore must have unit tests and other relevant tests.
Infrastructure: The infrastructure is subject to non-functional requirements, which must be checked and tested.
Requirements: All requirements should be inspected carefully. The technology is new, so it's possible that the expectations are unrealistic or outside of legal or ethical boundaries.
Input/Output Values: The most basic test objects are the input and output values. This is where the acceptance criteria are verified. The input values are crucial when it comes to understanding how the data is processed.
It is important to take into account that project expectations and the technical capabilities of a machine are not always compatible. Another consideration to keep in mind is that after the data is collected and stored in a central database, there must be assurances that the collection and processing of this data are within legal constraints.
Testing a Machine Learning Process
As a tester, you should know how machine learning works. The below sections detail how machine learning works and how you can contribute to this process.
Understand the Learning Process
There are two types of learning processes-Supervised Learning and Unsupervised Learning. In supervised learning, the desired output will be available in the training data itself.
For example, in the case of credit card fraud, whether the payment is fraudulent or not will be available in the historical data itself. Supervised learning is the most common learning process and involves human intervention.
In contrast, unsupervised learning does not involve human intervention and instead relies solely on machine learning algorithms.
Verify the Training Data
Training data is the pre-processed data that's sourced from raw real-life data. Training data contains different columns of pre-processed data, which are called "feature values," and a desired output, which is called the "target value" (in the case of supervised learning). As a tester, you may need to test the validity of the data.
Train a Model
A model is built to predict the results of the target value. Candidate models are generated from the algorithm after every data feed and the best model will be chosen as the final model. Choosing the best model is especially important in the success of a machine learning system.
A candidate model is created by choosing predictive data from the features available in the training data and supplying it to the learning algorithm. Only 75% of features are made available to the chosen algorithm. As a tester, you should be aware of the selected feature sets, chosen algorithm, and generated candidate model.
Test the Candidate Models
We have seen how candidate models are generated. Now you need to test whether the candidate model created is good or bad. For that, we input the remaining 25% of the features from the training data to this candidate model.
This will allow the model to generate the target value. Since you already know the target value (as the target value is already there in the training data), you can compare whether the generated result matches the target value.
Importance of Testing During the Machine Learning Process
A good machine learning model is built on sophisticated test cases. The process of machine learning involves:
Pre-processing of the data
Training using the processed data and a machine learning algorithm
Testing the learned model with the test data
Better Performance Equals Better Model
The primary obstacle to putting any machine learning model in production is the performance of the model on unseen test data.
For example, when it comes to creating a machine learning model for an autonomous vehicle, the model may perform well 99% of the time but there will always be scenarios that involve unseen test data that could result in an incorrect prediction. This unseen test data can become a large obstacle to account for and overcome.
Testers have an enormous potential to devise unconventional test cases for machine learning models and see how the model is responding before it's put into production. Right now, there are limited tools or frameworks that a tester can use to measure models with different data without fully understanding the internal workings of the model.
If there is a recognized framework, the tester can "play" with the models, using different combinations of test cases. The role of a machine learning engineer is to build a model and analyze the correctness of the model. Then, a tester can take over the rest of the machine learning process.
Testing Problems During the Machine Learning Process
As a software tester, the main problems you will encounter while dealing with machine learning are:
Understanding the questions being asked: Choosing the right question that the machine should be solving is the first step during the development of the machine learning model. For example, when it comes to detecting a fraudulent credit card payment, the question the machine should solve is: "How can I predict whether a credit card payment is fraudulent or not?" As a tester, you should be aware of what the machine is being asked to learn.
Understanding the data supplied: If you have the questions ready, then it is time for the machine to listen and learn. How does the machine learn? You need the right data for each of these questions. So, understanding and supplying this data to the machine is critical. If you provide the wrong data, you will get unwanted results.
Understanding the measure of success: At the end of the day, the output from a ML model is only a prediction, not a definition. Since this output is only a probability, this may change every time. So, you need to know what output you need at the start of the test.
Important Considerations
Here are some other important considerations:
You need test scenarios. Three may be sufficient to represent the expected best case, average case, and worst case.
You will not reach mathematical optimization. We are working with algorithms that produce approximations, not exact results. Determine what levels of outcomes are acceptable for each scenario.
Defects will be reflected in the inability of the model to achieve the goals of the application.
Challenges
Finding predictive patterns is a bit of a challenge. You need to make use of data scientists' knowledge for this.
Process is iterative. Choosing the best model is iterative and takes time to develop, so testing can be time-consuming.
Failures are possible. Machine learning doesn't always end in success, so you are never quite sure of the output value.
Conclusion
During the machine learning process, testers can function as domain experts and data scientists. So, you should know more about the domain and the data. Ultimately, you act as a trainer for a machine that can predict the future!
