AI in Testing — Another Brain in the Equation

June 11, 2018

New and emerging artificial intelligence (AI) technology is changing how we approach software quality testing.

AI enables testers to take advantage of new ways to do tasks that formerly only humans could do. This enables engineers to re-imagine new approaches and solutions to known problems. Further, AI technology is changing the tools that test engineers have at their disposal. For example, AI-based testing tools such as Mabl and Test.AI are evolving traditional testing by adding a “brain” to testing activities.

Many test design and test execution decisions currently made by humans can be delegated to AI.

Artificial Intelligence (AI) and Machine LearningAI in its basic form is one or more algorithms; thus, some traditional testing, tuning and calibration techniques still apply. Understanding boundary conditions, out of scope/range situations, and component/communications failure are still fundamental verifications testers must perform. Since AI systems can learn and mimic how humans deliver these results and learn from them, it’s like adding the capability of another brain to the test team—ultimately accelerating the process.

Example: AI is changing how testers approach risk-based testing

Below are two example scenarios for risk-based testing; one without AI and one utilizing AI:

Standard Approach to Risk-Based Testing

Here’s a common risk-based testing approach:

  1. Humans survey the population of tests.
  2. Humans make judgments about the likelihood of an error occurring in the functionality covered by each test, as well as the detrimental impact if it does occur.
  3. Humans prioritize the tests to cover the highest risk first.

Currently, human testers do these activities based on conversations with stakeholders and their own previous experience. Then, based on run results that confirm or contradict the accuracy of the algorithm(s) used to establish relative risk, humans can adjust the algorithm(s) for the next test suite.

AI-Based Testing Approach to Risk-Based Testing

Think of the activities in the above risk-based testing approach as one or more algorithms that are executed within an adaptive neural network system. With AI-based testing tools, the system can:

  1. Replicate the initial human examination and prioritization process.
  2. Control and measure test execution.
  3. Compare test prioritization with defect containment failure statistics.

AI can then self-modify its algorithm based on that experience and in certain cases even identify and build new tests/scenarios to close the risk coverage gap. This approach can be applied to the majority of risk-based test approach implementations without regard to industry or application technology.

The emergence of a new class of test automation tools using AI will become an important part of the tester’s toolset.

Today many cases of AI/machine learning technology being applied is occurring in custom-coded frameworks developed in parallel with specific product development efforts. Expect to see more test automation tools using AI as testers readily embrace its power.

What will be the implication for QA professionals? Tester activity will shift from test case definition and execution to the definition and tuning of computational algorithms and definition of test and training data sets. Most of the tester’s time will shift to the evaluation of the AI’s learning progress and path. The sanity checking of the AI-generated test results will be based on the massive parallel processing and intuitive correlations human brains do best. And data analytics will further augment results. Testers will embrace AI as a powerful new partner, marrying the things machines traditionally do best with human traits and patterns.

Additional Articles