Custom Software Test Automation Framework for AI/Machine Learning Application

April 9, 2019

Our client is a rapidly growing Boston-based product company introducing an AI-based virtual assistant or coach. This bleeding edge application assists help-desk personnel in real time by analyzing voice patterns, dialogue and discourse cues to recommend more effective ways to interact with the caller. SQA was hired to address a unique software test automation initiative viewed as unsolvable via commercially available tooling.

SQA’s Quality Engineering team enacted Progressive Engineering (shift-left) techniques to deploy an automated regression software test suite capable of handling the unique variables of conversational and AI-generated input. With the organization shifting from R&D to product-driven, the goal was to mature testing – from ad hoc and intensely manual and subjective – to a more Agile and automated environment. Highlights of the project included:

  • Identifying and designing a technical solution that leveraged the recorded help desk interactions (.wav files with known outcomes and patterns as interpreted by human psychologists) used in the actual machine learning protocols;
  • Building a Java/Selenium Page Object Model Software Test Automation Framework to stimulate the AI with “known quantifiable states” and comparing the recorded interaction dialogues with the dialogues captured during the test;
  • Working closely with the data scientists involved in machine learning to develop a representative regression suite and a process for continually updating the expected result set and test suite content based on a never-ending evolution of system capability and sensitivity;
  • Leveraging SQA’s IP and experience with comparable technologies and similar organizational maturity to fast-track integration into their ongoing Agile development and build/release process.
  • Integrating the custom-built software test automation with a maturing Continuous Integration (CI) system.

SQA’s hands-on technical capabilities, innovative design approaches, strong methodological and market segment experience and adaptive mentoring styles proved to be a winning formula for our client. Without creating a “drag” on maturing processes and Agile transformative activities, the engagement resulted in a 25% uptick in both velocity and quality.


Find out how we can do the same for your organization.

Additional Articles