Custom-made test automation framework for an advanced AI/Machine Learning application.
Our client is a rapidly growing Boston-based product company introducing an AI-based virtual assistant or coach. This bleeding edge application assists help-desk personnel in real time by analyzing voice patterns, dialogue and discourse cues to recommend more effective ways to interact with the caller. SQA was hired to address a unique automation initiative viewed as unsolvable via commercially available tooling.
SQA’s Quality Engineering team enacted Progressive Engineering (shift-left) techniques to deploy an automated regression test suite capable of handling the unique variables of conversational and AI-generated input. With the organization shifting from R&D to product-driven, the goal was to mature testing – from ad hoc and intensely manual and subjective – to a more Agile and automated environment. Highlights of the project included:
- Identifying and designing a technical solution that leveraged the recorded help desk interactions (.wav files with known outcomes and patterns as interpreted by human psychologists) used in the actual machine learning protocols;
- Building a Java/Selenium Page Object Model Test Automation Framework to stimulate the AI with “known quantifiable states” and comparing the recorded interaction dialogues with the dialogues captured during the test;
- Working closely with the data scientists involved in machine learning to develop a representative regression suite and a process for continually updating the expected result set and test suite content based on a never-ending evolution of system capability and sensitivity;
- Leveraging SQA’s IP and experience with comparable technologies and similar organizational maturity to fast-track integration into their ongoing Agile development and build/release process.
- Integrating the custom-built test automation with a maturing Continuous Integration (CI) system.
SQA’s hands-on technical capabilities, innovative design approaches, strong methodological and market segment experience and adaptive mentoring styles proved to be a winning formula for our client. Without creating a “drag” on maturing processes and Agile transformative activities, the engagement resulted in a 25% uptick in both velocity and quality.