Complex Software Test Automation Solution for AI-Enabled Robotics

May 15, 2019

Our client, a robotics start-up, hired SQA to evolve its methodology and approach for testing embedded software. SQA’s Quality Engineering team developed a modular software test automation solution designed to serve as a foundational component for all software testing, including Spatial Recognition components. It is also a starting point for almost all integrated testing with the ability to run multiple times daily with no maintenance.

Due to the technical complexities involved, a lab based POC simulation was required for SQA’s verification approach. The “workspace” or product included a control system, robotic devices (such as a robotic manufacturing arm), visual spectrum sensors (cameras) and AI for real time interpretation.

Post POC, SQA’s approach included three primary software test objectives:

  1. A test to verify continuous system calibration;
  2. A test suite to verify systems safety according to technical specifications and regulatory requirements; and
  3. A test that verifies the ability of control system to establish the workspace space based on visual input triangulation.

Mid-implementation, our clients R&D discovered new issues with their critical spatial recognition system component causing a prioritization shift – testing the system’s capability to define and certify the workspace in question. The new challenge – establish the “ground truth” of spatial perception – was prioritized over other objectives during this stage of system development.

SQA pivoted in its approach, focusing on this specific task. Success now hinged on the team’s adaptiveness to:

  1. Quickly become proficient in using ROS (Robot Operating System – a specialized development IDE);
  2. Build out a physical lab; and
  3. Design and deliver a usable and robust spatial recognition test within a fluid and demanding development schedule.

The SQA team responded by developing a modular Python script that took as input a Python library of 3D image files to serve as the “ground truth” standard in a test that stimulated the spatial recognition software and control logic in a real-world equivalent lab setting. It then compared the created images with the saved 3D images to identify accuracy and areas of deviation.

This highly specialized and technical software testing effort is indicative of the breadth and depth of quality engineering expertise required when dealing with disruptive technology. Rapid convergence of well-understood problems and “first time ever” innovations requires a new paradigm of solutioning.


Find out how SQA can help build and implement a software test automation strategy for your organization.

Additional Articles