Automation can help to offset the limitations of manual testing, including poor scalability, long time-to-execute, and the intensive requirement for human resources. However, building test automation solutions that are operationally and cost effective remain a challenge for many organizations.
SQA conducted a webinar, Bridging the Gap Between Manual Test and Automation. Here are top questions the team has received since then about how to introduce automation into QA and testing, as well as answers based on our many years of applying cross-industry experience and multi-stack testing implementation to automation:
Q: What is the cost of an automation framework?
A: The cost can vary a great deal, and there are multiple factors to consider:
- Buying and implementing the automation tool
- Developing a significant body of automations to achieve specific purposes
- Maintaining the tool over time (additional development will need to be done)
If you are using low-cost or open source tools, the cost of entry can be relatively low. You simply must choose the right targets for automation, and approach it the right way.
We always start with our clients by producing a small prototype. This proof-of-concept verifies that the tool or approach you are using will work with the technology you’re targeting. It also helps you understand the total cost of ownership to develop the automation, extend it, and maintain it.
It’s also important to keep the business value in mind. The cost associated with automation should be treated as an investment because the automation should be offsetting other costs from manual toil. Those other costs can be dollars, time/efficiency, and quality.
Q: How can we reduce the cost of updating and maintaining a test automation framework?
A: Be sure to apply good application development engineering practices in the design of your test automation framework. While an automation framework is a specialized type of application, it should be developed with the same level of rigor as, say, a commercial application, to make it more maintainable.
Also, be sure that you have a good alignment between the applications you’re going to test and the problems you aim to solve. For example, if you have an application that is written in micro-services, it will probably lend itself well to API-level testing. For an old legacy application, you’ll probably need to use UI testing.
Another example: If you have a very simple toolset with small amounts of code and minimal changes down the road, and you’re trying to plug in significant automation when you really don’t need to or it’s not the right tool — that will be costly. So, to reduce the cost, think about what, why, and when you need which tool.
Another important aspect of reducing cost is to use an enterprise approach. To get more value in return for the cost of automation, look across the organization and find ways to apply your automation test framework as many places, and as often, as possible.
Q: What is a good roadmap for a team that has an existing application with no automation coverage to move to more automation?
A: First and foremost, you must understand your target — what testing you are doing manually and what you are going to replace with automation. Start with low-hanging fruit: where the technology is well-suited to automation and where the application is stable. Then, when you’re ready to move into more complex, business-critical automations, you will have experience in designing and deploying the automation.
One way to approach this is to have a requirements-gathering session with the product owners, and select targets that have clear value for them and where the problems faced are well understood.
Q: How do you best spend time to validate the test automation tools?
A: This gets into a risk-based approach in which you ask yourself if you really need to qualify every single piece of the tool’s functionality. When it comes to qualifying a tool for regulatory compliance, you should not spend a great deal of time trying to validate the automation tool itself, but rather spend time qualifying it for your environment so you can continue to focus on your product.
For example, if you’re using a tool for a minor risk area and the tool has a good reputation with many companies using it, and it’s installed on a qualified environment, then specify that reasoning in your validation plans and move on. In other cases, the risk can be extremely high with regard to where and why you’re using it, and it is worth spending more time to prove it is valid for the way you’re going to use it for your application.
Q: What are the trendy tools being used for test automation?
A: For test automation, it is more about using the right tool for the right job versus using a trendy tool. Several considerations should go into selecting the tool.
You should understand your:
- Business objectives
- Portfolio and technology
- Technology strategy for the coming years
- Team’s capabilities
- Testing objectives
- Toil (where are you spending time with manual testing today?)
At SQA, we typically complete a discovery session in which we uncover answers to these questions for our clients. Otherwise, simply picking a popular tool might not meet the need.
Keep in mind that there is a lot of variation in automation tools. Some are very specialized to do one thing. Some are designed to do everything. Be sure to understand what other tools you have in-house, and what is your particular need.
Q: What development model works best for automation?
A: You can use automation with any development model. So, the real question is not about what development methodology is best suited to automation, but rather how you fit automation into your model.
For example, if yours is a Waterfall team that takes things one step and phase at a time, test automation can work. If yours is an Agile shop and you work in sprints with constant chunks of work completed and deployed, automation can also work.
The bottom line is that test automation reduces human toil and risk. It can do that in any development model and at any point in the software development lifecycle.
Q: Do you suggest automation in an Agile environment, where the requirements are constantly changing in every sprint?
A: It’s a good practice to automate even in cases where there are a lot of changes. Just be mindful of the time and cost you are spending to do it.
For example, if you spend an hour developing test automation in a case in which you are doing dozens of code pushes in a week, the automation might save you a hundred hours. Even if you threw away the test at the end of the week, it would be worthwhile: one hour to save a hundred hours.
Automation can work extremely well, even in a changing environment, but it isn’t about automation for the sake of automation. You really need to weigh the business value versus the cost of the automation.
Q: As companies transition from manual testing to automation, the most important aspect is increasing coverage. How have you seen clients measure this?
A: Automation ties into the same type of quality of product and process metrics you would track in any case — whether you add automation or not:
- In the same way you measure any sort of testing, with automation be sure to measure against requirements coverage and feature coverage.
- You also should measure the amount of time reduced in testing by looking at time spent in human toil — the amount of hours it takes to accomplish the task — compared with the time it takes to run the automation.
- It’s also essential to look at the automation’s effectiveness at finding defects.
Take the next step
SQA brings cross-industry experience and multi-stack software testing implementation expertise to automation. The SQA team can implement and extend test automation for:
- Validating mobile and web experiences across devices and browsers
- Integrating services and APIs into your technology ecosystem
- Validating the quality and accuracy of data for all key data types and uses – streaming, batch, EDI/ETL, data warehouses to data marts, reporting, and analytics
- Driving regression testing to speed cycle time and find defects that would otherwise disrupt production and negatively impact customer satisfaction
- Assuring comprehensive and efficient coverage in combination with manual testing