Test automation in a validated environment: How we reduced the duration from 14 days to 2 hours

In the projects I am working, the general workload of the team is high and there are mostly just enough people to get the daily job done. Any additional workload coming up is difficult to fit into the already tight schedule and there is not much time to engage in specific topics in detail. Let’s look into it in this blog.

Why?

In such situations, I observe that teams leave aside the least appreciable tasks taking up the largest amount of their time with the least direct added value. This can have a strong impact on the quality of the deliverables. In our business of developing custom-made software or customizations of existing tools, performing thorough and sensible testing of new system features or integrations is often underrated and its benefits are misjudged. Especially in an agile working environment, where new features are expected every 2-4 weeks, the testing team is under high pressure to deliver fast results.

When moving to systems that are used in a validated setup, the time required to test changes of the validated systems take even longer. A detailed testing documentation is expected and supports the creation of a clear audit trail to demonstrate compliance with regulatory requirements and standards.

We all know the risks of deprioritizing testing:

• Unhappy clients
• Buggy application
• Regulation breaches

just to name a few.

Looking at the demographic evolution of Europe, covering the need for skilled labour resources will be increasingly difficult, so hiring a testing team alone may not do the trick in the long run. Also, testing might still take a decent amount of time, as "testing" not only includes testing of features itself, but also preparing the test cases and documenting test results for traceability, especially in a regulated environment such as Pharma or MedTech companies.

Another more effective option is to automate testing wherever possible, resolving the issues mentioned above:

• missing skilled labour resources
• uninteresting work skip
• risk of unhappy clients, buggy applications, and regulation breaches

Of course, setting up automated testing is not done in a day. Still, it might very well be that the investment will pay off sooner than you think. Let me tell you the story of how our testing experts, who have developed a test automation framework for one of our Life Science clients, have reduced the testing time of new features and tool integrations in a regulated environment from 14 person days to 2 work hours.

Pain Points

Our client was building a large platform in-house, onto which several systems were to be integrated. The platform would contain systems of various complexity within themselves as well as on the platform, regarding the number of connections between the systems. Furthermore, the platform was also to be embedded in a highly regulated environment.

A significant amount of testing was required to ensure a stable working solution. Each integration and adaption needed to be tested with each change applied. The regulated environment, greatly increased the complexity of the testing, as well as the level of accuracy and correctness expected from the test results.

Manual testing is known to be highly time consuming, error prone and mostly repetitive. It also requires detailed test execution evidence to be created for each test run performed to ensure traceability and that the testing is in line with the validation compliance. Our expert testing team invested around 2 weeks in testing and documentation for each change and new feature that was to be deployed to the platform. The testing of a new feature or a change took approximately the same amount of time as developing the feature itself.

Solution

The team started to think about the testing process they were going through and how it could be improved to lower the time consumption as well as the number of errors expected along the testing efforts. In a first step, the team summarized the benefits for the business regarding the implementation of a test automation framework and defined the challenges, the framework should overcome in order to deliver the expected benefits:

Benefits

• Heavy cut down in person days for test execution procedure
• Fit into the agile way of working of the company
• Reuse code base of informal testing

The test automation framework should

• Have a qualified solution design
• Be able to create testing execution evidence with regards to timestamp
• Be integrable in Application Life Management (ALM) tools which are in line with validation compliance

Phase 1

Before starting to build the framework, the team needed to understand the challenges in detail. Several brainstorming sessions were held to fully understand the expectations and break down the challenges into work packages that could be developed. Furthermore, the expected workflow of the applications within the platform was clarified to understand the interdependencies of the systems on the platform.

In order to select the best fitting testing framework to meet the project needs, the team analysed the required applications. This is eminent as each type of application is best suited to a specific automation framework. In our case, UI (User Interface) based applications were planned to be part of the platform, so Selenium was chosen as the testing framework.

Phase 2

As the applications were UI based, all actions performed by an end user needed to be tested against the defined features and requirements to be developed. When a human performs the testing, he or she interacts with all the UI elements, such as buttons and dropdown boxes of the applications. When a program performs the tests, the system needs to know exactly where the UI elements are within the application. Our team used the application workflows defined in Phase 1 and divided it into page object models (POM) to identify the locators needed by the test system to find the application elements.

For the test automation framework, it is crucial to choose a locator strategy that is reliable, unique, and resistant to change. For example, if a locator is not unique, multiple elements may be found on a page of the application, leading to unexpected behaviour of the test script. If the locator is not resistant to change, any adaptations on the UI can break the test script.

Once the customer’s needs, scope, and framework had been defined, the initial phase of the test automation scripts began. Together with the client, we built the environment, including the setup of hardware, software as well as network infrastructure. We prepared the test data, designed the test scripts, including the hierarchy and organisation as well as the modularization of the scripts into reusable components. We then developed the test scripts and defined the necessary settings for their execution.

Our focus lied on the exchange with the stakeholders and incorporation of their benefits. Regular alignments ensured the coverage of their needs such as

• Strong increase in testing efficiency
• Scalability and
• Maintainability of the solution

A well-planned initialization phase of the test automation script is of highest importance to lay the groundwork for the subsequent development, maintenance, and execution. The better the initial phase is structured and planned, the easier it will be to automate, maintain and scale the test suite.

Phase 3

Once the automated testing framework was ready for use, the stakeholders were approached again and detailed feedback regarding the implementation was collected. As before, our team focused on understanding and incorporating our customers’ inputs into the solution.

In addition to the automation of the testing itself, providing automated delivery of all test execution evidence to Application Life Management (ALM) tools was of utmost importance to our stakeholders. As the framework is to be used in a validated environment, the test execution evidences need to be saved and made available in ALMs for historical and documentation reasons.

The final phase of the project included the preparation of all the handover documentation and a formal presentation of the product to the stakeholders. Our team delivered all the necessary information to the stakeholders to be able to fully take over the automated test suite.

Achievements

The main achievement and benefit for our client is the reduction of testing time from 14 person days to 80-90 minutes. This time reduction has a significant impact on our client’s development and delivery cycle.

1. Firstly, new features can be tested immediately after deployment in the test environment, giving instant feedback on the quality of the development and on the points to improve.

2. Secondly, the reduction of time needed for testing allows the dev team to use agile frameworks and develop new features, even for systems in a validated environment.

3. Thirdly, the reduction in testing time allows many more releases per year in a validated environment, removing the present systemic inertia.

Another achievement is a strong increase in the quality of the developed systems. The automated testing framework increases the number of bugs and gaps found during testing and has a positive impact on the final quality of the delivered systems.

Thanks to the automated testing framework, employees can use their time to work on tasks beneficial for the company but also for themselves. Instead of performing repetitive work, new and more complex tasks can be taken on, lifting the well-being of employees and their connectedness to the company.

Ultimately, the time saved also leads into cost reductions and efficiency gains. Employees can use 13 days per requested change for additional developments, supporting other team members in their tasks, or testing of additional features, just to name a few, and directly help to reduce the labor shortage faced by companies.

Implementing an automated testing framework is complex and requires time and efforts, as well as a thorough understanding of the situation and the goals to be achieved.

The investment in creating an automated framework pays for itself and has a strong positive impact on

• The efficiency of the department or company
• The quality of delivered systems, also in validated environments
• The happiness and ultimately the productivity of the employees

Make good use of the talents and abilities in your company! If you need help, reach out to us.