Our main goal in any project is to make the testing process transparent and measurable. That's why we pay a lot of attention to building a testing infrastructure in our company. In this article, we will explain why this is important, and also share our approaches. This will be interesting for testers, project managers, technical directors, and our clients. Welcome!
Why Infrastructure is Necessary
Testing infrastructure refers to the artifacts required for conducting testing, which includes the testing environment, testing tools, office environment, and procedures.
Testing infrastructure is the preparatory work through which you can quickly and effectively test new features. Our task is to establish a transparent and measurable testing process. Regulations and properly selected tools assist in this regard.
When processes are not described, both the team and the client are unclear on how we work and what is actually happening. On the other hand, when the infrastructure is well-thought-out, any participant in the process can enter the project's knowledge base and refresh all the nuances in memory, look at reports, metrics, task work regulations, and the tools we use. This makes the process transparent. And a transparent process helps save time and money.
Findings: Testing infrastructure is necessary to save resources.
When processes are not described, both the team and the client are unclear on how we work and what is actually happening. On the other hand, when the infrastructure is well-thought-out, any participant in the process can enter the project's knowledge base and refresh all the nuances in memory, look at reports, metrics, task work regulations, and the tools we use. This makes the process transparent. And a transparent process helps save time and money.
Findings: Testing infrastructure is necessary to save resources.
Step #1: Project Analysis
We start a new project with an investigation.
We are obsessed with investigations in our company, and that's cool. Investigations help quickly identify the project's vulnerable aspects and devise ways to cover these vulnerabilities. Therefore, the first thing we do is analyze the current processes and their level of maturity. There are a total of 5 levels of maturity:
We are obsessed with investigations in our company, and that's cool. Investigations help quickly identify the project's vulnerable aspects and devise ways to cover these vulnerabilities. Therefore, the first thing we do is analyze the current processes and their level of maturity. There are a total of 5 levels of maturity:
The lower the level, the more chaotic the process on the project.
Once the level is known, we develop a plan to improve the processes. We always strive for the next level of process maturity. Often on projects with a low level of process maturity, teams neglect testing management systems, for example, Test IT, Allure, Zephyr, Qase, and others. Instead, they use Telegram chats, Google Docs, or Trello.
This is inconvenient — test cases get lost, forgotten, and are not recorded. Therefore, we usually suggest choosing one of the systems. However, if the client doesn't want to, we don't press. Here, the master is the lord.
Also at this stage, we analyze the volume of work: whether there's a test model, the volume of testing, and the form of documentation. This is also important because it helps us to get a picture of the project and understand how we will fit into it.
Also at this stage, we analyze the volume of work: whether there's a test model, the volume of testing, and the form of documentation. This is also important because it helps us to get a picture of the project and understand how we will fit into it.
Step #2: Preparing Documentation
Once the analysis is complete, we develop 2 main documents:
- Test Plan — this comprehensive document describes everything necessary for successful testing. It answers the question "what to test?".
- Testing Strategy — this outlines how to carry out testing to achieve testing goals under given conditions. It helps to determine the scope and area of testing. This gives us a clear understanding of the project, and the likelihood of missing any test action is reduced. It answers the question "how will we test?".
These documents are our holy grail. The testing process is detailed in these documents. Anyone who opens them will understand what we are testing and how. This is the way to the transparency we talked about earlier. Both documents are stored in a knowledge base – for instance, in Wiki.
Here’s what we specify in them:
1. How we test the product, what tools we use, where tasks are stored. If it's Jira, we describe the process of working with Jira. If it’s YouTrack — with YouTrack. The document covers absolutely everything: from which status we take tasks, how to properly report bugs, and when to take them for re-testing.
2. How we work with defects: the rules for their creation, workflow, severity gradations. We aren't afraid to specify even the most elementary things — even someone who has never worked in QA should understand the process.
3. Rules for working with the Test Management System, in which we will maintain test documentation. In our projects, we use systems like Test IT, Allure TestOps, DoQA, and others. The systems can be absolutely any. The main thing is that the system meets the project's needs.
4. Regulations for test case development and rules for going through test runs. A test run is the execution of a given set of tests. Essentially, it's our report on the test results: how many tests have been passed, how many failed due to bugs.
Once the processes are specified and the infrastructure is ready, all that remains is to apply everything in practice.
1. How we test the product, what tools we use, where tasks are stored. If it's Jira, we describe the process of working with Jira. If it’s YouTrack — with YouTrack. The document covers absolutely everything: from which status we take tasks, how to properly report bugs, and when to take them for re-testing.
2. How we work with defects: the rules for their creation, workflow, severity gradations. We aren't afraid to specify even the most elementary things — even someone who has never worked in QA should understand the process.
3. Rules for working with the Test Management System, in which we will maintain test documentation. In our projects, we use systems like Test IT, Allure TestOps, DoQA, and others. The systems can be absolutely any. The main thing is that the system meets the project's needs.
4. Regulations for test case development and rules for going through test runs. A test run is the execution of a given set of tests. Essentially, it's our report on the test results: how many tests have been passed, how many failed due to bugs.
Once the processes are specified and the infrastructure is ready, all that remains is to apply everything in practice.
Step #1.5: If the processes are not set up
Let's take a half step back.
Previously, we described connecting to an existing process. We connected to the project, it has been running for a long time, some processes exist on it, we studied them and proposed improvements. If our ideas are liked — great. If not — we simply integrate into the existing process.
But what if the project is starting from scratch, and no regulations and rules exist?
In such a case, we just build the process from scratch, relying on our experience and knowledge. We also have a ready-made solution for this scenario — Imagaban. We won't go into detail about what this approach is in this article. But we also can't say anything about it. Imagaban is the processes and tools we have developed and used.
We offer the client to work in an already ready infrastructure, to use our regulations, and to work according to our workflow. Firstly, this allows us to start work quickly, as there is nothing to invent. Secondly, it saves resources for all participants: all processes are fine-tuned, so there is no need to spend additional effort.
Below we will talk about how and with which tools we like to work.
Previously, we described connecting to an existing process. We connected to the project, it has been running for a long time, some processes exist on it, we studied them and proposed improvements. If our ideas are liked — great. If not — we simply integrate into the existing process.
But what if the project is starting from scratch, and no regulations and rules exist?
In such a case, we just build the process from scratch, relying on our experience and knowledge. We also have a ready-made solution for this scenario — Imagaban. We won't go into detail about what this approach is in this article. But we also can't say anything about it. Imagaban is the processes and tools we have developed and used.
We offer the client to work in an already ready infrastructure, to use our regulations, and to work according to our workflow. Firstly, this allows us to start work quickly, as there is nothing to invent. Secondly, it saves resources for all participants: all processes are fine-tuned, so there is no need to spend additional effort.
Below we will talk about how and with which tools we like to work.
Step #3: Metrics
Testing metrics deserve a separate article. But when we talk about infrastructure, we can't ignore them. Let me remind you, our task is to make the process transparent and measurable. We've discussed transparency. Now about measurability.
Metrics allow us to monitor performance indicators, for instance, defect density per task. We already have a set of basic metrics and an extended set, which we use if the basic list is insufficient.
And the main news: the costs for implementing metrics are minimal.
If it's a manual collection of metrics, it will require very little time from the tester, while the benefits they bring are immense. Even at the start of a project, we can implement metrics and control quantitative indicators.
Here are examples of some of our basic metrics:
Metrics allow us to monitor performance indicators, for instance, defect density per task. We already have a set of basic metrics and an extended set, which we use if the basic list is insufficient.
And the main news: the costs for implementing metrics are minimal.
If it's a manual collection of metrics, it will require very little time from the tester, while the benefits they bring are immense. Even at the start of a project, we can implement metrics and control quantitative indicators.
Here are examples of some of our basic metrics:
- number of defects logged by the team for the reporting period;
- defect density per task;
- percentage of canceled defects;
- percentage of fixed defects;
- percentage of reopened defects.
When certain indicators drop, the client loses money. Therefore, regular collection of metrics and analysis of indicators helps to make timely changes to the process and save resources. Plus, this way we can identify bottlenecks and areas for growth.
If there are few tasks coming in for testing — the development team can be expanded or the number of tasks in the sprint can be increased. If testing finds too many bugs — perhaps there's a problem within the development team. For instance, a new developer has joined or development tasks are coming in with "raw" requirements. Metrics help us identify such situations, analyze the problem, and implement an effective solution.
We also have the capability to automate the collection of metrics using the Metabase tool. For this, we need read access to the Jira database.
We also have the capability to automate the collection of metrics using the Metabase tool. For this, we need read access to the Jira database.
The Tools We Love
This and the following section are about the resources that allow us to make testing as effective as possible.
At the infrastructure creation stage, we decide with the business customer which tools we will use in work. We are ready and able to work with any, but there are those that seem more convenient to us.
We have already mentioned project management systems. We prefer Jira, have a good attitude towards Redmine and YouTrack. Although we had different projects and different working options in our practice. When we enter a project, we tell the customer about the advantages and disadvantages of each of these systems. But if the processes are already arranged differently — we will adapt.
We develop test cases in the test management system. We often work in Test IT, Allure TestOps, and DoQA. Each has its own pros and cons. Test IT has a more convenient interface, while TestOps makes automation easier. Using these systems is an additional cost for the customer, but we better control the testing process. Even if the project budget is limited, we are ready to find a very cheap option.
Implementing such systems is important — it organizes the process. Keeping test cases, for instance in Google documents, is inconvenient and ultimately dangerous. They can be lost. And any specialized system makes the process not only more transparent but also more efficient.
We discuss all of this with the customer in advance. Our common goal is to create a quality project, and we explain what will help us in this.
We always focus on the project and the needs of the customer. Here's an example from one of our projects:
We have already mentioned project management systems. We prefer Jira, have a good attitude towards Redmine and YouTrack. Although we had different projects and different working options in our practice. When we enter a project, we tell the customer about the advantages and disadvantages of each of these systems. But if the processes are already arranged differently — we will adapt.
We develop test cases in the test management system. We often work in Test IT, Allure TestOps, and DoQA. Each has its own pros and cons. Test IT has a more convenient interface, while TestOps makes automation easier. Using these systems is an additional cost for the customer, but we better control the testing process. Even if the project budget is limited, we are ready to find a very cheap option.
Implementing such systems is important — it organizes the process. Keeping test cases, for instance in Google documents, is inconvenient and ultimately dangerous. They can be lost. And any specialized system makes the process not only more transparent but also more efficient.
We discuss all of this with the customer in advance. Our common goal is to create a quality project, and we explain what will help us in this.
We always focus on the project and the needs of the customer. Here's an example from one of our projects:
- There we worked on two-week sprints, the sprint started on Wednesday. Six days for testing new functions. On Wednesday evening there was a feature freeze, only features that were developed before this could come to us. Regression started on Thursday and lasted for three days. On Tuesday, there was a smoke test on preprod. And on Tuesday night, the release was rolled out to prod.
In parallel, we developed autotests. The goal was to reduce the time for conducting regression. We gradually automated the regression model and applied various methods to shorten it without losing quality.
Summaries and Conclusions
Let's take a sports analogy.
To start running, one needs to prepare in advance. Acquire the appropriate footwear, learn at least something about the technique, determine the route, and warm up. Recall all the physical education classes in school when we were made to do special running exercises.
Creating a testing infrastructure is about the same thing. The most logical and correct process to start testing a product is to describe the processes, select and organize the tools, obtain accesses, and set up accounts.
After this, we can focus on the testing itself — transparent and measurable. Which means, effective and accurate.
Creating a testing infrastructure is about the same thing. The most logical and correct process to start testing a product is to describe the processes, select and organize the tools, obtain accesses, and set up accounts.
After this, we can focus on the testing itself — transparent and measurable. Which means, effective and accurate.
