Manual vs Automation tests has always been a topic of debate. While automating your tests is definitely a winner in a lot of aspect, .is it always the most efficient way of testing as we go up the pyramid ? This post talks about test automation with go and an API testing approach with a balance of both manual and automated tests. WHAT is an API testing pyramid ? The testing pyramid above is a graphical representation for an API testing strategy. The X axis represents the number of tests The Y axis represents the cost and automation complexity The bottom of the pyramid has the unit tests for our APIs. Unit tests are written by developers for every function public or private in the code. These are easy to be automated and integrated in the pipeline (see below). Most modern languages have testing libraries to implement them. The cost associated with a fix for an issue identified in a unit test is much lower since it is still in the development pipeline. The second level on the pyramid has the integration tests for our APIs. Integration tests are more complex and involve multiple APIs interacting with each other. The number of these tests will be much lesser than the unit tests. These can also be automated and integrated into the pipeline too but need more skills and effort as compared to unit tests (see below). The cost associated with a fix for an issue identified in an integration test is higher as this may involve multiple APIs. The third level has the regression tests and load tests for our system. Regression tests primarily consist of the core functionalities of the system and critical bug fixes that have been addressed in previous releases. Load tests are written to analyze and benchmark the performance of your APIs. The number of these tests will be lesser. The automation of Load tests are harder as the assertion of stats involved in these may not be as straightforward as the unit and integration tests. The cost associated with a fix for an issue identified in this level of the pyramid is high as the fixes are complex and also since these tests are performed later in the release cycle when the builds are more stable to benchmark. The topmost level has the User Acceptance tests. These are mostly preferred to be manual as these may involve multiple user interactions and may not need to be as many as the other tests. The number of UAT tests are limited and their automation may also be minimal. The cost associated with a fix for an issue identified in this level of the pyramid is significantly high almost close to the cost involved if the issue was found in production. WHY an API testing pyramid ?
Mike Cohn's Agile test automation pyramid describes how automation tests become less efficient as we go up the pyramid
Fig 1.1
While a good coverage of unit tests is recommended, it may not be so for the UI or interface level tests. Instead we rather have a strong automation test suite for the Service/API tests.
Let us consider Mike Cohn's example of a simple calculator interface where given 2 numbers we can perform 4 operations on them namely add, multiply, subtract and divide. If we write a test suite for the multiply operation with 10 different input pairs, it is the same multiply button UI interface code being tested 10 times although there could be 10 different workflows in the service/API code. Thus, the automation of Service/API level tests is utmost important and thus the need for an API testing pyramid.
HOW to implement test strategies using the API test pyramid ?
1. Unit tests
Go's built-in support for unit testing using the "testing" package makes it easier to write unit tests as you go.
All tests for a given package must be in the same package.
Naming conventions for the test files is name_of_file_tobetested_test.go
go test runs all the tests *_test.go files in that package.
TestMain can provide you with setup and tear down abilities for all tests in your package. There can be only one TestMain per package. Using external test libraries like ginkgo have caused our CI/CD pipelines to slow down. (see code below)
Assertions and Mocks: github.com/stretchr/testify can be used for assertions and mocks in conjunction with the go test. (see code below)
Integrate with CI/CD pipeline using go test and go tool cover for coverage go test -cover -covermode count -outputdir .cover -coverprofile coverage.cov go tool cover -func=cover/coverage.cov
import (
"fmt"
"log"
"os"
"testing"
"github.com/sshetty10/go-seed-db/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMain(m *testing.M) {
// Before tests
code, err := run(m)
if err != nil {
fmt.Println(err)
}
// After tests
os.Exit(code)
}
func run(m *testing.M) (code int, err error) {
// pseudo-code, some implementation excluded:
//
db, dbCloser, _, err := InitTestDatabase("mydb")
if err != nil {
return -1, err
}
// tear down function for DB
if dbCloser != nil {
defer dbCloser()
}
logger := log.New(os.Stdout, "go-seed-test", log.Ldate|log.Ltime|log.Llongfile)
api = &API{db: db, logger: logger}
resolver = api.NewResolver()
// Run the tests in this package using m.Run()
return m.Run(), nil
}
func TestUpdateDBTrainer(t *testing.T) {
tr := &model.Trainer{
ID: id,
Name: "somename",
City: "somecity",
Age: 70,
LicenseID: "TX-38274",
}
err := api.UpdateDBTrainer(tr)
require.NoError(t, err)
assert.Equal(t, tr.Name, "somename") //assertion using testify
}2. Integration Tests
Integration test automation can be achieved using Postman workflow and Newman. Check my post here to see how.
You can add these to your CI/CD pipeline as a manual stage initially to see how things work with your existing version control workflow.
3. Load Tests
Load test automation scripts can be written using "hey".
You can also use "hey" to load test locally.
Check my post here to see an implementation of load test using "hey"
4. User Acceptance Tests
User acceptance tests have been manual in all of our projects.
These should ideally be done in an isolated staging environment which ideally has its own DB and other third party integrations setup.
Conclusion
This blog represents my suggestions based on my experiences for the testing strategy in an agile environment. A testing strategy, I believe, must be a part of the planning or design phase of every new release. Test lesser but test smarter.
References
The cover story, https://go.dev/blog/cover
Cohn, Mike. "Succeeding with Agile". The Addison Wesley Signature Series.