What is testing?

Software testing

Software testing is a process of evaluating and verifying the various software applications and products in terms of performance, correctness, and completeness. It helps identify bugs, gaps in the product, defects, and missing requirements with set expectations.

In the early days of computers, software developers relied heavily on debugging, a process for removing and detecting potential errors. After the 1980s software grew in size, several different testing types of products also grew in parallel depending on the requirements. Testing was primarily done in the later stages of the software lifecycle. Now it’s evolved to be integrated at the early stages as well.

TIP

The ideal testing scenario is to have the least tests written to find the largest number of defects.

While software testing is important in any scenario, the real test of the products comes when it’s launched the market, there, it’s judged by stakeholders and users.

Benefits of testing

Testing helps detect:

poor designs
change inefficient flow or functionality
address scalability concerns
find security vulnerabilities

Testing helps provide:

AB testing to find the best suitable options
address compatibility with platforms and devices
provide assurance to stake holders
a better experience for end users

Best Practices for Effective and Reusable Software Testing

There are a few good practices that must be followed in testing to achieve optimal results.

Test code allowing reusability of tests.
Tests must be traceable to the requirements set.
Tests written must be purpose-driven, efficient, and allow for repeatability.

These testing techniques can then follow a procedural approach according to the type of testing used.

Testing lifecycle

planning
preparation
execution
reporting

The steps involved in achieving this can include:

writing scripts and test cases
compiling test results
correcting defects based on them
generating reports from our test results

Test Cases

They are a general sets of actions containing steps, data, pre and post-conditions written for a specific purpose.

This purpose can improve functionality, flow, and finding defects.

A well written test case

provides good coverage
reusability
better user experience
reduce costs
increases overall satisfaction

As the tech industry is ever-growing, several test and categories, types, tools, and products have evolved, which are tailored to best meet the requirements of the software in question.

For example, a webpage will have different testing needs than an Android-based game.

Testing can be categorized by several different factors. For example, depending on the amount we know about the internal implementation, we can call it black box or white box testing.

There are also many testing types using practice. These include compatibility, ad hoc, usability, and regression testing.

There is no one-size-fits-all solution.

When testing products, it’s also important to understand when to stop. There is no application will ever be 100 percent perfect. Otherwise, a developer may feel the product is well-tested, but realize it’s full of bugs and flaws as soon as it’s released to the end-users.

A few metrics can be established for this purpose, given that there are well-written test cases in place.

These include:

a certain number of tests cycles
passing percentage of test cases
time deadlines
time intervals between subsequent test failures

Types of testing

Quality assurance today has become an important component in the software development life cycles. Much of the credit goes to development of testing tools and techniques.

main levels or categories of testing

units
integration
system
acceptance testing

There are white box and black box tests.

White box test

White box testing is whether tester has knowledge of the code design and functionalities.

Black box test

Black box tests function with no such information and the tester has no idea about the internal implementation.

There are also other ways to categorize different tests as functional, non functional, and maintenance tests.

Functional tests

Functional tests are based on the business requirements stated. They determine if the features and functionalities are in line with the expectations.

Non-functional tests

Non functional tests are more complex to define and involve metrics such as overall performance and quality of the product.

Maintenance tests

Maintenance tests occur when the system and its operational environment is corrected, changed or extended.

But there are also manual and automated testing methods that are dependent on the scale of the software.

The most broadly accepted categorization is in terms of the levels of testing as you move ahead in the software lifecycle.

The four main levels of testing

unit/component testing
integration testing
system testing
acceptance testing

Unit / Component testing

In unit or component testing, the program tests specific individual components by isolating them. The components are low level which means that they are closer to the actual written code. They often involve use of automation for continuous integration given their small sizes.

So you usually write these tests while writing the code.

Integration testing

Integration testing, combines the unit tests and test the flow of data from one component to another. The key word here is an interface. This means that you test if the data is correctly fetched from a database within the python code, and if you have sent it to the web page.

There are different approaches to it such as top down, bottom up and sandwich approaches.

Your approach depends on what code level interfaces you attempt first. It builds on the unit testing and a tester deals with it.

System testing

Which tests all the software. You tested against the set requirements and expectations to ensure completeness. This includes measurements of the location of deployed components such as reliability, performance, security and load balancing.

It also measures operability in the working environment such as the platform and the operating system.

This is the most important stage handled by team of testers.

WARNING

It’s also the most critical stage as the shipping of software to the stakeholders and end user happened after this phase.

Acceptance testing

When the product arrives at this stage, it’s generally considered to be ready for deployment. It’s expected to be bug free and meet the set standards.

The stakeholders and the select few end users are involved in acceptance testing.

It normally involves alpha, beta and regression testing. One way of approaching this is to

Give pre-written scenarios to the users.
You use the results for improvements and try to find bugs that were missed earlier.

All the different testing levels are designed to optimize software at different stages. The key to testing is testing early and testing frequently. While each of the testing phases is important, early detection

saves time
effort
money

As the code gets increasingly complex mistakes become harder to fix. It doesn’t necessarily mean that unit testing will happen only at the beginning and acceptance at a later stage.

There are many testing cycles where these levels are approached iteratively. A typical example is the agile model, here you release different versions of the product iteratively and you perform acceptance testing every few weeks.

Test automation packages

In the past, machines have substituted human efforts in making goods, which helps us save both time and effort.

In programming, the tests chosen for automation are the ones that have high repeatability and volume, predictable environments, and data, and determinants outcomes.

There are a number of testing types that can be automated. These include:

units
regression
integration

Steps in test automation

preparing the test environment
running the test scripts
analyzing the results

Important Python testing framework

Unittest

The built-in testing package pyunit or unittest.

The unittest framework supports

test automation
independent testing modules
aggregation of tests into collections

Pytest

A native Python library that is simple, easy to use, and reasonably scalable.

Functional

It can handle several functional test types such as unit, integration, and end-to-end.

Parameterized

There is support for parameterized testing which enables us to execute unit tests multiple times with different parameters passed.

Parallel

It can run parallel tests and generates HTML, XML, or plain texts reports.

You can also integrate it with other frameworks like Pyunits and nose too, and web frameworks like Flask and Django.

While primarily used with testing APIs, it’s also well used with UI, database connections, and other web applications.

Easy creation and quick bug fixes a why Pytest is the most popular testing framework for automation.

Robot

Which is popular primarily for its keyword-driven development capabilities. These keywords are used in test cases and can be predefined or user-defined.

Robot is very versatile and use for

acceptance testing
robotic process automation or RPA
test-driven development

It can be used for many domains, including:

Android
APIs
mainframes

Selenium

Selenium is another open-source testing framework that has gained popularity over time, and it’s primarily driven towards a web applications.

It has support for the majority of web browsers and OS.
There a browser-specific web drivers that enable testing functionalities like login, button clicks, and filling out forms.
It allows the test set to select the speed and execution of tests and it has an option to run specific tests or tests suites.

Apart from the popular frameworks Pytest, robot, and selenium, there are many more.

It’s important to know that a number of these testing frameworks are often used with other tools such as plug-ins, widgets, extensions, test runners, and drivers.

Writing tests with PyTest

Py test is one of the most popular modules for unit testing in python. This is because it allows you to do simple tests with minimal effort and it also has simple clean code with good documentation.

# addition.py
 
def add(a, b):
    return a + b
 
def sub(a, b):
    return a - b

# test_addition.py
 
import addition
import pytest
 
def test_add():
    assert addition.add(4, 5) == 9
 
def test_sub():
    assert addition.sub(4, 5) == -1

Import the file that consists of the functions that need to be tested. Import the pytest module.

WARNING

Each test case should be named test underscore then the name of the function to be tested.

assert

Use the assert keyword inside these functions because tests primarily rely on this keyword. It checks for conditions in your code and expects a boolean value of true or false. When the return value is true, the test passes when it is false the test fails.

python -m pytest test_addition.py

=========================================== test session starts ============================================
platform win32 -- Python 3.12.2, pytest-8.1.1, pluggy-1.5.0
rootdir: C:\Path
collected 2 items
 
test_addition.py ..                                                                                   [100%]
 
============================================ 2 passed in 0.06s =============================================

TIP

These two dots after test edition dot py in the terminal also indicate that both tests passed.

If you change -1 to -2:

=========================================== test session starts ============================================
platform win32 -- Python 3.12.2, pytest-8.1.1, pluggy-1.5.0
rootdir: C:\Path
collected 2 items
 
test_addition.py .F                                                                                   [100%]
 
================================================= FAILURES ================================================= 
_________________________________________________ test_sub _________________________________________________ 
 
    def test_sub():
>       assert addition.sub(4, 5) == -2
E       assert -1 == -2
E        +  where -1 = <function sub at 0x000002A6C666DB20>(4, 5)
E        +    where <function sub at 0x000002A6C666DB20> = addition.sub
 
test_addition.py:8: AssertionError
========================================= short test summary info ==========================================
FAILED test_addition.py::test_sub - assert -1 == -2
======================================= 1 failed, 1 passed in 0.18s ========================================

pass

I can also write these tests without the assert statement and just add pass. This passes the test regardless of any errors.

NOTE

I used an equality operator here but I could have used less than, greater than, or keywords such as is, in or not in. All that matters, is that the assert statement gets a boolean value.

TIP

You can add multiple assert too. All the assert statements within a given function should return a true value for the test to pass.

TIP

Using the test underscore prefix for both the file name as well as the function name is good practice.

Testing over a specific function

If I want to run my test over a specific function, I just add a double colon at the end of the file name and then I write the function name.

python -m pytest test_addition.py::test_add

PyTest cheat sheet

Installation

Run the following on the Terminal:

pip3 install pytest (Mac)

pip install pytest (Windows)

# Nomenclature

Add suffix ‘test_’ to the file that needs to be tested.
Add suffix ‘test_’ to the functions to be tested.

Running pytest

This is the command that has to be executed on the Terminal prompt:

python3 -m pytest test_file.py

Alternative method

py.test will look for the keyword test and run the tests over those files and functions automatically.

py.test test_file.py

When you run pytest for a specific function add :: to run a specific function in a given file.

Flags used

For example, -v is the flag:

python3 -m pytest abc.py -v

Some other flag options are:

-v for verbose
-q quiet mode
-s allows the print statement inside the functions to be executed
-x is to flag the tests to stop execution after first failure
-m is used to mark a specific function
-k is a flag for searching and running tests with a specific keyword
—tb is to disable the traceback code of errors
—maxfail n specifies maximum number of test fails allowed

Tips

The rule of thumb is that the assert statement looks for a Boolean result. You can use in, not in, is, <, >, other than == to check Boolean values.
You can add multiple assert statements inside a single test function.

Additional reading

Fixtures

Fixtures are a type of function that is applied to functions to be tested. These functions must run before that test is executed. The purpose of fixtures is to supply data from multiple sources including URLs and databases to the test before running the test. Fixtures are used in cases where code repeats initialization.

Format:

@pytest.fixture

Markers

Markers are used to ‘mark’ specific functions to be executed by letting users create special names. There are many built-in markers such as xfail, xpass, skip and so on.

They follow a format such as:

@pytest.mark.<markername>

For example:

@pytest.mark.alpha

Running the specific marked test in the command line can be done with the following command:

pytest -m <markername> -v

Which will be as follows for a marker called alpha.

pytest -m alpha -v

Test-driven development (TDD)

Software development is time sensitive and in the process, developers often find testing gets squeezed into the time remaining after the code is written. This doesn’t leave enough time to test and can lead to the software containing bugs that need to be dealt with over time.

TDD

Test-driven development or TDD is an alternative programming practice in which the tests are written first and the code is written so that the tests will pass.

This difference from the convention of first writing the code and testing the application progressively.

TDD follows an iterative approach beginning with writing the test cases. The initial work requires feature and test planning by the team.

Standard steps

You write a test for a feature that fails.
You write code in accordance with the tests.
Step 3 requires that you run the test expecting them to fail.
You evaluate the error and refactor the code as needed.
You rerun the process.

This process is also called the red-green-refactor cycle.

Red implies the failed tests and green shows the passing tests after refactoring. The whole point of following this cycle is to fail the tests and rewrite until you don’t have to. A feature is complete when everything is green and you no longer need to rerun.

You can use a package libraries such as Pytest when automation becomes a priority. Pytest only requires writing functions while a unit test requires classes. This means that pytest has the advantage of being easier because it requires less effort.

Advantages of TDD

Writing tests first and refactoring code based on it ensures the test cover the code. You can now write tests with a specific feature and outcome in mind.
The need for such forecasting provides clarity from the beginning.
The forecasting also plays a role in integrating different components where you add new features and interfaces in accordance with the components that are already there.
Working in cycles over the code gives us develop a competence that easily refactor in terms of additional changes.

Overall, smaller code with early bug fixes, code extensibility, and eventual ease of debugging are the primary reasons TDD is growing in acceptance.

Differences between TDD and traditional testing

The main difference is that with TDD, the requirements and standards are highlighted from the beginning, making it purposeful.

Modern-day development often employs a combination of both these forms of testing, depending on the different parts and stages of the software development cycle.

There are several subtypes and variations of test-driven development. These include:

behavior-driven acceptance
test-driven
scaling
developer test-driven development

Applying TDD

In conventional testing, you follow the process of writing code and then writing test cases to ensure the integrity of that code. In test-driven development or TDD, the approach is the other way round, and test case is where you must begin your thinking.

Steps

Write test cases with some functionality in mind.
Write code in accordance with the test cases ensuring that they pass.
Refactor code in case the tests fail.

Example

Let’s say I’m checking students enrollments for a class exam against the list of names that I already have. To keep things straight forward, I’m going to use a Python list with names instead of a database.

I want to make sure that the names I enter are on the list. I also want to ensure data integrity, which means that I must be sure that the names are entered in the correct format.

# test_findstring.py
 
from curses.ascii import isdigit
import findstring
import pytest
 
def test_ispresent():
    assert findstring.ispresent("Al")

# findstring.py
 
def ispresent(person):
    names = ["Al", "Bo", "Chi", "Ma"]
    if person in names:
        return True
    else:
        return False

I might not want numeric characters in the names.

# test_findstring.py
 
from curses.ascii import isdigit
import findstring
import pytest
 
def test_ispresent():
    assert findstring.ispresent("Al")
 
def test_nodigit():
    assert findstring.nodigit("N7")

# findstring.py
 
def ispresent(person):
    names = ["Al", "Bo", "Chi", "Ma"]
    if person in names:
        return True
    else:
        return False
    
def nodigit():
    if person.isalpha():
        return True
    else:
        return False

Additional resources

The following resources will be helpful as additional references in dealing with different concepts related to the topics we have covered in this module.

Test TDD Module Package framework library Python

Previous one → 9.Popular Packages, Libraries and Frameworks | Next one → 1.Introduction to Version Control

📚Back-end Notes

Explorer

10.Testing Tools

What is testing?

Benefits of testing

Best Practices for Effective and Reusable Software Testing

Testing lifecycle

Test Cases

A well written test case

Types of testing

main levels or categories of testing

White box test

Black box test

Functional tests

Non-functional tests

Maintenance tests

The four main levels of testing

Unit / Component testing

Integration testing

System testing

Acceptance testing

Test automation packages

Steps in test automation

Important Python testing framework

Unittest

Pytest

Functional

Parameterized

Parallel

Robot

Selenium

Writing tests with PyTest

Testing over a specific function

PyTest cheat sheet

Installation

Running pytest

Flags used

Tips

Additional reading

Fixtures

Markers

Test-driven development (TDD)

Standard steps

Advantages of TDD

Differences between TDD and traditional testing

Applying TDD

Steps

Example

Additional resources

Graph View

Table of Contents

Backlinks