Software Testing and CI/CD#

Video: CS50W - Lecture 7 - Testing and CI/CD This uses Django rather than Flask, but the same concepts apply.

Introduction to Software Testing#

In the field of software development, ensuring the quality and reliability of your application is paramount. This is where software testing comes in. Testing is the process of evaluating a system or its components with the intent to find whether it satisfies the specified requirements or not. This process is integral to identifying and fixing defects before the product reaches the end-user, thus enhancing user experience and satisfaction.

There are several types of software testing, each with its own purpose and place in the software development life cycle. Here are a few of them:

  • Unit Testing: This is a level of software testing where individual components of a software are tested. The purpose is to validate that each unit of the software performs as designed.

  • Integration Testing: This type of testing involves combining individual units and testing them as a group. The purpose here is to detect faults in the interaction between integrated units.

  • System Testing: In system testing, the complete system is tested as per the requirements. It’s a series of different tests whose sole purpose is to evaluate the system’s compliance with the defined requirements.

  • Acceptance Testing: This is a level of software testing where a system is tested for acceptability. The purpose of this test is to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery.

In this chapter, we will focus on unit testing and continuous integration. Unit testing is a crucial part of maintaining code quality and ensuring that individual components of the software behave as expected. Continuous Integration (CI), on the other hand, is a practice that involves regularly integrating code changes into a shared repository to catch integration-related bugs early.

As you navigate through this chapter, you will learn about these concepts in depth, along with hands-on examples of automated testing using GitHub Actions, a powerful tool for implementing CI/CD pipelines.

Unit Testing#

Unit testing is a crucial aspect of software development. It involves testing individual components or functions of a program to verify that they behave as expected. The “unit” in unit testing refers to the smallest piece of software that can be independently tested, often a function or method in the code.

Definition and Purpose of Unit Testing#

Unit testing serves several purposes:

  • It helps ensure that each part of your application works as expected, which can help you catch bugs early in the development process.

  • It encourages you to write more modular, decoupled code, since such code is easier to unit test.

  • It can serve as a form of documentation, by showing other developers how a function or method is intended to be used.

  • It can make the process of refactoring or updating code safer, by ensuring that changes don’t break existing functionality.

Examples of Unit Tests for Web Development#

Here are examples of unit tests in two popular web development languages: JavaScript and Python.

In JavaScript, using the Jest testing framework:

const sum = require('./sum');

test('adds 1 + 2 to equal 3', () => {
  expect(sum(1, 2)).toBe(3);
});

In Python, using the PyTest framework:

def sum(a, b):
    return a + b

def test_sum():
    assert sum(1, 2) == 3

Testing Frameworks and Tools#

There are various testing frameworks and tools available for different languages. Here are a few examples:

  • JavaScript: Jest, Mocha, Jasmine

  • Python: PyTest, unittest

  • Ruby: RSpec, Minitest

  • Java: JUnit, TestNG

Writing Effective Unit Tests#

When writing unit tests, keep these best practices in mind:

  • Assertion: Each unit test should assert something about the result of a function. If the test doesn’t have an assertion, it’s not checking anything.

  • Setup and Teardown: Some tests need to set up conditions before they run and clean up after they finish. Most testing frameworks provide setup and teardown methods for this purpose.

  • Independence: Each test should be independent and able to run in isolation. A test should not rely on the results of other tests or modify global state.

Test-Driven Development (TDD)#

Test-Driven Development (TDD) is a software development methodology that involves writing the tests before writing the code. The process is often described as “Red, Green, Refactor”: first write a test that fails (red), then write the code to make it pass (green), then refactor the code while ensuring that the test still passes.

Continuous Integration and Testing Tools#

Continuous Integration (CI) is a practice in software engineering where all developers’ working copies are merged to a shared mainline multiple times a day. It typically triggers an automated build with testing and is often used in conjunction with automated unit tests written through the practices of test-driven development. A build server compiles the code periodically, running tests and implementing other quality control processes to improve software quality and delivery time.

CI is a part of continuous delivery or continuous deployment, often intertwined in a CI/CD pipeline. Continuous delivery ensures the software checked in on the mainline is always in a state that can be deployed to users, while continuous deployment fully automates the deployment process.

3.1 GitHub Actions#

GitHub Actions offers CI workflows that can build the code in your repository and run your tests. These workflows can run on GitHub-hosted virtual machines, or on machines that you host yourself. You can configure your CI workflow to run when a GitHub event occurs, such as when new code is pushed to your repository, on a set schedule, or when an external event occurs using the repository dispatch webhook. GitHub runs your CI tests and provides the results of each test in the pull request, so you can see whether the change in your branch introduces an error.

3.2 TestProject#

TestProject offers Web and Mobile Test Recorders that allow for the creation of tests with no coding skills required. The Test Recorders can identify elements within applications, record steps interactively, and allow debugging and replay of tests. The tests can be exported to Selenium or Appium code for further customization by developers.

3.3 Selenium IDE#

Selenium IDE is an open-source record and playback test automation tool for the web. It’s a simple, turn-key solution that quickly authors reliable end-to-end tests for any web app. It offers easy debugging with rich IDE features and cross-browser execution. Selenium IDE records multiple locators for each element it interacts with, so if one locator fails during playback, the others will be tried until one is successful. It also allows for test case reuse and has an extensive control flow structure.

GitHub Actions for CI/CD#

GitHub Actions is a powerful automation tool built into GitHub, allowing you to define custom software development life cycle (SDLC) workflows directly in your GitHub repository.

Introduction to GitHub Actions#

GitHub Actions makes it easy to automate all your software workflows. You can build, test, and deploy your code right from GitHub. Additionally, you can assign tasks, such as triaging issues and managing pull requests, to be triggered by specific events within your repositories.

How GitHub Actions Enable CI/CD#

GitHub Actions provides a set of features that facilitate Continuous Integration (CI) and Continuous Deployment (CD). You can create workflows that run on any GitHub event (e.g., code pushes, pull requests, issue comments, etc.). These workflows are composed of one or more jobs and can be set to run automatically on specific events. For example, you can set a workflow to run tests whenever code is pushed to your repository, ensuring that all tests pass before any code is merged into the main branch.

Setting up a GitHub Actions Workflow#

Setting up a GitHub Actions workflow involves creating a YAML file within the .github/workflows directory in your repository. The YAML file defines the workflow and includes information such as the events that trigger the workflow, the jobs that the workflow should execute, and the steps within each job.

Here’s a simple example of a workflow that runs tests whenever code is pushed:

name: Run Tests

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Use Node.js
      uses: actions/setup-node@v2
      with:
        node-version: '14'

    - name: Install Dependencies
      run: npm ci

    - name: Run Tests
      run: npm test

In this example, the workflow is triggered on a push event. It then checks out the code, sets up Node.js, installs the dependencies, and finally runs the tests.

Examples of Automated Testing with GitHub Actions#

With GitHub Actions, you can set up complex workflows that automate your entire testing process. For example, you could create a workflow that:

  • Runs unit tests on every push

  • Runs integration tests on every pull request

  • Deploys the application to a staging environment whenever changes are merged to the main branch

  • Deploys the application to production whenever a new release is created

Monitoring and Troubleshooting GitHub Actions#

GitHub Actions provides detailed logs for each workflow run. You can use these logs to understand what happened during a workflow run and to troubleshoot failures. GitHub also provides several status badges that you can add to your repository’s README to show the status of your workflows.

In addition, you can set up notifications to be alerted of workflow failures. This can be done either through GitHub’s built-in notification system or through external tools like email or Slack.

In conclusion, GitHub Actions is a robust tool for automating your testing and deployment processes. By incorporating it into your projects, you can ensure that your code is always tested and ready for deployment.

Integration Testing and End-to-End (E2E) Testing#

Following unit testing, there are other important types of testing that should be incorporated into your testing strategy. This section explores integration testing and end-to-end testing, both vital in ensuring the correct interaction between different parts of your web application.

Definition and Importance of Integration Testing and E2E Testing#

Integration Testing is a level of software testing where individual units are combined and tested as a group. The purpose of this level of testing is to expose faults in the interaction between integrated units.

End-to-End Testing (E2E Testing), as the name suggests, validates the software system along with its integration with external interfaces. The purpose of this level of testing is to simulate real user scenarios and validate the system under test and its components for integration and data integrity.

Both integration testing and end-to-end testing are key to ensuring the correct functioning of your application as a whole, as they test how the different parts of your application interact with each other and with external systems.

How Integration Testing and E2E Testing Fit Into CI/CD#

In a Continuous Integration/Continuous Deployment (CI/CD) pipeline, integration tests and end-to-end tests are typically run after unit tests.

  • Integration tests are run after unit tests and are used to catch issues that might arise when different units of code interact with each other.

  • End-to-end tests are usually run last, after all other tests have passed. They simulate user behavior and workflows within the application and verify that the entire application is working as expected from start to finish.

Tools for Integration and E2E Testing#

There are many tools available for performing integration and end-to-end testing. Here are a few examples:

  • Cypress: A next generation front end testing tool built for the modern web. Cypress enables you to write all types of tests including end-to-end tests, integration tests, and unit tests in your JavaScript apps.

  • Selenium: A suite of tools to automate web browsers across many platforms. Selenium provides a way for developers to write tests in a number of popular programming languages such as C#, Java, Python, and Ruby.

Examples of Integration and E2E Tests#

Here’s a simple example of an integration test using Jest (a JavaScript testing framework):

const calculator = require('./calculator');

test('test addition and subtraction', () => {
  const sum = calculator.add(1, 2);
  const diff = calculator.subtract(sum, 1);
  expect(diff).toBe(2);
});

This test checks if the add and subtract functions of the calculator module are working correctly together.

In conclusion, integration and end-to-end testing play a critical role in ensuring your application works as expected when all its parts come together. They should be an integral part of your testing and CI/CD strategy.

Best Practices for Testing and CI/CD#

In this section, we will cover some of the best practices you should adopt in your testing and CI/CD processes to ensure you are effectively catching bugs and delivering quality software.

Importance of Code Coverage#

Code coverage is a metric that can help you understand the degree to which your source code is executed during a test. It is a useful tool for finding untested parts of a codebase. Code coverage doesn’t guarantee that your code is completely tested - a code segment might be covered by a test, but that doesn’t mean the test cases are comprehensive or correct. However, a low code coverage percentage is often indicative of insufficient testing.

There are several types of code coverage, including:

  • Statement Coverage: Has each point of execution been invoked?

  • Branch Coverage: Has each outcome of each decision point (e.g., if statement) been executed?

  • Function Coverage: Has each function (or subroutine) in the program been called?

Strategies for Maintaining a Healthy CI/CD Pipeline#

  • Keep your pipelines fast: A slow-running pipeline can become a bottleneck for deployment or even development if tests take too long to run. Consider strategies like running tests in parallel, optimizing your tests, and avoiding unnecessary operations to keep your pipelines running quickly.

  • Fail fast: If a step in your pipeline fails, it should fail as early as possible. There’s no point in deploying an application or running further tests on an application that has already failed a test.

  • Make build results visible: Everyone on the team should be able to easily see the state of the build and the changes that have been made to the code.

  • Automate everything: The whole point of a CI/CD pipeline is to automate your testing and deployment processes, so as a rule of thumb, automate everything you can.

The Role of Testing in Agile and DevOps#

In Agile and DevOps, testing is integrated into the development process, rather than being a separate phase. This approach, known as Shift Left Testing, allows for faster feedback loops and helps catch issues earlier in the development process.

Continuous testing is a key part of this approach, with automated tests being run regularly throughout the development process, rather than at a separate testing phase. This allows for quicker detection and resolution of bugs, and helps ensure that the software is always in a releasable state.

In conclusion, implementing best practices in your testing and CI/CD processes can help you catch bugs more effectively, deliver higher quality software, and make your development process more efficient.

Conclusion#

Testing plays a crucial role in software development, particularly in web development where applications are accessed by a variety of users across different platforms and devices. Incorporating a comprehensive testing strategy, including unit tests, integration tests, and end-to-end tests, ensures that your software works as expected and provides a good user experience.

Continuous integration and continuous deployment (CI/CD) practices, facilitated by tools like GitHub Actions, are key to maintaining a healthy development pipeline. By automatically running your tests every time code is pushed to the repository, you ensure that any issues are caught and addressed early, preventing bugs from making their way to the production environment.

In this chapter, we’ve explored the fundamentals of testing and CI/CD, looked at examples of how to implement these practices using various tools, and discussed the importance of these concepts in modern web development. As you continue to develop your web development skills, remember to incorporate testing and CI/CD practices into your projects. Not only will this lead to higher quality software, but it will also make you a more effective and sought-after developer.

Resources#

Link and commentary from Mark Liffiton’s Software Dev Class:

  • Unit Tests: Flask 0.12 Documentation. “Test Adding Messages” section

  • Continuous Integration (CI): What is Continuous Integration?. Explains the basic idea of CI and why its valuable.

  • Continuous Integration and Github Actions: About Continuous Integration. Explains a tiny bit about how to do CI using Github Actions (one tool of many for CI).

  • Additional motivation: Continuous Integration by Martin Fowler(2006). The tools mentioned are out of date, but the ideas are all very relevant. You can stop when you hit “Keep the Build Fast,” and skip down to “Benefits of Continuous Integration.” The skipped parts are worth reading, but they are less relevant to our specific case in this class.

How to Set Up Github Actions To set up Github Actions for your project, follow these instructions carefully. They need to be done once, by one person, in any given repository.

Create a .github folder at the top level of your repository, and then create a workflows folder inside that.

Then, create a new file .github/workflows/python_tests.yml with the following contents:

This workflow will install Python dependencies, run tests, and lint with flake8. For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

name: Python linting and tests

*Triggers the workflow on push or pull request events on: [push, pull_request]

jobs: build:

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.10
  uses: actions/setup-python@v4
  with:
    python-version: "3.10"
- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    python -m pip install flake8 pytest
    python -m pip install flask
    if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
  run: |
    # stop the build if there are Python syntax errors or undefined names
    flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
- name: Test with pytest
  run: |
    pytest your_unit_tests_script.py

Be sure to include the correct name for your unit tests script.

Commit that new file, then check the Actions tab in your repository’s page on GitHub to ensure that the action is being triggered

As soon as a commit containing the new file is pushed to Github, Github should automatically detect it and start running your tests. It will report the results to you 1) in the web interface, 2) via indicators in Github’s interface, and 3) via email if the workflow fails.

Feel free to look into GitHub actions to see what other sorts of things can be automated. There are lots of possibilities.

Unit Tests for Flask Applications#

Follow the unit tests code and the explanation in the Flask v0.12 documentation. Where that code has import flaskr, instead write import app as flaskr. Explanation: Their code assumes the application is in flaskr.py. Yours is in app.py, so you will import from app instead, and you can use the as flaskr to give it that name in this file so the rest of the example code will work unchanged. Omit any tests of the login/logout system, and omit any code related to logging in from other unit tests. I removed the login/logout system from the base code on which you are building. Stop after the “Test Adding Messages” section. For the assignment, you will need to add additional test methods to cover more functionality. To run unit tests in PyCharm: First, right-click anywhere in the code editor with the tests file open, and select the “Run ‘Unittests in…’” entry with a green arrow to run the tests in the open file. After doing that once, a new temporary entry should appear in the Configurations drop down menu at the top of the screen (where “run” and “initdb” are). You should be able to choose to save that configuration so it is always available to be run (in addition to the “Run” and “Initdb” configurations). When running tests in PyCharm, pay attention to the special tests interface it produces in the lower section of the IDE. You may wish to watch a video or read a quick tutorial on PyCharm’s testing interface to familiarize yourself with it.