Continous Integration

Motivation

Continous Integration (CI) systems are built to ensure that input and output files of each teams are

  • Correct

  • Reproducable

  • Consistent with other teams

CI system is automatically triggered on

  • Main branch: the master branch of the codebase

  • A pull request on main branch

Workflows

Principles

Continous Integration system consists a number of workflows, each of which is designed to validate a specific aspect of the codebase. For the work of each team, there is at least 1 dedicated workflow.

Workflows can categorized in two types

Generation flow

Such type of workflow is designed to ensure that golden files (netlists, bitstreams, etc.) are reproduciable. A generation workflow consists of three steps:

  • Detect changes on input files, e.g., architecture files, IPs and related scripts.

    • If no changes detected, the workflow ends, since the golden outputs are not changed in a pull request

    • If any changes are detected, the workflow will continue to the next steps

  • Regenerate golden files by calling scripts. By the end of this step, it will compare the newly generated files with the golden reference (current branch) - If there are no changes, the workflow ends. - If any changes on golden reference are detected, this will error out. It means that the current golden reference are not reproduciable.

Warning

If any changes on golden references are detected, code review has to be enforced. Ensure that all the teams impacted agree on the changes.

Validation flow

Such type of workflow is designed to verify the correctness of golden files A validation workflow consists of three steps:

  • Detect changes on golden reference (some pull requests update golden references)

    • If no changes detected, the workflow ends. There is no need to validate the correctness of the golden reference (previous pull request should already do so).

    • If any changes are detected, the workflow will continue to the next steps

  • Run validation by calling scripts. For example, verification may call HDL simulations to verify the correctness of netlists. - If the new golden reference passes all the tests, this will end. - If the new golden reference fails any test, this will error out. It means that the current golden reference can not meet basic requirements.

Warning

If any validation flow failed, the pull request cannot be merged in general.

CI/CD setup

OpenFPGA implements CI/CD system using Github actions. The following figure shows the Actions implements flow. The source building is skipped if there are changes only in openfpga_flow or docs directory, in which case the docker image compiled for the latest master branch is used for running a regression.

digraph G { node [fontname = "Handlee"]; edge [fontname = "Handlee"]; Trigger [ label = "Action triggered" ]; masterCompare [ label = "Diff with current master" ]; Build [ label = "Changes only in\n openfpga_flow/doc?" shape = diamond ]; BuildDocker [ label = "Run build regression test\nBuild docker images" shape = box ]; PushDockersCond [ label = "Is merge\non master?" shape = diamond ]; PushDockers [ label = "Push docker Images\n(maintain compiled binary\nin docker + Example tasks)" shape = box ]; RunRegression [ label = "Run functional regression test" shape = box ]; Trigger ->masterCompare; masterCompare ->Build; Build -> BuildDocker [ label = "No" ]; BuildDocker -> PushDockersCond; edge[weight=0.5] Build -> RunRegression [ label = "Yes" ]; edge[weight=10] PushDockersCond -> RunRegression [ label = "No" ]; PushDockersCond -> PushDockers [ label = "Yes" ]; edge[weight=2] PushDockers -> RunRegression; { rank=same; PushDockersCond PushDockers; }; }

Build regression test

The OpenFPGA source is compiled with the following set of compilers.

  1. gcc-7

  2. gcc-8

  3. gcc-9

  4. gcc-10

  5. gcc-11

  6. clang-6

  7. clang-7

  8. clang-8

  9. clang-10

The docker images for these build environment are available on github packages.

Functional regression test

OpenFPGA maintains a set of functional tests to validate the different functionality. The test are broadly catagories into basic_reg_test, fpga_verilog_reg_test, fpga_bitstream_reg_test, fpga_sdc_reg_test, and fpga_spice_reg_test. A functional regression test is run for every commit on every branch.

How to debug failed regression test

In case the functional regression test fails, the actions script will collect all .log files from the task directory and upload as a artifacts on github storage. These artifacts can be downloaded from the github website actions tab, for more reference follow this article.

NOTE : The retention time of these artifacts is 1 day, so in case user want to reserve the failure log for longer duration back it up locally

Release Docker Images

ghcr.io/lnis-uofu/openfpga-master:latest

This is a bleeding-edge release from the current master branch of OpenFPGA. It is updated automatically whenever there is activity on the master branch. Due to high development activity, we recommend the user to use the bleeding-edge version to get access to all new features and report an issue in case there are any bugs.

CI after cloning repository

If you clone the repository the CI setup will still function, except the based images are still pulled from “lnis-uofu” repository and the master branch of cloned repo will not push final docker image to any repository .

In case you want to host your own copies of OpenFPGA base images and final release create a github secret variable with name DOCKER_REPO and set it to true. This will make ci script to download base images from your own repo packages, and upload final release to the same.

If you don not want to use docker images based regression test and like to compile all the binaries for each CI run. You can set IGNORE_DOCKER_TEST secrete variable to true.

Note

Once you add DOCKER_REPO variable, you need to generate base images. To do this trigger manual workflow Build docker CI images