roland/org-howto
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
org-howto/env/github-continuous-integration.org

16 KiB
Raw Blame History

github continuous integration

#

#

#

Motivation

We want to use a custom (docker) container with github actions, and prepare it using nix.

Several advantages:

  1. control over build software (e.g. gcc version).
  2. reproducibility – nix-to-container pipeline guarantees perfect reproducibility
  3. flexibility – container can include prepared software that isn't available on ubuntu
  4. size – container with no excess baggage

Contents

repos under https://github.com/Rconybea

scheme builder src-repo action-repo
scheme 1 ubuntu hello-example (same)
scheme 2 ubuntu docker-action-example (same)
scheme 3 container docker-nix-hello docker-action-example2
scheme 4 container docker-cpp-builder docker-action-example3

Progressive Implementation

We'll present several progressive CI-with-github examples, that converge on goals above.

Scheme 1 - github build using base o/s platform

Preliminaries

source for Scheme 1 is on github here: https://github.com/Rconybea/hello-example

Directory structure 
hello-example
+- .github
|  \- workflow
|     \- main.yml
+- CMakeLists.txt
\- hello.cpp
Source 
  // hello.cpp

  #include <iostream>

  using namespace std;

  int
  main(int argc, char * argv[]) {
      std::string subject = "World";

      if (argc > 1)
          subject = argv[1];

      cout << "Hello, " << subject << "!" << std::endl;
  } /*main*/
Cmake build files 
  # CMakeLists.txt

  cmake_minimum_required(VERSION 3.10)

  project(hello-example VERSION 1.0)

  enable_language(CXX)

  # write compile_commands.json for LSP
  set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE INTERNAL "")

  set(SELF_EXE hello-example)
  set(SELF_SOURCE_FILES hello.cpp)

  add_executable(${SELF_EXE} ${SELF_SOURCE_FILES})

  install(TARGETS ${SELF_EXE} DESTINATION bin)

  if(CMAKE_EXPORT_COMPILE_COMMANDS)
      # include otherwise-omitted system directories in compile_commands.json,
      # so LSP knows exactly what compiler is using
      set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES ${CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES})
  endif()
Build + Run 
$ cd hello-example
$ mkdir build
$ cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: /home/roland/proj/hello-example/build
$ make
[ 50%] Building CXX object CMakeFiles/hello-example.dir/hello.cpp.o
[100%] Linking CXX executable hello-example
[100%] Built target hello-example
$ ./hello-example
Hello, World!
Continuous Integration with Github

We need to do two things:

  1. setup a github repo holding our sources:

    $ cd hello-world
$ git remote -v
origin  git@github.com:rconybea/hello-example.git (fetch)
origin  git@github.com:rconybea/hello-example.git (push)

  2. configure github actions for that repo We add one file, hello-world/.github/workflows/main.yml

    name: c++ build with cmake

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

env:
  BUILD_TYPE: Release

jobs:
  build_job:
    runs-on: ubuntu-latest

    steps:
    - name: checkout self (hello-example)
      uses: actions/checkout@v3

    - name: configure self (hello-example)
      run: cmake -B ${{github.workspace}}/build -DCMAKE_INSTALL_PREFIX=${{github.workspace}}/local -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}}

    - name: build self (hello-example)
      run: cmake --build ${{github.workspace}}/build --verbose

    - name: test self (hello-example)
      working-directory: ${{github.workspace}}/build
      run: ./hello-example

    github will automatically trigger a build whenever new sources are pushed. build will run the contents of steps in order.

Scheme 2 - docker build using base o/s platform

Instead of building on github-provided base ubuntu platform, use github-built docker container.

Initially replacing c++ program with shell script, we'll restore it later.

Preliminaries

source for Scheme 2 is on github here: https://github.comt/Rconybea/docker-action-example

Build docker container (using resources from ubuntu base platform). Container:

  • invokes a shell script to say hello
  • can be used as a github action
Directory Structure 
docker-action-example
+- .github
|  \- workflows
|     \- main.yml
+- action.yml
\- entrypoint.sh
Source

entrypoint.sh:

  #!/usr/bin/env bash
  echo "Hello $1"
  time=$(date)
  echo "time=$time" >> $GITHUB_OUTPUT
Docker Container

DockerFile:

# base container image
FROM ubuntu:22.04

# copy files from repo to container filesystem
COPY entrypoint.sh /entrypoint.sh

# default startup executable
ENTRYPOINT ["/entrypoint.sh"]
Use Container as github action

toplevel action.yml describes a custom github action that operates by invoking docker image (automagically built from enclosed DockerFile)

action.yml:

name: 'Hello World'

description: 'Greet someone + record the time'

inputs:
  who-to-greet:
    description: 'Who to greet'
    required: true
    default: "World"

outputs:
  time:
    description: "time when greeting made"

runs:
  using: 'docker'
  image: 'DockerFile'
  args:
    - ${{ inputs.who-to-greet }}
Continuous Integration with Github

Configure github actions for repo https://github.com/Rconybea/docker-action-example;

File .github/workflows/main.yml

  on: [push]

  jobs:
    hello_world_job:
      runs-on: ubuntu-latest
      name: a job to say hello, perhaps fiercely
      steps:
        - name: hello world action step
          id: hello
          uses: rconybea/docker-action-example@v1
          with:
            who-to-greet: 'Iffy!  What are you doing??'
            # do something with output
            - name: get output time
              run: echo "The time was ${{ steps.hello.outputs.time }}"

In main.yml here:

  • rconybea/docker-action-example is our github repo,
  • v1 is target github tag

Scheme 3 - custom docker container via nix (no c++)

Preliminaries

Source for Scheme 3 uses two repos:

Directory structure 
docker-nix-hello
+- flake.nix
\- flake.lock    # automatically created by nix

docker-action-example2
\- .github
   \- workflows
      \- main.yml
Source

Prepare minimal custom docker container using a nix flake.

flake.nix:

{
  description = "hello world";

  # dependencies
  inputs = rec {
    nixpkgs.url = "github:nixos/nixpkgs/23.05";
  };

  outputs = { self, nixpkgs } :
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };

      hello_deriv = pkgs.writeShellScriptBin "entrypoint.sh" ''
        echo "Hello $1"
        time=$(date)
        echo "time=$time" >> $GITHUB_OUTPUT
      '';

      docker_hello_deriv =
        pkgs.dockerTools.buildLayeredImage {
          name = "docker-nix-hello";
          tag = "v1";
          contents = [ self.packages.${system}.hello
                       self.packages.${system}.bash
                       # for /bin/tail,  assumed by github actions when invoking a docker container
                       self.packages.${system}.coreutils ];

          config = {
            Cmd = [ "/bin/entrypoint.sh" ];
            WorkingDir = "/";
          };
        };

    in rec {
      packages.${system} = {
        default = docker_hello_deriv;

        docker_hello = docker_hello_deriv;
        hello = hello_deriv;

        bash = pkgs.bash;
        # for example,  github actions creates container with --entrypoint "tail",
        # so container must provide executable with that name in $PATH
        #
        coreutils = pkgs.coreutils;
      };
    };
}
Build Container + Upload to github

Container will be docker-nix-hello.

Steps:

  1. get github personal access token, so docker can send images to ghcr.io. on https://github.com/Rconybea:

    • visit profile (upper rhs of github webpage)
    • developer settings (bottom of sidebar)
    • personal access token
    • tokens (classic)
    • generate personal access token with scopes: read:packages write:packages delete:packages

  2. have docker login to github

      export CR_PAT=${token}
  echo $CR_PAT | docker login ghcr.io -u rconybea --password-stdin

  3. nix builds custom image (using docker-nix-hello/flake.nix)

      cd ~/proj/docker-nix-hello
  nix build

  4. load image into docker

      docker load <~/proj/docker-nix-hello/result

  5. tag image the way github expects: ghcr.io/${username}/${imagename}:${tag}

      docker image tag docker-nix-hello:v1 ghcr.io/rconybea/docker-nix-hello:v1

  6. push to github container registry (will show up at https://github.com/Rconybea?tab=packages)

      docker image push ghcr.io/rconybea/docker-nix-hello:v1
  7. make package public from package's settings link
Workflow Using Custom Container

in docker-action-example2/.github/workflows/main.yml:

  on: [push]

  jobs:
    hello_world_job:
      name: a job to say hello,  using separate docker container
      runs-on: ubuntu-latest
      container:
        image: ghcr.io/rconybea/docker-nix-hello:v1
        steps:
          - name: hello world action step
            id: hello
            run: /bin/entrypoint.sh 'Xioni!'
            - name: get output time
              run: echo "The time was ${{ steps.hello.outputs.time }}"

Scheme 4 - custom docker container via nix (with gcc)

Source for Scheme 4 in two repos:

Directory structure 
docker-cpp-builder
+- flake.nix
\- flake.lock    # automatically created by nix

docker-action-example3
+- .github
|  \- workflows
|     \- main.yml
+- Makefile
+- hello.cpp
Source

Prepare custom docker container to deliver build stack (gcc, cmake, …)

flake.nix:

{
  description = "docker c++ builder (using nix)";

  # dependencies
  inputs = rec {
    nixpkgs.url = "github:nixos/nixpkgs/23.05";
  };

  outputs = { self, nixpkgs } :
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };

      docker_builder_deriv =
        pkgs.dockerTools.buildLayeredImage {
          name = "docker-cpp-builder";
          tag = "v3";
          contents = [ self.packages.${system}.git
                       self.packages.${system}.cacert
                       self.packages.${system}.gnumake
                       self.packages.${system}.gcc
                       self.packages.${system}.binutils
                       self.packages.${system}.bash
                       # for /bin/tail,  assumed by github actions when invoking a docker contianer
                       self.packages.${system}.coreutils ];
        };

    in rec {
      packages.${system} = {
        default = docker_builder_deriv;

        docker_builder = docker_builder_deriv;

        git = pkgs.git;
        cacert = pkgs.cacert;
        gnumake = pkgs.gnumake;
        gcc = pkgs.gcc;
        binutils = pkgs.binutils;
        bash = pkgs.bash;
        coreutils = pkgs.coreutils;
      };
    };
}
Build Container + Upload to github

Instructions are the same as for Scheme 3, but using ~/proj/docker-cpp-builder instead of ~/proj/docker-nix-hello

  cd ~/proj/docker-cpp-builder
  git tag v1
  nix build
  docker load <~/proj/docker-cpp-builder/result
  docker image tag docker-cpp-builder:v1 ghcr.io/rconybea/rconybea/docker-cpp-builder:v1
  docker image push ghcr.io/rconybea/docker-cpp-builder:v1

After uploading package (docker image) for gcc builder appears here: https://github.com/Rconybea?tab=packages:

Workflow using Custom Container

in docker-action-example3/.github/workflows/main.yml:

  on: [push]

  env:
    # personal accesss token (using automatically-supplied GIT_TOKEN) with read access to public repos
    GIT_USER: rconybea

  jobs:
    build_job:
      name: compile hello world, using prepared docker container
      runs-on: ubuntu-latest
      container:
        image: ghcr.io/rconybea/docker-cpp-builder:v3
      steps:
        - name: checkout
          run:
            # not using usual checkout action: bc complications from within container
            GIT_SSL_NO_VERIFY=true git clone https://${{env.GIT_USER}}:${{env.GIT_TOKEN}}@github.com/rconybea/docker-action-example3.git

        - name: compile
          run:
            # make,g++ will run in container..
            make -C docker-action-example3

        - name: hello
          run:
            ./docker-action-example3/hello Roland
            #${{github.workspace}}/hello Roland

Note: in non-container build we can checkout code with something like:

  steps:
  - name: checkout
  - uses: rconybea/my-repo@my-git-tag

Unfortunately that doesn't work out-of-the-box when we use a container for build, because of a permissioning problem. Spent some time on various internet-advised workarounds, before settling on the solution above: including git in custom docker image, and checking source out from inside the container.

We need to set GIT_SSL_NO_VERIFY to prevent an obscure error (passed along from SSL) mentioning an `unrecognized scheme`. Presumably there's a problem authenticating github.com's certificate