The right arm of the V, turned into a build

01 — Premise

The process was right. The loop was too slow.

Nothing on the previous page is wrong engineering. Requirements were written, traced, and verified; the architecture was decomposed before implementation; three test stages ran in the correct order. The failure mode was throughput — 59 tethered trials to build the controller and three untethered runs to validate it, because every iteration was a person changing a constant, rebuilding, reflashing, and walking a track.

A team hitting the same wall does not fix it by working harder. It fixes it by making the verification arm of the V run without a human in it, so the human budget goes to the trials that need judgement.

02 — Built

What actually runs

A Jenkins controller, in Docker, configured entirely through JCasC — no setup wizard, no clicked configuration. Deleting the volume and running docker compose up reproduces the same server and the same job. Three stages run in parallel on every build.

[Verify] parallel branches: Static analysis, Traceability gate, Artefact integrity

  cppcheck  6/6 files checked 100% done
            4 findings, published through warnings-ng

  requirements:  17
  traced:        11
  untraced:      6  HR-F-02 HR-F-03 HR-F-05 HR-F-06 HR-F-07 SR-F-07
  broken links:  0
  coverage:      64.7%  (baseline 64.7%)
  PASS

  artefacts   5 models, entry point present in each generated directory

The traceability gate

This is the stage worth reading the source of. It pulls the seventeen requirement IDs directly out of docs/requirements-specification.xlsx — the specification itself is the input, not a transcription of it that can drift — resolves each against a committed map, and separates two different failures:

• A broken link — a requirement pointing at a file that has been renamed or deleted — always fails the build. This is the case that actually matters, because it is the one that happens silently.
• A coverage regression below the committed baseline fails the build. Incomplete coverage does not, which is why the baseline sits at the real number rather than at an aspirational one.

Six requirements are untraced deliberately. HR-F-02, HR-F-03, HR-F-05, HR-F-06 and HR-F-07 are satisfied by physical hardware, and SR-F-07 by material in the writeup — none of which is a file in this repository. Mapping them to something merely to raise the percentage would defeat the purpose of the gate.

03 — Mapping

Every right-arm box becomes a job

I annotated its V-model with a tool per stage — Excel, Visio, PlantUML, Simulink, Data Inspector. The retrofit keeps the same V and replaces "a person opens the tool" with "a job runs and reports".

Pipeline shape

Fast, hardware-free jobs run on every push and gate the merge. Anything that needs a physical board — or a long parameter sweep — runs behind that gate, on hardware nobody has to babysit.

The diagram below is the target shape, not the current one. Of the jobs drawn, trace and static exist and run; the remaining seven are specified and unbuilt. Nothing below the merge gate exists at all, because none of the hardware is wired up.

04 — Definitions

One build, three orchestrators

The same pipeline is expressed three ways on purpose. Automotive suppliers run GitLab and Jenkins far more than they run GitHub Actions, and the interesting engineering is in where the three disagree — hosted runners cannot hold a development board, GitLab and Jenkins disagree about where configuration lives, and only one of them treats "config as code" as the default rather than a plugin.

name: ci
on: [push, pull_request]

jobs:
  trace:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Requirement IDs referenced must exist
        run: python tools/trace_check.py docs/requirements.csv models/ tests/

  static:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: sudo apt-get install -y cppcheck clang-tidy
      - run: cppcheck --error-exitcode=1 --enable=warning,portability firmware/
      - run: tools/tidy.sh firmware/

  codegen-diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Generated C must match the committed baseline
        run: git diff --exit-code -- firmware/generated/

  compile:
    needs: [static, codegen-diff]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: arduino/setup-arduino-cli@v2
      - run: arduino-cli core install arduino:samd@1.8.14
      - run: |
          arduino-cli compile \
            --fqbn arduino:samd:nano_33_iot \
            --output-dir build/ firmware/
      - uses: actions/upload-artifact@v4
        with:
          name: firmware-${{ github.sha }}
          path: build/*.bin

stages: [trace, static, model, build, report]

default:
  image: python:3.13-slim
  interruptible: true

trace:requirements:
  stage: trace
  script:
    - python tools/trace_check.py docs/requirements.csv models/ tests/
  artifacts:
    reports:
      junit: reports/trace.xml

static:c:
  stage: static
  image: silkeh/clang:18
  script:
    - cppcheck --error-exitcode=1 --enable=warning,portability firmware/

model:mil:
  stage: model
  tags: [matlab]          # runner holding the MathWorks licence
  script:
    - matlab -batch "run_mil_suite"
  artifacts:
    paths: [reports/mil/]
    expire_in: 90 days

build:firmware:
  stage: build
  needs: [static:c, model:mil]
  script:
    - arduino-cli compile --fqbn arduino:samd:nano_33_iot --output-dir build/ firmware/
  artifacts:
    paths: [build/]

report:verification:
  stage: report
  script:
    - python tools/build_verification_table.py reports/ > public/verification.md
  artifacts:
    paths: [public/]

pipeline {
  agent any
  options {
    timestamps()
    buildDiscarder(logRotator(numToKeepStr: '20'))
  }

  stages {
    stage('Verify') {
      parallel {
        stage('Static analysis') {
          steps {
            sh '''
              mkdir -p ci/reports
              # Vendor headers are absent by design, so missingInclude
              # is suppressed - it would otherwise drown real findings.
              cppcheck --enable=warning,portability \
                       --suppress=missingInclude \
                       --suppress=missingIncludeSystem \
                       --inline-suppr \
                       --xml --xml-version=2 \
                       firmware/generated 2> ci/reports/cppcheck.xml
            '''
          }
        }

        stage('Traceability gate') {
          steps { sh 'python3 ci/trace_check.py' }
        }

        stage('Artefact integrity') {
          steps {
            sh '''
              fail=0
              # Every generated directory must carry an entry point.
              for d in firmware/generated/*/; do
                if [ ! -f "$d/ert_main.c" ]; then
                  echo "MISSING entry point in $d"; fail=1
                fi
              done
              [ -f PROVENANCE.md ] || { echo "PROVENANCE.md missing"; fail=1; }
              exit $fail
            '''
          }
        }
      }
    }
  }

  post {
    always {
      junit allowEmptyResults: true, testResults: 'ci/reports/traceability.xml'
      recordIssues enabledForFailure: true,
                   tools: [cppCheck(pattern: 'ci/reports/cppcheck.xml')]
      archiveArtifacts artifacts: 'ci/reports/*', allowEmptyArchive: true
    }
  }
}

jenkins:
  systemMessage: "concept-car - verification pipeline"
  numExecutors: 2
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: "${JENKINS_ADMIN_ID}"
          password: "${JENKINS_ADMIN_PASSWORD}"
  authorizationStrategy:
    globalMatrix:
      entries:
        - user:
            name: "${JENKINS_ADMIN_ID}"
            permissions: ["Overall/Administer"]

jobs:
  - script: >
      pipelineJob('concept-car-verification') {
        definition {
          cpsScm {
            scm { git { remote { url('/workspace') } branch('main') } }
            scriptPath('Jenkinsfile')
          }
        }
      }

05 — Scale

What changes when it is a team, not a person

A solo V-model and a team V-model fail differently. Alone, the risk is throughput. With five engineers, the risk is that two of them change the same model and nobody can tell.

Simulink models are binaries, and that is the hard problem

A .slx is a zip container. Git will happily store it, show binary files differ, and offer no merge. On a solo project this never surfaces. On a team it is the single largest source of lost work in model-based development.

• Lock, do not merge. Model files get lockable in .gitattributes and are checked out exclusively — the team convention is enforced by tooling, not by a wiki page.
• Decompose so locks are cheap. One monolithic model means one lock and a queue. Model references split the design so the encoder subsystem and the path logic can be edited at the same time.
• Review the generated C, not the binary. The codegen job makes every model change visible as a readable diff in firmware/generated/. That diff is what a reviewer actually reads on a merge request.

# Simulink and Stateflow artifacts are binary — no textual merge exists
*.slx   binary lockable
*.mdl   binary lockable
*.mlx   binary lockable
*.sldd  binary lockable

# Generated C is the reviewable surface of a model change
firmware/generated/** text eol=lf

Hardware is a shared, finite resource

One board, many engineers. The HIL agent is a single-executor Jenkins node with disableConcurrentBuilds(), so hardware jobs queue rather than collide. Scaling means adding labelled agents — hil-nano33, later hil-samd51 — and letting the label selector route work, not rewriting the pipeline.

Traceability stops being paperwork

In the project the verification table was written once, by hand, at the end. At team scale that document is stale the day after it is written. The retrofit inverts it: SR- IDs appear in commit trailers, model annotations, and test names, the trace job fails the build when an ID does not exist, and the table is regenerated from job results on every run.

feat(control): clamp integral term in distance PID

Windup on the straight section pushed the correction past the
motor duty limit, which showed up as overshoot at the first corner.

Verifies: SR-F-02, SR-F-04
Trial-data: reports/sweep/2026-09-05T14-22Z/

The merge gate, written down

06 — Result

What the retrofit is actually worth

Three of ten. That is the honest position, and it is the reason the first three rows are the ones worth reading — they are the rows where a claim can be checked against a build log rather than taken on trust.

The engineering conclusion of the project does not change: the V-model is the right shape for safety-critical embedded work, and it was applied correctly. What changes is that the right arm of the V stops being a thing one person does once, and starts being a thing the repository does continuously — which is the only version of it that survives contact with a team. The remaining seven rows are what finishing that argument costs.

← Back to the project and the V-model