Pipelines: build → test → scan → artifact → deploy

A pipeline is an automated assembly line for your software: a commit goes in one end, and a release comes out the other, passing through a fixed series of stages on the way. Each stage either passes (and the change flows on) or fails (and the change stops, loudly, before it can hurt anyone). This lesson builds up that assembly line stage by stage, then nails down three terms that are constantly muddled — Continuous Integration, Continuous Delivery, and Continuous Deployment — which are not synonyms.

Why automate the path at all?

Picture shipping a change by hand: you run the tests on your laptop (or forget to), build the app, copy files to a server, restart something, and hope. Do that across a team and you get the exact problems IaC solved for infrastructure — it's slow (a human in the loop for every release), error-prone (steps forgotten, environments inconsistent), and unrepeatable (it worked on your machine). The fix is the same as for infrastructure: write the steps down as code and let a machine run them identically every time. That codified, automated path is the pipeline.

The stages of a pipeline

A pipeline is a sequence of stages; each stage contains one or more jobs (units of work that can run in parallel), and the pipeline is kicked off by a trigger (typically "a commit was pushed" or "a pull request was opened"). The canonical order of stages is:

• Build — Turn source code into something runnable: compile it, install dependencies, assemble a container image. If the build fails, nothing downstream runs.
• Test — Run automated checks against the freshly built code. Unit tests verify small pieces in isolation; integration tests verify pieces working together. Fast, reliable tests here are the foundation of the whole pipeline — they're the gate that catches bugs in seconds instead of in production.
• Scan — Inspect the code and its dependencies for problems a test won't catch: known security vulnerabilities, risky licenses, leaked secrets, and (increasingly) generating a bill of materials of everything inside. We dedicate lesson 5.3 to this.
• Artifact — Package the result into an artifact: the single, built, versioned thing the pipeline produces — for a containerized app, a container image. The golden rule, expanded in 5.3 and 5.4: build the artifact once, then promote that exact artifact through every environment. Never rebuild per environment.
• Deploy — Take the artifact and make it run in a target environment (dev, staging, production). How this happens — pushed by the pipeline, or pulled by an in-cluster agent — is the GitOps question of lesson 5.5.

:::note Trigger, stage, job — the vocabulary A trigger starts a pipeline (a push, a PR, a schedule, a manual click). A stage is a phase that must pass before the next begins (build, then test, then scan…). A job is a concrete task within a stage; jobs in the same stage often run in parallel to save time (e.g. run unit tests and lint simultaneously). You'll see these three words in every CI tool, whatever the YAML looks like. :::

The crucial trio: CI vs CD vs CD

"CI/CD" hides three distinct ideas. They form a ladder — each builds on the one before — and the second "CD" is ambiguous on purpose, because it stands for two different things.

• Continuous Integration (CI) — Developers merge their work into a shared main branch frequently (ideally many times a day), and every merge automatically triggers a build and test run. The point is to catch integration problems early and small, instead of discovering on release day that ten people's branches don't fit together. CI is about the build + test loop on a shared branch. (Lesson 5.2 covers the branching discipline that makes this work.)
• Continuous Delivery (CD) — Every change that passes CI is automatically taken all the way to a release-ready state — built, tested, and deployed to staging — so that shipping to production is a single, safe, manual click whenever the business wants. You are always ready to ship; a human chooses when.
• Continuous Deployment (CD) — Goes one step further: there is no manual gate. Every change that passes the pipeline is automatically deployed to production. The release is fully automated end to end.

The difference between the two CDs is exactly one thing: the manual approval before production. Delivery keeps a human in the loop for the final go; deployment removes it. Both require that everything up to that point is automated and trustworthy.

:::tip Which "CD" should a team use? Continuous Delivery is the safe default and the right goal for most teams: full automation up to production, with a human pressing the final button. Continuous Deployment (no human gate) is a powerful end state, but it demands very high confidence in your tests, scans, and especially your progressive-delivery and automatic-rollback machinery (lesson 5.6) — because there's no human to catch a bad change before it reaches users. Earn deployment by first being excellent at delivery. :::

Tracing a pipeline run

Let's trace one change through a real pipeline, the way a CI tool would execute it:

1. Trigger. A developer opens a pull request. The CI system (e.g. GitHub Actions, GitLab CI, Jenkins, or Tekton) sees the event and starts the pipeline.
2. Build. It checks out the code and runs the build — say, docker build — producing a container image. Build fails? The PR is marked red and stops here.
3. Test. It runs unit and integration tests in parallel with linting. Any failure stops the change and reports exactly which check failed.
4. Scan. It scans the image and dependencies for known vulnerabilities and generates an SBOM. A critical vulnerability can fail the gate.
5. Artifact. All green, so it tags the image with an immutable version and pushes it once to a registry. This exact artifact is now the only thing that will be deployed anywhere.
6. Deploy. On merge to main, it deploys that artifact to staging automatically (delivery), and — depending on the model — either waits for a click (delivery) or proceeds to production on its own (deployment).

Notice that a failure at any stage halts the change before the next one runs. That's the whole safety value: problems are caught at the cheapest, earliest possible point.

:::tip Durable vs dated The pipeline model — staged gates, build-test-scan-artifact-deploy, and the CI/Delivery/Deployment distinction — is durable; it has been stable for over a decade and applies whatever tool you use. The specific tool and its YAML (GitHub Actions vs GitLab CI vs Jenkins vs Tekton) are dated and churn. Learn the stages and the trio; the syntax is a lookup. :::

Pipeline-as-code, beyond "jobs run in parallel"

A pipeline isn't just a flat list of stages and parallel jobs — the pipeline definition is itself code, and mature teams apply the same DRY, cacheable, scalable discipline to it that they apply to application code. Four ideas turn copy-pasted YAML into a real platform capability:

• Reusable workflows & templates. Instead of pasting the same fifty lines of build-test-scan YAML into every repo, you define the pipeline once and call it from many. In GitHub Actions this is a reusable workflow (a whole workflow another repo invokes with uses:) or a composite action (several steps packaged as one reusable step); GitLab CI has include: and extends:; Jenkins has shared libraries. The payoff is the same as a Terraform module (Chapter 3) or a golden path (Chapter 7): the security scan, the SBOM step, the deploy logic live in one governed place, and a fix or policy change propagates to every pipeline at once instead of being re-pasted (and forgotten) per repo.
• Build caching. A from-scratch build re-downloads every dependency and recompiles everything on every run — slow and wasteful. Caching stores the expensive, rarely-changing parts (downloaded packages, compiled outputs, Docker layers) keyed on a fingerprint (e.g. a hash of the lockfile) and restores them on the next run, so only what actually changed is rebuilt. Done well it turns a ten-minute build into a one-minute one. The durable rule: cache on a key that changes exactly when the cached thing should change — too loose a key serves stale artifacts, too tight a key never hits.
• Ephemeral build runners. A runner (or agent) is the machine that actually executes a pipeline's jobs. The modern default is ephemeral: a fresh, throwaway environment is spun up per job and destroyed afterward. This guarantees a clean, identical starting state every time (no leftover files or state from a previous build leaking in — the "works on the build server but not from scratch" bug) and gives a tidy security boundary, since nothing persists between untrusted builds.
• Self-hosted runners. Hosted runners (GitHub/GitLab's own fleet) are the easy default, but teams reach for self-hosted runners — runners they operate on their own infrastructure — when they need things the hosted fleet can't give: access to a private network or internal services, special hardware (lots of RAM, GPUs for ML builds), specific compliance/residency requirements, or cheaper sustained throughput at scale. The trade-off is that you now own patching, scaling, and isolating those runners (a self-hosted runner exposed to untrusted pull requests is a real attack surface). Many teams combine both: ephemeral self-hosted runners — autoscaled, throwaway machines on their own infra — to get isolation and control at once.

:::note These are the same patterns, one layer up Reusable workflows are modules for pipelines. Caching is the build-once instinct applied within a run. Ephemeral runners are immutable, cattle-not-pets infrastructure (Chapter 4) applied to the build fleet. You already know the underlying ideas — pipeline-as-code is where they show up in CI. :::

Common pitfalls

• Confusing the two CDs. "We do CD" is ambiguous. Be precise: are you always ready to ship (delivery), or do you automatically ship (deployment)? They demand very different safety nets.
• Copy-pasted pipeline YAML. Re-pasting the same build/test/scan steps into every repo means a fix or a new security gate has to be applied N times and inevitably drifts. Factor shared logic into reusable workflows / composite actions / shared libraries.
• No build cache (or a broken one). Uncached builds are needlessly slow; a cache keyed too loosely silently serves stale dependencies. Cache on a fingerprint that changes exactly when the cached thing should.
• Stateful, shared runners. Reusing a long-lived runner lets state leak between builds and grows a security blast radius. Prefer ephemeral, per-job runners; isolate self-hosted ones from untrusted input.
• Slow or flaky tests. If the test stage is slow or unreliable, people stop trusting it and route around it — and the whole pipeline's value collapses. Fast, deterministic tests are the foundation, not a nice-to-have.
• Rebuilding per environment. Building a new artifact for staging and another for production means you never actually tested what you ship. Build once; promote that artifact (lessons 5.3–5.4).
• Treating deploy as the last manual step. Automating build and test but copying files to prod by hand keeps the slowest, riskiest part manual. The deploy stage belongs in the pipeline.

Why it matters

A pipeline is an automated assembly line that turns a commit into a release through staged gates — build → test → scan → artifact → deploy — where a failure at any stage stops the change at the cheapest possible moment. Hiding inside "CI/CD" are three distinct ideas: Continuous Integration (merge and test on a shared branch, often), Continuous Delivery (every passing change made ready to ship, human presses the final button), and Continuous Deployment (every passing change shipped automatically, no human gate). The single difference between the two CDs is the manual approval before production. Get this vocabulary exact and the rest of the chapter — branching, artifacts, environments, GitOps, progressive delivery — slots cleanly onto it. Next we look at the branching discipline that makes Continuous Integration actually continuous.

Next: Branching for CI: trunk-based development →