Building a CI/CD Pipeline That Actually Works

Beyond the Hello World Pipeline

Every CI/CD tutorial starts the same way: push code, run tests, build image, deploy. It looks simple in a five-minute demo. But in practice, pipelines fail in ways that tutorials never prepare you for. Flaky tests, credential rotation, database migrations, rollback strategies — these are the real problems.

The Architecture I Default To

After building pipelines for projects of various sizes, I have settled on a pattern that works reliably:

┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐
│  Push    │──▶│  Lint &  │──▶│  Build   │──▶│  Deploy  │
│  to main │    │  Test    │    │  Image   │    │  Stage   │
└────────┘    └────────┘    └────────┘    └────────┘
                                                  │
                                              ┌──▼──┐
                                              │Manual │
                                              │Promote│
                                              │ to    │
                                              │Prod   │
                                              └───────┘

The key insight: separate staging and production deploys with a manual approval gate. Automated deploys to staging, manual promotion to production. This one decision has saved me more times than I can count.

Docker Images That Do Not Suck

Most Docker images I see in the wild are bloated, insecure, and slow to build. Here are the rules I follow:

• Multi-stage builds — build stage compiles, runtime stage only has the binary
• Distroless or scratch base — no package manager, no shell, smaller attack surface
• Pinned base image tags — never use latest in production
• Non-root user — run as a dedicated user, never as root

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o server .

FROM gcr.io/distroless/static-debian12
COPY --from=builder /app/server /server
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/server"]

Secrets Management

Never store secrets in environment variables baked into images. Use a proper secrets manager — HashiCorp Vault, AWS Secrets Manager, or even a simple encrypted file mounted at runtime. The principle is straightforward: secrets should exist only in memory during runtime, never on disk, never in version control, and never in build artifacts.

What I Wish I Knew Earlier

The biggest lesson: invest in your pipeline early. A well-built CI/CD pipeline pays compound interest. Every hour spent making builds faster, tests more reliable, and deploys safer comes back tenfold when you are pushing changes multiple times a day under pressure. Start simple, but start on day one.