Software Supply Chain Security: Cosign, SBOMs, and Admission Controls

TL;DR

Software Supply Chain Security: Cosign, SBOMs, and Admission Controls without the fluff: focus on outcomes, measure them, and stop pretending slides are progress.

“If you cannot answer ‘who built this image, with what, and when?’ you are not in control of your supply chain—you are just hoping nobody notices.”

Why Supply Chain Security?

Modern attacks target build systems, dependencies, and artifact registries. Trust must be earned at every stage—from source code to runtime. This guide shows how to implement practical controls that don’t slow teams down: sign artifacts, attach SBOMs, enforce provenance at admission, and keep dashboards that prove compliance.

Threats to Cover

• Malicious commits/dependencies: Attackers inject code upstream or in vendor packages.
• Compromised builders: CI runners or build agents tampered with to produce malicious artifacts.
• Registry drift: Tags overwritten, unverified images pushed directly.
• Deployment bypass: Operators kubectl apply YAML outside pipelines.
• Runtime tampering: Pods replaced with unsigned images post-deploy.

Controls must cover each link: code → build → artifact → deploy → runtime.

Core Controls

• Artifact signing & attestations: Prove who built what, when, and with which sources.
• SBOMs: Document dependencies for audits and CVE management.
• Admission enforcement: Verify signatures, provenance, and registries before pods run.
• Continuous monitoring: Detect drift, unsigned workloads, and aging SBOMs.

Sign Artifacts with Cosign

• Integrate signing into CI after builds and scans; fail the pipeline on signing errors.
• Use keyless signing (Sigstore Cosign + OIDC) to avoid managing private keys.
• Attach attestations (SBOM, SLSA provenance) to images so admission can reason about them.

Example CI step:

cosign sign --keyless registry.acme.io/my-app@sha256:...
syft packages registry.acme.io/my-app@sha256:... -o cyclonedx-json > sbom.json
cosign attest --keyless --predicate sbom.json registry.acme.io/my-app@sha256:...

Store SBOMs and provenance in the registry or object storage; keep references in Git for auditing.

Enforce in Admission

Kyverno or OPA Gatekeeper can verify signatures, attestations, and allowed registries.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-provenance
spec:
  validationFailureAction: enforce
  rules:
    - name: require-signed-and-sbom
      match:
        resources:
          kinds: [Pod, Deployment]
      verifyImages:
        - image: "registry.acme.io/*"
          keyless: true
          attestations:
          - name: sbom
            predicateType: https://spdx.dev/Document
      validate:
        message: "Image must include SBOM attestation and run by digest"
        pattern:
          spec:
            containers:
            - image: "*@sha256:*"

OPABased example (Gatekeeper) can use Rego to enforce digests, registries, or require annotations linking to SBOM stores.

Runbook: Secure from Commit to Runtime

1. Harden CI: Use ephemeral runners, restricted secrets, and artifact caching. Log builder identity, commit SHA, and build metadata.
2. Build → scan → sign: After tests, run vulnerability scans, generate SBOMs, sign images, and push to registry under immutable digests.
3. Publish attestations: Store SBOM/provenance attestations (Cosign, in-toto) in registry/OCI so they travel with the image.
4. CI policy checks: Run cosign verify or policy-controller in pipelines to catch issues before merge.
5. Admission: Enforce signatures, registry restrictions, SBOM presence, and digest use with Kyverno/OPA. Deny if anything missing.
6. Runtime monitoring: Use kubectl audit2rbac, Falco, or Sigstore policy-controller audit mode to detect pods running unsigned images. Alert teams.
7. Incident response: If signature verification fails, rollback to known good image; run cosign triangulate to find root cause; rotate credentials.

“Shipping unsigned images because ‘it’s faster’ is like leaving production SSH keys on a sticky note. It works—until it doesn’t.”

Operational Practices

• Registry hygiene: Enforce digests or immutable tags; restrict who can push to prod repos; garbage-collect vulnerable images.
• Vulnerability management: Tie SBOMs to scanners (Trivy, Grype, commercial SCA). Set SLAs for patching high/critical CVEs.
• Runbooks: Document how to troubleshoot signature failures, missing SBOMs, or policy blocks with clear contacts and rollback steps.
• Evidence collection: Keep Rekor log entries, attestation metadata, and Git commit references for audits.
• Exception workflow: Provide short-lived waivers with owner, justification, mitigation, and expiry. Track in dashboards.

Pitfalls

• Key sprawl: Too many private keys lead to leaks. Prefer keyless; if keys unavoidable, store in KMS/HSM, rotate, and log access.
• Allowing tags in prod: Tags drift; always deploy by digest.
• Ignoring developer UX: Without easy CLI commands and templates, engineers bypass controls. Provide paved roads.
• One-and-done SBOMs: SBOMs that aren’t updated become lies. Regenerate on every build.
• Policy dead zones: Audit mode never flipped to enforce; treat policy backlog like technical debt.

Tooling Stack

• Signing/attestation: Cosign (keyless), Notary v2; Rekor transparency log.
• SBOM: Syft, Anchore, Trivy; output CycloneDX/SPDX; optionally store in Dependency-Track.
• Policy: Kyverno, OPA Gatekeeper, Sigstore policy-controller (validates attestations).
• CI/CD: GitHub Actions, GitLab, Jenkins pipelines with cosign/kyverno CLI.
• Monitoring: Kubecost for cost of drift, Falco for runtime policy violations, Splunk/ELK for audit logs.

Bundle templates so teams can copy ci-signing.yml rather than re-learning Sigstore every time.

SLSA and Provenance Levels

• SLSA 1: Scripts documenting build steps. Achieved by adding CI metadata and SBOMs.
• SLSA 2: Build service with authenticated provenance (Cosign attestations + builder identity).
• SLSA 3+: Hermetic builds, isolated runners, strong verification. Aim for 2 quickly, roadmap to 3.
• Publish provenance statements in JSON so auditors can verify build steps/inputs.

Metrics That Matter

• Signed image coverage: % of workloads running verified images.
• SBOM coverage: % of deployed images with SBOM attestation.
• Policy violation MTTR: Time from violation detection to remediation.
• Exception backlog: Count and age of waivers—should trend down.
• Time to patch: Average days from CVE disclosure to patched image in prod (per severity).
• Audit readiness: Number of clicks to prove provenance (goal: one dashboard).

Governance and Exceptions

• Create an “Exception” CRD or simple YAML format capturing workload, reason, owner, expiry, mitigation. Policies read from it.
• Require security approval for exceptions; auto-notify when expiry approaches.
• Keep a living register of trusted registries, allowed base images, and builder identities. Update quarterly.
• Tie policy compliance to team OKRs; failures go to the same retro board as incidents.

Developer Enablement

• Provide CLI wrappers: make build-and-sign, make sbom.
• Embed Cosign and Syft containers in build runners to avoid per-team installs.
• Publish sample GitHub Actions / GitLab templates; include docs and guardrail checklists.
• Offer Slack/Teams channel where platform/security answer signing questions quickly.

30/60/90 Plan

• 30 days: Add Cosign signing and SBOM generation in CI; document runbooks; tag registries as immutable; build compliance dashboard.
• 60 days: Deploy admission policies in audit mode (signatures, digests, registries). Train devs; integrate policy checks in CI; begin logging violations.
• 90 days: Enforce policies in prod; add SBOM/provenance checks; automate exception workflow; run a supply-chain incident drill (simulate malicious image) to test response.

Case Study (Hypothetical)

A SaaS platform sees inconsistent rollbacks due to mutable tags. They adopt Cosign keyless signing, generate SBOMs with Syft, and enforce digests via Kyverno. Within 6 weeks, 98% of pods run signed images. An engineer attempts to deploy a hotfix from their laptop; admission denies with “signature missing,” alerting Slack and preventing drift. Auditors now point to Rekor log entries and dashboards rather than spreadsheets.

Adoption Checklist

• CI pipelines sign images and attach SBOM attestations.
• Registries enforce immutability; only CI service accounts can push.
• Admission policies verify signatures, SBOMs, digests, and allowed registries.
• Alerts fire on unsigned workloads or policy bypass attempts.
• Exceptions have owners, expiry dates, and are reviewed monthly.
• Dashboards show signed coverage, SBOM coverage, and CVE remediation metrics.
• Incident runbooks tested: simulate unsigned deploy, verify detection and response.
• Auditors can click one link to retrieve provenance (Rekor entry + Git commit).

Conclusion

Supply chain security is continuous. With signing, SBOMs, and admission controls, you’ll trust what you ship—and catch issues before they reach production. Treat provenance like code: versioned, tested, enforced, and visible.

Glossary (Tooltips)

• SBOM: The dependency “receipt” you can attach to images and verify at deploy time.
• SLSA: A framework for improving build trustworthiness step by step.
• CVE: What you track when measuring time-to-patch.
• CRD: Useful for exception workflows and policy metadata.
• OKR: A practical way to make supply-chain hygiene visible and owned.