Container Security

Securing the container lifecycle: from image build to runtime in Kubernetes. Containers reduce attack surface compared to VMs but introduce their own threat model.

Securing the container lifecycle: from image build to runtime in Kubernetes. Containers reduce attack surface compared to VMs but introduce their own threat model.


Container Threat Model

Image vulnerabilities:
  - Outdated base image with known CVEs
  - Unnecessary packages installed (larger attack surface)
  - Secrets baked into image layers (visible in docker history)
  - Running as root inside container

Registry risks:
  - Unsigned images (image substitution attack)
  - Public registry with no access control
  - No scanning before deployment

Runtime threats:
  - Container escape to host kernel
  - Privilege escalation within container
  - Lateral movement between containers
  - Network access to restricted services

Secure Dockerfile

# Multi-stage build — only ship what's needed
FROM python:3.12-slim AS builder
WORKDIR /build

# Install build deps only in builder stage
RUN pip install --no-cache-dir poetry
COPY pyproject.toml poetry.lock ./
RUN poetry export -f requirements.txt --without-hashes > requirements.txt
RUN pip install --no-cache-dir -r requirements.txt --target /install

COPY src/ ./src/

# Lean production image
FROM gcr.io/distroless/python3-debian12:nonroot AS production
# distroless: no shell, no package manager, minimal CVE surface

WORKDIR /app

# Copy only installed packages + app code
COPY --from=builder /install /install
COPY --from=builder /build/src ./src

# Non-root user (distroless images use uid 65532 by default)
USER nonroot:nonroot

ENV PYTHONPATH=/install
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

EXPOSE 8000

# Use exec form — signals reach process, no shell wrapping
CMD ["python", "-m", "gunicorn", "myapp.wsgi:application", "--bind", "0.0.0.0:8000"]

Image Scanning with Trivy

# .github/workflows/image-scan.yaml
- name: Build image
  run: docker build -t myapp:${{ github.sha }} .

- name: Scan for vulnerabilities
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myapp:${{ github.sha }}
    format: sarif
    output: trivy-results.sarif
    severity: HIGH,CRITICAL
    exit-code: '1'          # fail pipeline on HIGH/CRITICAL
    ignore-unfixed: true    # skip vulnerabilities with no fix yet

- name: Upload scan results
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: trivy-results.sarif
# Local scanning
trivy image myapp:latest
trivy image --severity HIGH,CRITICAL python:3.12-slim
trivy fs --scanners secret,vuln .   # scan local filesystem for secrets + CVEs
trivy k8s --report summary cluster  # scan entire cluster

Kubernetes Pod Security

# Pod security — restrict what containers can do
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534        # nobody
        runAsGroup: 65534
        fsGroup: 65534
        seccompProfile:
          type: RuntimeDefault  # restrict syscalls to safe set

      containers:
      - name: myapp
        image: myapp:1.2.3
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true     # container can't write to FS
          capabilities:
            drop: ["ALL"]                  # drop all Linux capabilities
            add: ["NET_BIND_SERVICE"]      # only add back what's needed

        volumeMounts:
        - name: tmp
          mountPath: /tmp                  # writable tmp when readOnly root

      volumes:
      - name: tmp
        emptyDir: {}                       # in-memory volume for /tmp

NetworkPolicy — Zero-Trust Networking

# Default deny all ingress/egress, then allow explicitly
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: myapp-netpol
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
    - Ingress
    - Egress

  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: nginx-ingress    # only accept from ingress controller
    ports:
    - port: 8000

  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres         # allow DB access
    ports:
    - port: 5432
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system     # allow DNS
    ports:
    - port: 53
      protocol: UDP

Runtime Security with Falco

# falco-rules.yaml — detect suspicious runtime behaviour
- rule: Shell Spawned in Container
  desc: Detect any shell spawned inside a container
  condition: >
    spawned_process and container
    and proc.name in (shell_binaries)
    and not proc.pname in (allowed_parent_processes)
  output: >
    Shell spawned in container (user=%user.name command=%proc.cmdline
    container=%container.name image=%container.image.repository)
  priority: WARNING

- rule: Sensitive File Access
  desc: Detect access to sensitive files inside containers
  condition: >
    open_read and container
    and fd.name in (/etc/passwd, /etc/shadow, /etc/sudoers, /root/.ssh/id_rsa)
  output: >
    Sensitive file opened for reading (file=%fd.name command=%proc.cmdline)
  priority: ERROR

- rule: Container Running as Root
  desc: Detect containers started as root
  condition: container.start_ts != 0 and proc.vpid=1 and user.uid=0 and container
  output: "Container started with root user (image=%container.image.repository)"
  priority: WARNING

Image Signing with Cosign

# Sign the image after build
cosign sign --key cosign.key myregistry.io/myapp:1.2.3

# Verify before deployment (in CI or admission webhook)
cosign verify --key cosign.pub myregistry.io/myapp:1.2.3

# Keyless signing with OIDC (no key management)
cosign sign --rekor-url https://rekor.sigstore.dev \
            --oidc-issuer https://token.actions.githubusercontent.com \
            myregistry.io/myapp:1.2.3

Common Failure Cases

Trivy scan blocking on a CVE with no upstream fix, stalling deploys indefinitely Why: ignore-unfixed: false (the default) fails the pipeline on HIGH/CRITICAL CVEs even when no patched version of the affected package exists, blocking all releases. Detect: Pipeline fails at the scan step with a CVE that has been present for weeks; the CVE advisory page shows "No fix available" or "Fix in progress". Fix: Set ignore-unfixed: true in the Trivy action config so unfixable CVEs do not block; track unfixed CVEs in a .trivyignore file with a review date comment, and use OS base image updates as the primary remediation path.

readOnlyRootFilesystem breaking an application that writes to its own directory Why: The app writes temp files, sockets, or compiled artifacts to paths like /tmp, /app/cache, or /usr/local/lib; setting readOnlyRootFilesystem: true causes runtime failures on those writes. Detect: Container exits immediately with Permission denied errors on file writes; kubectl describe pod shows the container terminated with exit code 1. Fix: Mount an emptyDir volume at every path the app writes to (at minimum /tmp); audit write paths during testing by running strace -e trace=openat,write against the container.

Falco rule producing false-positive alerts on legitimate shell use Why: A rule fires on Shell Spawned in Container but init containers, health check scripts, or sidecar management processes legitimately spawn shells. Detect: Falco alert volume is so high that real threats are buried; proc.pname in the alert output shows s6-overlay or docker-entrypoint.sh rather than an unexpected process. Fix: Add the legitimate parent processes to the allowed_parent_processes macro in the Falco rule, or scope the rule with container.image.repository to exclude known management images.

Image signing verification not enforced at admission, only in CI Why: Cosign sign/verify runs in CI but no Kubernetes admission webhook (Sigstore Policy Controller, Kyverno) enforces signature verification at deploy time; a compromised image pushed directly to the registry bypasses the check. Detect: Unsigned images can be deployed to production by bypassing the CI pipeline (e.g., via kubectl set image); there is no admission rejection event in the audit log. Fix: Deploy Sigstore Policy Controller or a Kyverno ClusterPolicy that verifies cosign signatures on all image pull events; block unsigned images at the admission webhook level, not just in CI.

Connections

cloud-hub · cloud/cloud-security · cloud/kubernetes-operators · technical-qa/security-automation · technical-qa/infrastructure-testing · cs-fundamentals/security-fundamentals-se

Open Questions

  • What monitoring and alerting matter most when this is deployed in production?
  • At what scale or workload does this approach hit its practical limits?