Edge AI on Kubernetes: Model Serving, Acceleration, and Observability

TL;DR

Edge AI on Kubernetes is viable at scale when you design for constrained resources, intermittent connectivity, and hardware heterogeneity. Use KServe or Triton for standardized serving, schedule GPUs with node labels/taints or MIG slices, autoscale with KEDA on real demand signals, and instrument the entire inference path using OpenTelemetry metrics, logs, traces, and exemplars that link latency to GPU utilization. Treat model rollouts as progressive delivery (canary, blue/green, or shadow) with safety valves and automated rollback tied to error budgets.

“If you pretend edge clusters are just small clouds, they will remind you—loudly—that networks drop, power blips, and GPUs disappear at the worst moments.”

Reference Architecture

Keep the architecture simple and resilient: containerized inference (KServe/Triton) with light preprocess/postprocess where needed; accelerators can be full GPUs, MIG slices, iGPU, or CPU with ONNX EPs. Run GitOps + policy centrally, but let local clusters keep working when disconnected. Cache models locally, ship telemetry upstream when online, and size KEDA autoscaling to real demand with per-site safety limits.

Key ideas:

• Keep control planes and policies centralized, but treat each edge site as an independent failure domain.
• Assume intermittent connectivity: models and configs must be cached; observability exports must buffer.
• Embrace hardware diversity: some sites have datacenter GPUs, others tiny integrated GPUs, others just CPUs.
• Prefer boring, repeatable patterns over clever snowflakes.

“Design for the store with the worst bandwidth and the oldest GPU; everyone else is the easy case.”

flowchart TD
Client --> Gateway(GW)
GW --> VS(Inference Service)
VS --> Accel[GPU/MIG/iGPU/CPU]
VS --> Cache[(Model Cache)]
VS --> OTel[OTel SDK]
OTel --> Collector
Collector -->|Online| Regional[Regional Observability]
Collector -.->|Offline Buffer| Local[(Edge Storage)]

GPU- and Accelerator-Aware Scheduling

Edge capacity is uneven. Some sites will have a single small GPU; others might share a large GPU across workloads using MIG; others will fall back to CPU. Kubernetes can help, but only if you give it the right signals.

Node labeling and taints

apiVersion: v1
kind: Node
metadata:
  name: edge-node-1
  labels:
    edge.site: store-042
    accelerator.vendor: nvidia
    accelerator.gpu.count: "1"
    nvidia.com/mig.present: "true" # if MIG configured
# Taint nodes so only GPU workloads land here
# kubectl taint nodes edge-node-1 accelerator=gpu:NoSchedule

Requests/limits with device plugins (NVIDIA)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-gpu
spec:
  replicas: 1
  selector:
    matchLabels: { app: triton-gpu }
  template:
    metadata:
      labels: { app: triton-gpu }
    spec:
      nodeSelector:
        accelerator.vendor: nvidia
      tolerations:
        - key: "accelerator"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:24.06-py3
          args: ["tritonserver", "--model-repository=/models", "--strict-model-config=false"]
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
              nvidia.com/gpu: 1
            limits:
              cpu: "2"
              memory: "4Gi"
              nvidia.com/gpu: 1
          volumeMounts:
            - name: models
              mountPath: /models
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: model-repo-pvc

MIG (Multi-Instance GPU) slices

For A100/H100, carve GPUs into MIG instances for strict QoS isolation:

# Example (host): Create one 1g.5gb and one 2g.10gb profile on GPU 0
sudo nvidia-smi mig -cgi 19,14 -C
sudo nvidia-smi mig -i 0 -cgi 19 -C
sudo nvidia-smi mig -i 0 -cgi 14 -C

Then advertise MIG resources via device plugin and request them in pods:

resources:
  limits:
    nvidia.com/mig-1g.5gb: 1
  requests:
    nvidia.com/mig-1g.5gb: 1

Model Serving with KServe or Triton

KServe InferenceService example (Triton backend)

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: edge-vision
spec:
  predictor:
    triton:
      runtimeVersion: 24.06-py3
      resources:
        requests: { cpu: "500m", memory: "1Gi", nvidia.com/gpu: 1 }
        limits:   { cpu: "2",    memory: "4Gi", nvidia.com/gpu: 1 }
      storageUri: s3://models/vision/
      env:
        - name: AWS_REGION
          value: us-east-1

Shadow and canary rollouts

Use shadow to mirror a sliver of traffic and compare metrics offline; start canaries at 1–5% and only advance when SLOs hold for a fixed window; blue/green for the final switch when you’re confident.

With Istio routing for canary:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: edge-vision
spec:
  hosts: ["edge-vision"]
  http:
    - route:
        - destination: { host: edge-vision-v1, subset: stable, weight: 95 }
        - destination: { host: edge-vision-v2, subset: canary, weight: 5 }

Autoscaling with KEDA

Scale on real signals (queue depth, RPS, or GPU utilization approximations) rather than guesses:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: edge-vision
spec:
  scaleTargetRef:
    name: edge-vision-deploy
  minReplicaCount: 0
  maxReplicaCount: 4
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        query: sum(rate(http_requests_total{app="edge-vision"}[1m]))
        threshold: "20" # RPS target per replica

Wrap KEDA with per-site safety limits: maximum replicas per cluster, and backpressure behavior when upstream queues or local power constraints kick in. “Unlimited scaling” is a fantasy at the edge; you have to pick which workloads deserve capacity when contention hits.

Observability: Metrics, Traces, Logs, Exemplars

Instrument your inference path end-to-end:

• Metrics: p50/p95/p99 latency, throughput, success rate, GPU memory/utilization, batch size
• Traces: Span from gateway → preproc → model → postproc; include model version in span attributes
• Logs: Structured logs with model, version, site, request_id, latency_ms
• Exemplars: Link metrics to traces for outliers

OpenTelemetry Collector at the edge

receivers:
  otlp:
    protocols: { http: {}, grpc: {} }
processors:
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_mib: 256
exporters:
  prometheus:
    endpoint: ":9464"
  file/traces:
    path: /var/otel/traces.json # offline buffer
  otlphttp/region:
    endpoint: https://otel.example.com/v1/
service:
  pipelines:
    metrics: { receivers: [otlp], processors: [batch], exporters: [prometheus, otlphttp/region] }
    traces:  { receivers: [otlp], processors: [batch], exporters: [file/traces, otlphttp/region] }

Prometheus SLOs and alerts

# SLI: Successful requests
- record: sli:inference_success:ratio
  expr: sum(rate(http_requests_total{app="edge-vision",code=~"2.."}[5m])) \
        / sum(rate(http_requests_total{app="edge-vision"}[5m]))

# Alert on burn rate (multi-window)
- alert: InferenceErrorBudgetBurn
  expr: (
    (1 - sli:inference_success:ratio[5m]) > (1 - 0.995) * 14
  ) and (
    (1 - sli:inference_success:ratio[1h]) > (1 - 0.995) * 6
  )
  for: 2m
  labels: { severity: critical }
  annotations:
    summary: "Error budget burn for edge-vision"

Offline-First Considerations

• Model cache pre-warm; validate hash/signature (Cosign) before activation
• Write-ahead logging for telemetry; backpressure when storage fills
• Graceful degradation to CPU paths when GPU unavailable
• Rate limits and circuit breakers to protect kiosks/PoS

Security and Supply Chain

• Sign model artifacts and container images; verify at admission (Kyverno/OPA)
• Pin digest, run as non-root, drop capabilities
• Segment networks; restrict egress; mutual TLS for gateways

Rollout Checklist

• Baselines: p50/p95, GPU util, memory footprint per batch size
• Canary gates: success rate and latency SLOs green for 30–60 minutes
• Rollback: Immediate on burn-rate alerts or regression in drift/quality
• Post-deploy: Compare shadow/canary vs stable; store evaluation reports

“At the edge, rollouts are logistics problems as much as software problems. You need a plan for sites that fail halfway and stay offline for a day.”

Example: Retail Edge Deployment

Picture a retailer with 500 stores. Each store has a small Kubernetes cluster with one GPU node and a few CPU-only nodes. The AI workload: local vision models that detect shelf stock levels and send restock tasks.

• Models live in an object store and sync via GitOps; each edge cluster caches them on a local PVC.
• Inference runs via KServe with Triton; GPUs are advertised using the NVIDIA device plugin; stores with newer hardware also carve GPUs into MIG slices to isolate workloads.
• KEDA scales replicas per store based on camera frame rate and queue depth, but maxReplicaCount is capped to avoid starving other workloads or blowing power budgets.
• OpenTelemetry sends metrics/traces to a local Collector, which exports to Prometheus on-site and batches traces to a regional cluster whenever the WAN is healthy.
• SLO alerts fire when error rates or latency exceed thresholds; rollouts are canaried to a small cohort of stores before global rollout.

When connectivity drops, stores keep running on cached models and local dashboards; when it returns, they push buffered telemetry, and central teams can see which sites experienced issues.

Conclusion

Edge AI on Kubernetes is production-proven when you respect constraints and bake in guardrails. Standardize on KServe/Triton, schedule accelerators correctly, scale on real demand, and connect robust observability—from edge to region. Tie rollouts to SLOs, not gut feel, and you’ll ship faster without surprises.

“The edge is just prod with fewer excuses: bad networks, tight power budgets, and real people staring at your screens. Build like you’ll have to debug it in person, on a bad Wi‑Fi hotspot, with someone waiting behind you.”