A Kubernetes Platform Engineering lab built on a 3-node K3s cluster.
This project demonstrates how to bootstrap, operate, and validate a small but realistic Kubernetes platform using GitOps, ingress, observability, storage, and policy-as-code.
This repository is Project 2 in the portfolio.
It focuses on platform engineering:
- Kubernetes cluster operations
- GitOps platform delivery
- Ingress and LoadBalancer setup
- Monitoring and observability
- Persistent storage
- Policy-as-code
- Application onboarding
A separate repository, gitops-cicd-k3s-lab, is Project 1. It focuses on application CI/CD. This platform onboards that application through ArgoCD to prove an end-to-end DevOps workflow.
| Layer | Tool | Purpose |
|---|---|---|
| Kubernetes | K3s | Lightweight 3-node Kubernetes cluster |
| CNI | Cilium | Pod networking and network policy foundation |
| GitOps | ArgoCD | App-of-Apps deployment model |
| Ingress | ingress-nginx | HTTP routing into the cluster |
| LoadBalancer | MetalLB | Bare-metal/private-network LoadBalancer IPs |
| TLS | cert-manager | Certificate automation baseline |
| Monitoring | kube-prometheus-stack | Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics |
| Storage | Longhorn | CSI distributed block storage |
| Policy | Kyverno | Policy-as-code and audit reports |
| Node | Role | Internal IP |
|---|---|---|
| p2-node-1 | control-plane, etcd | 192.168.0.11 |
| p2-node-2 | control-plane, etcd | 192.168.0.12 |
| p2-node-3 | control-plane, etcd | 192.168.0.13 |
The cluster is administered through a kubeconfig file on the admin machine.
The platform uses ArgoCD App-of-Apps.
Root application:
p2-root-apps
Platform applications:
p2-platform-namespacesp2-argocd-crdsp2-ingress-nginxp2-cert-managerp2-metallbp2-kube-prometheus-stackp2-longhornp2-kyvernop2-portfolio-gitops-cicd-demo
Most upstream Helm charts are rendered locally and committed into Git. This keeps the lab reproducible and reduces runtime dependency on external Helm repositories during ArgoCD reconciliation.
MetalLB provides private LoadBalancer IPs.
Current ingress-nginx LoadBalancer IP:
192.168.0.241
The onboarded demo application is exposed through:
- Host:
demo.p2.local - Ingress class:
nginx - LoadBalancer IP:
192.168.0.241
Because the LoadBalancer IP is on the cluster private network, direct curl from the admin machine may fail if the admin machine cannot route to 192.168.0.0/24. Valid tests are performed from cluster nodes or from inside the cluster.
The platform onboards Project 1:
- Repository:
gitops-cicd-k3s-lab - App:
gitops-cicd-demo-app - Image:
hoangdonguit/gitops-cicd-demo-app:sha-c73671f - Project 1 overlay:
k8s/overlays/p2-platform - Project 2 ArgoCD app:
p2-portfolio-gitops-cicd-demo - Namespace:
apps
This proves the platform can run a real application from a separate CI/CD repository through GitOps.
More details:
docs/application-onboarding.md
Kyverno is installed as the policy-as-code layer.
Current baseline policies:
p2-baseline-require-common-labelsp2-baseline-disallow-privileged-appsp2-baseline-require-resource-requests-apps
The baseline starts in Audit mode to avoid breaking platform workloads during bootstrap.
The onboarded demo app passes the current Kyverno baseline with fail = 0.
Longhorn is installed as a non-default StorageClass.
StorageClasses:
local-pathas defaultlonghornlonghorn-static
A PVC smoke test verified that data persisted across pod recreation before cleanup.
This project demonstrates a platform engineering workflow:
- Built a 3-node K3s platform with Cilium, ArgoCD App-of-Apps, ingress-nginx, MetalLB, cert-manager, kube-prometheus-stack, Longhorn, and Kyverno.
- Onboarded a separate GitOps CI/CD Flask application from another repository.
- Exposed the app through ingress-nginx and MetalLB.
- Added custom Prometheus application metrics and ServiceMonitor scraping.
- Added PrometheusRule alerting for target down, no ready pods, high 5xx ratio, and high p95 latency.
- Validated alert firing through a controlled fire-drill and GitOps cleanup.
- Added Grafana dashboard-as-code for app version, scrape health, request rate, p95 latency, 5xx ratio, CPU, memory, and alerts.
- Recorded evidence and runbooks for each major platform milestone.
Detailed CV summary:
docs/cv-project-summary.md
Major evidence checkpoints:
-
evidence/final-platform-review -
evidence/phase1-post-reset -
evidence/phase1-k3s-cilium-bootstrap -
evidence/phase1-argocd-app-of-apps -
evidence/phase1-ingress-nginx-nodeport -
evidence/phase1-cert-manager-selfsigned -
evidence/phase1-argocd-applicationset-crd-fix -
evidence/phase1-metallb-loadbalancer -
evidence/phase1-ingress-nginx-loadbalancer -
evidence/phase1-kube-prometheus-stack -
evidence/phase1-longhorn-pvc-smoke -
evidence/phase1-kyverno-policy-baseline -
evidence/phase1-platform-mvp-final-state -
evidence/phase2-portfolio-app-onboarding -
evidence/phase2-workload-observability-baseline -
evidence/phase2-custom-app-metrics -
evidence/phase2-custom-app-metrics-route-label -
evidence/phase3-app-alerting-baseline -
evidence/phase3-alert-fire-drill -
evidence/phase3-grafana-app-dashboard
Set kubeconfig:
export KUBECONFIG="$HOME/.kube/p2-k8s-platform-lab-p2-node-1.yaml"
Check ArgoCD apps:
kubectl -n argocd get applications.argoproj.io -o wide
Check all non-running pods:
kubectl get pods -A -o wide | grep -Ev 'Running|Completed' || true
Check app runtime:
kubectl -n apps get deploy,svc,ingress,pod,endpoints,endpointslice -o wide
Test app from a cluster node:
curl -si -H 'Host: demo.p2.local' http://192.168.0.241/
Port-forward Grafana:
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80 --address 127.0.0.1
Port-forward Longhorn UI:
kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80 --address 127.0.0.1
Next improvements:
- Improve repository documentation and runbooks
- Add dashboard access hardening
- Add more Kyverno policies and selected Enforce mode
- Add workload monitoring evidence for the onboarded app
- Add backup and restore testing with Longhorn or Velero