Taming Vulnerability Chaos Across The Multiverse of Kubernetes and Edge
·
3min
·
Michiel V.
The talk focused on the unique challenges of vulnerability management in distributed edge environments versus traditional datacenters, and how modern tooling and standards can help.
Some Context
- Datacenters: centralized, stable connectivity, easier to manage.
- Edge environments: distributed, often with constrained or intermittent connectivity.
- Key challenge: vulnerability management across heterogeneous systems and environments.
From Virus Scanners to CVE Scanners
- Security has evolved from traditional virus scanning to scanning for CVEs (Common Vulnerabilities and Exposures).
- Modern scans occur at three stages:
- Build-time (image creation)
- Registry scanning (before deployment)
- Runtime scanning (live containers)
Demos
Demo 1: Trivy vs Grype
- A Claude-driven script ran both scanners on common Linux base images.
- Findings:
- Trivy struggled on Alpine images.
- Grype consistently found more vulnerabilities.
- Python images were also tested — again, Grype found significantly more CVEs.
Demo 2: SBOMs
- SBOM (Software Bill of Materials) shows exactly which packages and dependencies are in a build.
- Using Syft, they generated SBOMs, then scanned them with Trivy and Grype.
- Findings:
- Some software (especially with licensed dependencies) can obscure or remove evidence of components during the build.
- Grype again outperformed Trivy in uncovering vulnerabilities.
Demo 3: Removing CVEs via Metadata Manipulation
- A script modified
/var/lib/dpkg/status
to remove CVE references. - This tricks scanners into thinking vulnerabilities do not exist.
- Highlights the need for trusted build and provenance verification rather than relying on local metadata alone.
For a view on the concerned code have a peek at Denis' repository.
Securing the Build & Deployment Pipeline
Sigstore
- Provides digital signatures for container images, ensuring integrity and protecting against metadata manipulation.
- Should be integrated into admission control policies.
- Sigstore
- Cosign
- Rekor
SLSA Framework
- Supply-chain Levels for Software Artifacts — defines levels of build assurance.
- Encourages provenance tracking, reproducible builds, and higher trust.
- https://slsa.dev/
Way of Working & Recommended Tools
- Secure from source: Utilize hardened base images such as Wolfi or Chainguard to ensure a minimal and secure foundation.
- Generate SBOMs during the build process: Employ tools like Syft to create Software Bill of Materials, providing transparency into your application's components.
- Build APK packages declaratively: Use Melange to define your application's dependencies and build process in a single YAML file, facilitating reproducible and auditable builds.
- Compose images with Apko: Leverage Apko to create single-layer container images based on the APK package format, integrating the APKs built with Melange into your container images.
- Ensure full provenance and signing: Implement tools like Sigstore to sign your images and maintain a transparent record of your build process.
- Aim for SLSA Level 3: Strive to achieve SLSA Level 3 to ensure the highest level of build and supply chain integrity.
True security begins at the source, not just at the container runtime.