Getting Started

Before deploying Spegel read the compatibility section to make sure that the Kubernetes flavor of your choosing is supported or requires specific configuration to work.

Deploying

Use the Helm chart to deploy Optike and Spegel into your Kubernetes cluster. Refer to the Helm documentation for detailed configuration options.

CLI

To deploy Optik and Spegel with the Helm CLI run the command.

helm upgrade --create-namespace --namespace kvick --install spegel oci://ghcr.io/kvick-org/helm-charts/optik
helm upgrade --create-namespace --namespace kvick --install spegel oci://ghcr.io/kvick-org/helm-charts/spegel

Flux

To deploy Optik and Spegel with Flux commit the configuration.

apiVersion: v1
kind: Namespace
metadata:
  name: kvick
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: kvick
  namespace: kvick
spec:
  type: "oci"
  interval: 5m0s
  url: oci://ghcr.io/kvick-org/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: optik
  namespace: kvick
spec:
  interval: 1m
  chart:
    spec:
      chart: optik
      interval: 5m
      sourceRef:
        kind: HelmRepository
        name: kvick
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: spegel
  namespace: kvick
spec:
  interval: 1m
  chart:
    spec:
      chart: spegel
      interval: 5m
      sourceRef:
        kind: HelmRepository
        name: kvick
  dependsOn:
    - name: optik

Compatibility

Currently, Spegel only works with Containerd, in the future other container runtime interfaces may be supported. Spegel relies on Containerd registry mirroring to route requests to the correct destination. This requires Containerd to be properly configured, if it is not Spegel will exit. First of all the registry config path needs to be set, this is not done by default in Containerd. Second of all discarding unpacked layers cannot be enabled. Some Kubernetes flavors come with this setting out of the box, while others do not. Spegel is not able to write this configuration for you as it requires a restart of Containerd to take effect.

version = 3

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"
[plugins."io.containerd.cri.v1.images"]
  discard_unpacked_layers = false

Spegel has been tested on the following Kubernetes distributions for compatibility. Green status means Spegel will work out of the box, yellow will require additional configuration, and red means that Spegel will not work.

EKS

AL2023

Discard unpacked layers is enabled by default, meaning that layers that are not required for the container runtime will be removed after consumed. This needs to be disabled as otherwise all of the required layers of an image would not be present on the node. AL2023 based EKS AMIs allows customization of Containerd using nodeadm configuration.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  containerd:
    config: |
      [plugins."io.containerd.cri.v1.images".registry]
        config_path = "/etc/containerd/certs.d"
      [plugins.'io.containerd.cri.v1.images']
        discard_unpacked_layers = false

--BOUNDARY--

Bottlerocket

Bottlerocket does not allow to modify the Containerd configuration after the AMI has been deployed. Bootstrap containers are used to modify the host configuration.

Below you can find an example of a bootstrap container configuration that detects the node IP and adds a mirror to the Containerd configuration which points all registries to Spegel.

#!/bin/sh

IP="$(awk '/32 host/ { print f } { f=$2 }' /proc/net/fib_trie | grep -v '127.0.0.1' | head -n1)"
[ -n "$IP" ] || exit 1

apiclient set --json "{
  \"container-registry\": {
    \"mirrors\": {
      \"*\": [
        \"http://$IP:30021\"
      ]
    }
  }
}"

Disable Spegel writing mirror configuration as it is already done in the bootstrap container.

spegel:
  containerdMirrorAdd: false

K0S

As K0S packages its own Containerd some paths will be different than the standard defaults. First of all the Containerd configuration needs to be appended to. The following configuration must be written to /etc/k0s/containerd.d/spegel.toml before K0S starts for the configuration to be picked up.

version = 3

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"
[plugins."io.containerd.cri.v1.images"]
  discard_unpacked_layers = false

After K0S has started Spegel can be installed with the slight modified values as the Containerd socket and content path will be different.

spegel:
  containerdSock: "/run/k0s/containerd.sock"
  containerdContentPath: "/var/lib/k0s/containerd/io.containerd.content.v1.content"

K3S and RKE2

K3S and RKE2 embeds Spegel, refer to their documentation for deployment information.

Kind

Spegel uses Kind for its end-to-end tests.

Discard unpacked layers is enabled by default, meaning that layers that are not required for the container runtime will be removed after consumed. This needs to be disabled as otherwise all of the required layers of an image would not be present on the node.

In order to be more like a “real” Kubernetes cluster, you may want to set Containerd’s metadata sharing policy to isolated.

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"
  [plugins."io.containerd.grpc.v1.cri".containerd]
    discard_unpacked_layers = false
  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "isolated"

Talos

Talos will by default discard unpacked layers, which has to be disabled with a machine configuration.

machine:
  files:
    - path: /etc/cri/conf.d/20-customization.part
      op: create
      content: |
        [plugins."io.containerd.cri.v1.images"]
          discard_unpacked_layers = false

Talos also uses a different path as its Containerd registry config path.

spegel:
  containerdRegistryConfigPath: /etc/cri/conf.d/hosts

Talos comes with Pod Security Admission pre-configured. The default Talos security profile is too restrictive to allow Spegel to operate, therefore it’s necessary to adjust the security profile of the Spegel namespace.

kubectl label namespace spegel pod-security.kubernetes.io/enforce=privileged

VKE

VKE CNI Cello doesn’t support setting hostPorts at the same range of net.ipv4.ip_local_port_range. You can view the exact range by running sysctl net.ipv4.ip_local_port_range on your node.

To workaround this issue, you need to set a custom hostPort when deploying spegel. This can easily be done by setting hostPort in the helm chart.

service:
  registry:
    hostPort: 3020

GKE

GKE uses the default mirror configuration but discards unpacked layers. There is no simple way to override the configuration before Containerd starts.