HuggingFace

HuggingFace is a popular repository for pre-trained AI models, and is widely supported in the inference Kubernetes ecosystem as a source for models. These models tend to be large and take a long time to download, which is why keeping models cached is critical for responsive scaling. Kubernetes ephemeral nature means that by default data downloaded to a container will not be cached. While it is possible to mount a host volume into the container, any cache that builds up will be limited to that specific host.

SEE implements a HuggingFace compatible proxy that not only caches any downloaded model, but also distributes the cache in its p2p network. Any client that is already pulling from HuggingFace is able to reap the benefits of p2p caching. Scaling an inference job goes from taking minutes to seconds.

The HuggingFace entrypoint is enabled by default, but can be configured through the daemon configuration. It uses a disk store to persist downloaded models and serve them to other peers.

stores:
  - name: "hf"
    disk: {}
entrypoints:
  - name: "hf"
    stores:
      - "hf"
    huggingface: {}

KServe

Configuring KServe to pull its HuggingFace models through Spegel is as simple as setting a single addition environment variable in the model configuration. The example from the KServe getting started guide can be modified to pull through Spegel.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-llm
  namespace: kserve-test
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=qwen
      env:
        - name: HF_ENDPOINT
          value: "http://spegel-entrypoint.kvick/entrypoint/hf"
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      resources:
        limits:
          cpu: "2"
          memory: 6Gi
          nvidia.com/gpu: "1"
        requests:
          cpu: "1"
          memory: 4Gi
          nvidia.com/gpu: "1"

The HF_ENDPOINT environment variable tells the KServe init container where to download from, the value http://spegel-entrypoint.kvick/entrypoint/hf is the DNS name for the Spegel entrypoint. The first deployment will fetch directly from HuggingFace, once stored in the cluster all other fetches will be served quickly from within the cluster.