HuggingFace

HuggingFace is a popular repository for pre-trained AI models, and is widely supported in the inference Kubernetes ecosystem as a source for models. These models tend to be large and take a long time to download, which is why keeping models cached is critical for responsive scaling. Kubernetes ephemeral nature means that by default data downloaded to a container will not be cached. While it is possible to mount a host volume into the container, any cache that builds up will be limited to that specific host.

SEE implements a HuggingFace compatible proxy that not only caches any downloaded model, but also distributes the cache in its p2p network. Any client that is already pulling from HuggingFace is able to reap the benefits of p2p caching. Scaling an inference job goes from taking minutes to seconds.

The HuggingFace entrypoint is enabled by default, but can be configured through the daemon configuration. It uses a disk store to persist downloaded models and serve them to other peers.

stores:
  - name: "hf"
    disk: {}
entrypoints:
  - name: "hf"
    stores:
      - "hf"
    huggingface: {}

KServe

Configuring KServe to pull its HuggingFace models through Spegel is as simple as create a new cluster storage container with a additional environment variable. If you already have a default cluster storage container you will need to remove hf:// from the supporter URI formats to avoid conflicts.

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterStorageContainer
metadata:
  name: see-hf
spec:
  container:
    name: storage-initializer
    env:
      - name: HF_ENDPOINT
        value: "http://spegel-entrypoint.kvick/entrypoint/hf"
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      runAsNonRoot: true
  supportsMultiModelDownload: true
  supportedUriFormats:
    - regex: "hf://"

The HF_ENDPOINT environment variable tells the KServe init container where to download from, the value http://spegel-entrypoint.kvick/entrypoint/hf is the DNS name for the Spegel entrypoint. The first deployment will fetch directly from HuggingFace, once stored in the cluster all other fetches will be served quickly from within the cluster.