Resize disk

🌇 Sunsetting Kubernetes Deployments

This page covers our PostHog Kubernetes deployment, which we are currently in the process of sunsetting. Existing customers will receive support until May 31, 2023 and we will continue to provide security updates for the next year.

For existing customers

We highly recommend migrating to PostHog Cloud (US or EU). Take a look at this guide for more information on the migration process.

Looking to continue self-hosting?

We still maintain our Open-source Docker Compose deployment. Instructions for deploying can be found here.

Requirements

You need to run a Kubernetes cluster with the Volume Expansion feature enabled. This feature is supported on the majority of volume types since Kubernetes version >= 1.11 (see docs).

kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'
true

DEFAULT_STORAGE_CLASS=$(kubectl get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}')
kubectl patch storageclass "$DEFAULT_STORAGE_CLASS" -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched

How-to

1. List your pods

   Terminal

   kubectl get pods -n posthog
   NAME                          READY   STATUS    RESTARTS   AGE
   posthog-posthog-kafka-0       1/1     Running   0          5m15s

2. Connect to the Kafka container to verify the data directory filesystem size (in this example 15GB)

   Terminal

   kubectl -n posthog exec -it posthog-posthog-kafka-0 -- /bin/bash
   posthog-posthog-kafka-0:/$ df -h /bitnami/kafka
   Filesystem                                                                Size  Used Avail Use% Mounted on
   /dev/disk/by-id/scsi-0DO_Volume_pvc-97776a5e-9cdc-4fac-8dad-199f1728b857   15G   40M   14G   1% /bitnami/kafka

3. Resize the underlying PVC (in this example we are resizing it to 20G)

   Terminal

   kubectl -n posthog patch pvc data-posthog-posthog-kafka-0 -p '{ "spec": { "resources": { "requests": { "storage": "20Gi" }}}}'
   persistentvolumeclaim/data-posthog-posthog-kafka-0 patched

   Note: while resizing the PVC you might get an error disk resize is only supported on Unattached disk, current disk state: Attached (see below for more details).

   In this specific case you need to temporary scale down the StatefulSet replica value to zero. This will briefly disrupt the Kafka service availability and all the events after this point will be dropped as event ingestion will stop working

   You can do that by running: kubectl -n posthog patch statefulset posthog-posthog-kafka -p '{ "spec": { "replicas": 0 }}'

   After you successfully resized the PVC, you can restore the initial replica definition with: kubectl -n posthog patch statefulset posthog-posthog-kafka -p '{ "spec": { "replicas": 1 }}'

4. Delete the StatefulSet definition but leave its pods online (this is to avoid an impact on the ingestion pipeline availability): kubectl -n posthog delete sts --cascade=orphan posthog-posthog-kafka

5. In your Helm chart configuration, update the kafka.persistence value in value.yaml to the target size (20G in this example). You might want to update the retention policy too, more info here

6. Run a helm upgrade to recycle all the pods and re-deploy the StatefulSet definition

7. Connect to the Kafka container to verify the new filesystem size

   Terminal

   kubectl -n posthog exec -it posthog-posthog-kafka-0 -- /bin/bash
   posthog-posthog-kafka-0:/$ df -h /bitnami/kafka
   Filesystem                                                                Size  Used Avail Use% Mounted on
   /dev/disk/by-id/scsi-0DO_Volume_pvc-97776a5e-9cdc-4fac-8dad-199f1728b857   20G   40M   19G   1% /bitnami/kafka