Troubleshooting Storage Errors

Storage issues in Kubernetes typically manifest as PersistentVolumeClaims stuck in Pending, mount failures, or permission denied errors inside containers.

PVC Pending

A PVC stays Pending when no PersistentVolume matches its request or the storage provisioner cannot create one:

# Check PVC status
kubectl get pvc -n production

# Describe the PVC for events and error messages
kubectl describe pvc my-data -n production

# Check available PersistentVolumes
kubectl get pv

# Verify the StorageClass exists
kubectl get storageclass

# Check if the storage provisioner is running
kubectl get pods -A | grep provisioner

Common causes: no matching StorageClass, insufficient capacity in the storage backend, or zone mismatch between the PV and the requesting pod.

Mount Failures

Volumes that fail to mount prevent pods from starting:

# Check pod events for mount errors
kubectl describe pod myapp -n production | grep -A5 "Events"

# Look for "FailedMount" or "FailedAttachVolume" events
kubectl get events -n production --field-selector reason=FailedMount

# Verify the volume is not attached to another node
kubectl get volumeattachments

# Check if the node has the required drivers
kubectl get csidrivers

Permission Denied

Containers may lack permissions to read or write mounted volumes:

# Check the security context of the pod
kubectl get pod myapp -o jsonpath='{.spec.securityContext}'

# Inspect the container's filesystem user
kubectl exec myapp -- id

# Check volume mount permissions
kubectl exec myapp -- ls -la /data

# Fix by setting fsGroup in the pod security context
kubectl patch deployment myapp -p \
  '{"spec":{"template":{"spec":{"securityContext":{"fsGroup":1000}}}}}'

Storage Diagnostics

# Check PV reclaim policy
kubectl get pv -o custom-columns=NAME:.metadata.name,RECLAIM:.spec.persistentVolumeReclaimPolicy

# View CSI driver logs
kubectl logs -n kube-system -l app=csi-driver --tail=100

# Verify volume capacity
kubectl get pv -o custom-columns=NAME:.metadata.name,CAPACITY:.spec.capacity.storage,STATUS:.status.phase

For persistent storage issues, always verify the entire chain: StorageClass, provisioner pods, PV/PVC binding, and volume attachment.