After writing a few million scripts over the years,I’ve learnt a thing or two (I hope). One of those things is how painful it is to wait for some condition before continuing with the next step in a script.
I’ve used expect scripts, I’ve added sleep calls.
I’ve tried a few other things too. It’s never fun :(
kubectl comes with a wait option.
You can ask kubectl to wait for any number of conditions eg:
Wait for a namespace named shane to become Active
$ kubectl wait --for jsonpath='{.status.phase}=Active' --timeout=5s namespace/shane
Wait for a deployment to become Available
$ kubectl wait deployment -n cnpg-system cnpg-controller-manager --for condition=Available=True --timeout=210s
Note that the condition can be specified using the jsonpath, or, in some cases, by name.
The timeout is a maximum time to wait for the condition to become true.
Finding the condition can be difficult. Typically it’s a matter of digging through .yaml files looking for selectors, or, getting whatever you want running, then inspecting it. eg
$ kubectl get pods -n shane
NAME READY STATUS RESTARTS AGE
accountserver-c8897d469-js9qs 1/1 Running 0 17m
accountserver-c8897d469-mr7jn 1/1 Running 0 17m
accountserver-c8897d469-xdzpn 1/1 Running 0 17m
apiserver-57474fd944-7zxmb 1/1 Running 0 18m
postgres-1 1/1 Running 0 22m
postgres-2 1/1 Running 0 21m
postgres-3 1/1 Running 0 19m
Followed by getting all the metadata on the object, in json format
$ kubectl get pods postgres-1 -n shane -o json
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {
"cnpg.io/nodeSerial": "1",
"cnpg.io/operatorVersion": "1.21.0",
"cnpg.io/podEnvHash": "fcffbc754",
"cnpg.io/podSpec": "{\"volumes\":[{\"name\":\"pgdata\",\"persistentVolumeClaim\":{\"claimName\":\"postgres-1\"}},{\"name\":\"scratch-data\",\"emptyDir\":{}},{\"name\":\"shm\",\"emptyDir\":{\"medium\":\"Memory\"}},{\"name\":\"app-secret\",\"secret\":{\"secretName\":\"account-secret\"}}],\"initContainers\":[{\"name\":\"bootstrap-controller\",\"image\":\"ghcr.io/cloudnative-pg/cloudnative-pg:1.21.0\",\"command\":[\"/manager\",\"bootstrap\",\"/controller/manager\",\"--log-level=info\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/run\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"app-secret\",\"mountPath\":\"/etc/app-secret\"}],\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}],\"containers\":[{\"name\":\"postgres\",\"image\":\"ghcr.io/cloudnative-pg/postgresql:16.0\",\"command\":[\"/controller/manager\",\"instance\",\"run\",\"--log-level=info\"],\"ports\":[{\"name\":\"postgresql\",\"containerPort\":5432,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9187,\"protocol\":\"TCP\"},{\"name\":\"status\",\"containerPort\":8000,\"protocol\":\"TCP\"}],\"env\":[{\"name\":\"PGDATA\",\"value\":\"/var/lib/postgresql/data/pgdata\"},{\"name\":\"POD_NAME\",\"value\":\"postgres-1\"},{\"name\":\"NAMESPACE\",\"value\":\"shane\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"postgres\"},{\"name\":\"PGPORT\",\"value\":\"5432\"},{\"name\":\"PGHOST\",\"value\":\"/controller/run\"}],\"resources\":{},\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/run\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"app-secret\",\"mountPath\":\"/etc/app-secret\"}],\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8000},\"timeoutSeconds\":5,\"periodSeconds\":10},\"readinessProbe\":{\"httpGet\":{\"path\":\"/readyz\",\"port\":8000},\"timeoutSeconds\":5,\"periodSeconds\":10},\"startupProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8000},\"timeoutSeconds\":5,\"periodSeconds\":10,\"failureThreshold\":360},\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}],\"terminationGracePeriodSeconds\":1800,\"serviceAccountName\":\"postgres\",\"securityContext\":{\"runAsUser\":26,\"runAsGroup\":26,\"runAsNonRoot\":true,\"fsGroup\":26,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"hostname\":\"postgres-1\",\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"weight\":100,\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"cnpg.io/cluster\",\"operator\":\"In\",\"values\":[\"postgres\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"}}]}}}"
},
"creationTimestamp": "2023-12-18T07:40:02Z",
"labels": {
"cnpg.io/cluster": "postgres",
"cnpg.io/instanceName": "postgres-1",
"cnpg.io/instanceRole": "primary",
"cnpg.io/podRole": "instance",
"role": "primary"
},
"name": "postgres-1",
"namespace": "shane",
"ownerReferences": [
{
"apiVersion": "postgresql.cnpg.io/v1",
"blockOwnerDeletion": true,
"controller": true,
"kind": "Cluster",
"name": "postgres",
"uid": "261148e6-26ac-4d58-8c1e-b248f188aae1"
}
],
"resourceVersion": "9936",
"uid": "1a003c2f-ee83-4d25-af0e-cf192495a235"
},
"spec": {
"affinity": {
"podAntiAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"podAffinityTerm": {
"labelSelector": {
"matchExpressions": [
{
"key": "cnpg.io/cluster",
"operator": "In",
"values": [
"postgres"
]
}
]
},
"topologyKey": "kubernetes.io/hostname"
},
"weight": 100
}
]
}
},
"containers": [
{
"command": [
"/controller/manager",
"instance",
"run",
"--log-level=info"
],
"env": [
{
"name": "PGDATA",
"value": "/var/lib/postgresql/data/pgdata"
},
{
"name": "POD_NAME",
"value": "postgres-1"
},
{
"name": "NAMESPACE",
"value": "shane"
},
{
"name": "CLUSTER_NAME",
"value": "postgres"
},
{
"name": "PGPORT",
"value": "5432"
},
{
"name": "PGHOST",
"value": "/controller/run"
}
],
"image": "ghcr.io/cloudnative-pg/postgresql:16.0",
"imagePullPolicy": "IfNotPresent",
"livenessProbe": {
"failureThreshold": 3,
"httpGet": {
"path": "/healthz",
"port": 8000,
"scheme": "HTTP"
},
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 5
},
"name": "postgres",
"ports": [
{
"containerPort": 5432,
"name": "postgresql",
"protocol": "TCP"
},
{
"containerPort": 9187,
"name": "metrics",
"protocol": "TCP"
},
{
"containerPort": 8000,
"name": "status",
"protocol": "TCP"
}
],
"readinessProbe": {
"failureThreshold": 3,
"httpGet": {
"path": "/readyz",
"port": 8000,
"scheme": "HTTP"
},
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 5
},
"resources": {},
"securityContext": {
"allowPrivilegeEscalation": false,
"capabilities": {
"drop": [
"ALL"
]
},
"privileged": false,
"readOnlyRootFilesystem": true,
"runAsNonRoot": true,
"seccompProfile": {
"type": "RuntimeDefault"
}
},
"startupProbe": {
"failureThreshold": 360,
"httpGet": {
"path": "/healthz",
"port": 8000,
"scheme": "HTTP"
},
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 5
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/lib/postgresql/data",
"name": "pgdata"
},
{
"mountPath": "/run",
"name": "scratch-data"
},
{
"mountPath": "/controller",
"name": "scratch-data"
},
{
"mountPath": "/dev/shm",
"name": "shm"
},
{
"mountPath": "/etc/app-secret",
"name": "app-secret"
},
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "kube-api-access-s9n8l",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"enableServiceLinks": true,
"hostname": "postgres-1",
"initContainers": [
{
"command": [
"/manager",
"bootstrap",
"/controller/manager",
"--log-level=info"
],
"image": "ghcr.io/cloudnative-pg/cloudnative-pg:1.21.0",
"imagePullPolicy": "IfNotPresent",
"name": "bootstrap-controller",
"resources": {},
"securityContext": {
"allowPrivilegeEscalation": false,
"capabilities": {
"drop": [
"ALL"
]
},
"privileged": false,
"readOnlyRootFilesystem": true,
"runAsNonRoot": true,
"seccompProfile": {
"type": "RuntimeDefault"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/lib/postgresql/data",
"name": "pgdata"
},
{
"mountPath": "/run",
"name": "scratch-data"
},
{
"mountPath": "/controller",
"name": "scratch-data"
},
{
"mountPath": "/dev/shm",
"name": "shm"
},
{
"mountPath": "/etc/app-secret",
"name": "app-secret"
},
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "kube-api-access-s9n8l",
"readOnly": true
}
]
}
],
"nodeName": "minikube",
"preemptionPolicy": "PreemptLowerPriority",
"priority": 0,
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {
"fsGroup": 26,
"runAsGroup": 26,
"runAsNonRoot": true,
"runAsUser": 26,
"seccompProfile": {
"type": "RuntimeDefault"
}
},
"serviceAccount": "postgres",
"serviceAccountName": "postgres",
"terminationGracePeriodSeconds": 1800,
"tolerations": [
{
"effect": "NoExecute",
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"tolerationSeconds": 300
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"tolerationSeconds": 300
}
],
"volumes": [
{
"name": "pgdata",
"persistentVolumeClaim": {
"claimName": "postgres-1"
}
},
{
"emptyDir": {},
"name": "scratch-data"
},
{
"emptyDir": {
"medium": "Memory"
},
"name": "shm"
},
{
"name": "app-secret",
"secret": {
"defaultMode": 420,
"secretName": "account-secret"
}
},
{
"name": "kube-api-access-s9n8l",
"projected": {
"defaultMode": 420,
"sources": [
{
"serviceAccountToken": {
"expirationSeconds": 3607,
"path": "token"
}
},
{
"configMap": {
"items": [
{
"key": "ca.crt",
"path": "ca.crt"
}
],
"name": "kube-root-ca.crt"
}
},
{
"downwardAPI": {
"items": [
{
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
},
"path": "namespace"
}
]
}
}
]
}
}
]
},
"status": {
"conditions": [
{
"lastProbeTime": null,
"lastTransitionTime": "2023-12-18T07:40:08Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2023-12-18T07:40:22Z",
"status": "True",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2023-12-18T07:40:22Z",
"status": "True",
"type": "ContainersReady"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2023-12-18T07:40:02Z",
"status": "True",
"type": "PodScheduled"
}
],
"containerStatuses": [
{
"containerID": "docker://56603914ff01f3b781174deac8e7d25fae9e5bd764139eea547715d0c1690351",
"image": "ghcr.io/cloudnative-pg/postgresql:16.0",
"imageID": "docker-pullable://ghcr.io/cloudnative-pg/postgresql@sha256:5baf99e0fb30a051d8e108eeecb558af65a7119e068a280641873a2b98e35457",
"lastState": {},
"name": "postgres",
"ready": true,
"restartCount": 0,
"started": true,
"state": {
"running": {
"startedAt": "2023-12-18T07:40:10Z"
}
}
}
],
"hostIP": "192.168.49.2",
"initContainerStatuses": [
{
"containerID": "docker://70792f2a6c3f5f40424d2a1bb9818d201a768d4c8f63935e80be1bbe700838ff",
"image": "ghcr.io/cloudnative-pg/cloudnative-pg:1.21.0",
"imageID": "docker-pullable://ghcr.io/cloudnative-pg/cloudnative-pg@sha256:44f6132d5335528417427ea12634c5e6c96371e25e00dc898c3b6d091a279c21",
"lastState": {},
"name": "bootstrap-controller",
"ready": true,
"restartCount": 0,
"started": false,
"state": {
"terminated": {
"containerID": "docker://70792f2a6c3f5f40424d2a1bb9818d201a768d4c8f63935e80be1bbe700838ff",
"exitCode": 0,
"finishedAt": "2023-12-18T07:40:06Z",
"reason": "Completed",
"startedAt": "2023-12-18T07:40:06Z"
}
}
}
],
"phase": "Running",
"podIP": "10.244.0.18",
"podIPs": [
{
"ip": "10.244.0.18"
}
],
"qosClass": "BestEffort",
"startTime": "2023-12-18T07:40:02Z"
}
}
We can use almost anything in that, let’s say when the pod is assigned an IP. We can see the path is status.podIPs and it’s an array, with the field inside each podIP being ip
$ kubectl wait --for jsonpath='{.status.podIPs[0].ip}' --timeout=5s pod/postgres-1 -n shane
pod/postgres-1 condition met
Simple :)
Summary
Just a quick note for using what’s a very helpful feature of kubectl, wait, which is now firmly in my toolkit.