shutdown all pods in openshift and kubernetes

Posted on Updated on

We must changed some settings on our NFS-Server. To prevent data corruption we decided to shutdown all pods.

This sounds easy but I wanted that the same amount of pods are running after the shutdown as it was before the shutdown.

I have created a text file where the settings are saved.

for i in $( oc get project -o name|cut -d '/' -f2 ); do
  for j in $( oc get dc -n $i -o name |cut -d '/' -f2 ); do
    echo "${i}_${j}_"$( oc get dc -n $i $j \
       --template='{{ .spec.replicas }}') >> backup_dc_scale_data.txt
  done
done

Now scale all to 0

for i in $( < backup_dc_scale_data.txt ); do
  oc scale -n $( echo $i|cut -d'_' -f1) \
    dc/$( echo $i|cut -d'_' -f2) \
    --replicas=0
done

You can check with the following command how many pods are still running.

To shutdown all pods take some times.

watch 'oc get po --all-namespaces |egrep -c Running'

Maybe there are some pods still running then check this one and shutdown it manually.

The pods in openshift-infra must be shutdown via rc.

oc scale rc -n openshift-infra heapster --replicas=0
oc scale rc -n openshift-infra hawkular-metrics --replicas=0
oc scale rc -n openshift-infra hawkular-cassandra-1 --replicas=0

Now we have changed the setting on our NFS-Server.

After the change you can restore the amount of pod.

for i in $( < backup_dc_scale_data.txt ); do
  oc scale -n $( echo $i|cut -d'_' -f1) \
    dc/$( echo $i|cut -d'_' -f2) \
  --replicas=$( echo $i|cut -d'_' -f3)
done

and the metrics one.

oc scale rc -n openshift-infra heapster --replicas=1
oc scale rc -n openshift-infra hawkular-metrics --replicas=1
oc scale rc -n openshift-infra hawkular-cassandra-1 --replicas=1

You may also need to take care of the logging project, which we have not setuped.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s