shutdown all pods in openshift and kubernetes

Posted on November 18, 2016 Updated on November 18, 2016

We must changed some settings on our NFS-Server. To prevent data corruption we decided to shutdown all pods.

This sounds easy but I wanted that the same amount of pods are running after the shutdown as it was before the shutdown.

I have created a text file where the settings are saved.

for i in $( oc get project -o name|cut -d '/' -f2 ); do
  for j in $( oc get dc -n $i -o name |cut -d '/' -f2 ); do
    echo "${i}_${j}_"$( oc get dc -n $i $j \
       --template='{{ .spec.replicas }}') >> backup_dc_scale_data.txt
  done
done

Now scale all to 0

for i in $( < backup_dc_scale_data.txt ); do
  oc scale -n $( echo $i|cut -d'_' -f1) \
    dc/$( echo $i|cut -d'_' -f2) \
    --replicas=0
done

You can check with the following command how many pods are still running.

To shutdown all pods take some times.

watch 'oc get po --all-namespaces |egrep -c Running'

Maybe there are some pods still running then check this one and shutdown it manually.

The pods in openshift-infra must be shutdown via rc.

oc scale rc -n openshift-infra heapster --replicas=0
oc scale rc -n openshift-infra hawkular-metrics --replicas=0
oc scale rc -n openshift-infra hawkular-cassandra-1 --replicas=0

Now we have changed the setting on our NFS-Server.

After the change you can restore the amount of pod.

for i in $( < backup_dc_scale_data.txt ); do
  oc scale -n $( echo $i|cut -d'_' -f1) \
    dc/$( echo $i|cut -d'_' -f2) \
  --replicas=$( echo $i|cut -d'_' -f3)
done

and the metrics one.

oc scale rc -n openshift-infra heapster --replicas=1
oc scale rc -n openshift-infra hawkular-metrics --replicas=1
oc scale rc -n openshift-infra hawkular-cassandra-1 --replicas=1

You may also need to take care of the logging project, which we have not setuped.

Make OpenShift console available on port 443 (https)

Posted on March 11, 2016 Updated on March 11, 2016

Introduction

The main reason why this blog post exist is that OpenShift V3 and Kubernetes is very close binded to port 8443. This could be changed in the future.

We at Cloudwerkstatt GmbH use a dedicated haproxy pod to provide the OpenShift v3 Web console on port 443 (https).

This concept could be used also for different services in the PaaS.

There are some ansible variables for openshift_master_api_port and openshift_master_console_port which suggest that you are able to change the port.

This ports are ‘internal’ ports and not designed to be the public ports. So changing this ports could crash your OpenShift setup!

In case that you want to you this variables you will also need to change a lot of OpenShift v3 and Kubernetes.

The describe solution is a more global and flexible solution then the external service solution.
The external service solution is much easier to setup it is described here

You will need the following to run this setup.

• Time!
• Understanding of OpenShift v3, Kubernetes and docker
• SSL-Certificate for master.<your-domain> or *.<your-domain>
• write access to a git repository
• ssh key for deployment [optional]

Here a rudimentary picture which shows the idea and the flow.

Steps

Btw: Does I said you will need Time and Knowledge! 😉

git clone

Due to the fact that you need to change the haproxy conf you must have a git repository from which OpenShift is able to build the haproxy

You can try to use this repo as base .

git clone https://github.com/cloudwerkstatt/openshift-master.git

Adopt ENV

You need to change the OPENSHIFT_MASTER_SERVER variable in the Dockerfile

Adopt master.cfg

You need to change the container-files/etc/haproxy/master.cfg

Add this into the global section.

ca-base /etc/ssl
crt-base /etc/ssl

Add ssl options to bind line

you need to add this to the bind line

ssl no-sslv3 crt /etc/ssl/certificates-all.pem

Test run

You can try the build with a simple docker build command

docker build --rm -t myhaproxy .

Now run

docker run -it --rm --net host -e OPENSHIFT_MASTER_SERVER=<your-master-ip> myhaproxy

When everything works you need to push the data to your git repository

git stuff

git add .
git commit -m 'init'
git push -u origin master

Create the project

oc new-project infra-services

Add ssl keys to service account

oc secrets new manage-certificate  certificates-all.pem=.../zerts_manage-certificate_all.pem
oc secrets add serviceaccount/default secret/manage-certificate

oc new-app

now create the app.

oc new-app <your-repo-url> --name=openshift-master

oc edit dc

You need to add the secret into the container.

Please take a look into the concept of the secrets here.

oc edit dc -n infra-services openshift-master
spec:
....
    spec:
      containers:
        volumeMounts: <-- Add from here 
        - mountPath: /etc/ssl
          name: secret-volume
          readOnly: true <-- until this line
      terminationGracePeriodSeconds: 30
      volumes: <-- Add from here
      - name: secret-volume
        secret:
          secretName: manage-certificate

After saving the changes a rebuild will start.

oc expose

Make the setup public available over the OpenShift default router

oc expose service openshift-master --hostname=manage.<your-domain>

Test #1

After all this steps and build process you should now see a running pod 😉

A call to

curl -sS https://manage.<your-domain>|egrep hostPort

should now show the OpenShift internal masterPublicURL

egrep -i masterPublicURL /etc/origin/master/master-config.yaml

Ansible hosts file

To configure OpenShift with the new URL please add the following lines to the ansible hosts file

openshift_master_public_api_url=https://manage.{{ osm_default_subdomain }}
openshift_master_public_console_url={{ openshift_master_public_api_url }}/console
openshift_master_metrics_public_url={{ openshift_master_public_api_url }}/hawkular/metrics

and rerun the ansible playbook as described here

ANSIBLE_LOG_PATH=/tmp/ansible_log_$(date +%Y_%m_%d-%H_%M) ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Test #2

A call to

curl -sS https://manage.<your-domain>|egrep hostPort

should now show the OpenShift new public masterPublicURL

egrep -i masterPublicURL /etc/origin/master/master-config.yaml

Which should be the master.<your-domain>