Graceful shutdown of Sidekiq processes on Kubernetes

In our last blog, we explained how to handle rolling deployments of Rails applications with no downtime.

In this article we will walk you through how to handle graceful shutdown of processes in Kubernetes.

This post assumes that you have basic understanding of Kubernetes terms like pods and deployments.

Problem

When we deploy Rails applications on kubernetes it stops existing pods and spins up new ones. When old pod is terminated by Replicaset, then active Sidekiq processes are also terminated. We run our batch jobs using sidekiq and it is possible that sidekiq jobs might be running when deployment is being performed. Terminating old pod during deployment can kill the already running jobs.

Solution #1

As per default pod termination policy of kubernetes, kubernetes sends command to delete pod with a default grace period of 30 seconds. At this time kubernetes sends TERM signal. When the grace period expires, any processes still running in the Pod are killed with SIGKILL.

We can adjust the terminationGracePeriodSeconds timeout as per our need and can change it from 30 seconds to 2 minutes.

However there might be cases where we are not sure how much time a process takes to gracefully shutdown. In such cases we should consider using PreStop hook which is our next solution.

Solution #2

Kubernetes provides many Container lifecycle hooks.

PreStop hook is called immediately before a container is terminated. It is a blocking call. It means it is synchronous. It also means that this hook must be completed before the container is terminated.

Note that unlike solution1 this solution is not time bound. Kubernetes will wait as long as it takes for PreStop process to finish. It is never a good idea to have a process which takes more than a minute to shutdown but in real world there are cases where more time is needed. Use PreStop for such cases.

We decided to use preStop hook to stop Sidekiq because we had some really long running processes.

Using PreStop hooks in Sidekiq deployment

This is a simple deployment template which terminates Sidekiq process when pod is terminated during deployment.

1apiVersion: v1
2kind: Deployment
3metadata:
4  name: test-staging-sidekiq
5  labels:
6    app: test-staging
7  namespace: test
8spec:
9  template:
10    metadata:
11      labels:
12        app: test-staging
13    spec:
14      containers:
15        - image: <your-repo>/<your-image-name>:latest
16          name: test-staging
17          imagePullPolicy: Always
18          env:
19            - name: REDIS_HOST
20              value: test-staging-redis
21            - name: APP_ENV
22              value: staging
23            - name: CLIENT
24              value: test
25          volumeMounts:
26            - mountPath: /etc/sidekiq/config
27              name: test-staging-sidekiq
28          ports:
29            - containerPort: 80
30      volumes:
31        - name: test-staging-sidekiq
32          configMap:
33            name: test-staging-sidekiq
34            items:
35              - key: config
36                path: sidekiq.yml
37      imagePullSecrets:
38        - name: registrykey
Copy

Next we will use PreStop lifecycle hook to stop Sidekiq safely before pod termination.

We will add the following block in deployment manifest.

1lifecycle:
2  preStop:
3    exec:
4      command:
5        [
6          "/bin/bash",
7          "-l",
8          "-c",
9          "cd /opt/myapp/current; for f in tmp/pids/sidekiq*.pid; do bundle exec sidekiqctl stop $f; done",
10        ]
Copy

PreStop hook stops all the Sidekiq processes and does graceful shutdown of Sidekiq before terminating the pod.

We can add this configuration in original deployment manifest.

1apiVersion: v1
2kind: Deployment
3metadata:
4  name: test-staging-sidekiq
5  labels:
6    app: test-staging
7  namespace: test
8spec:
9  replicas: 1
10  template:
11    metadata:
12      labels:
13        app: test-staging
14    spec:
15      containers:
16        - image: <your-repo>/<your-image-name>:latest
17          name: test-staging
18          imagePullPolicy: Always
19          lifecycle:
20            preStop:
21              exec:
22                command:
23                  [
24                    "/bin/bash",
25                    "-l",
26                    "-c",
27                    "cd /opt/myapp/current; for f in tmp/pids/sidekiq*.pid; do bundle exec sidekiqctl stop $f; done",
28                  ]
29          env:
30            - name: REDIS_HOST
31              value: test-staging-redis
32            - name: APP_ENV
33              value: staging
34            - name: CLIENT
35              value: test
36          volumeMounts:
37            - mountPath: /etc/sidekiq/config
38              name: test-staging-sidekiq
39          ports:
40            - containerPort: 80
41      volumes:
42        - name: test-staging-sidekiq
43          configMap:
44            name: test-staging-sidekiq
45            items:
46              - key: config
47                path: sidekiq.yml
48
49      imagePullSecrets:
50        - name: registrykey
Copy

Let's launch this deployment and monitor the rolling deployment.

1
2$ kubectl apply -f test-deployment.yml
3deployment "test-staging-sidekiq" configured
4
Copy

We can confirm that existing Sidekiq jobs are completed before termination of old pod during the deployment process. In this way we handle a graceful shutdown of Sidekiq process. We can apply this technique to other processes as well.