When using Kubernetes, sometimes we want to make the launch of a service running on one pod dependent on the initialization of a service running on another pod.
Suppose we have the following requirement. Each implementation of a new version of the application should be automatically tested by a testing service run on an independent pod that performs functional tests of this application. We implement both the application and the testing service on the environment simultaneously with independent scripts, and the test starts immediately after launching the testing service.
We need to ensure that the container with the testing service is launched after startup and the application is initialized.
Unfortunately, Kubernetes does not provide tools that would allow you to easily meet such a requirement.
If we deploy the application and tests using separate Helm greyhounds, we can use the –wait flag. If this flag is set, the helm install/upgrade command will not finish until all Kubernetes objects have been created or have a minimum number of pods. Unfortunately, the command can also end with an error if we exceed the maximum waiting time, which is defined by the flag --timeout (by default 5min).
However, if we are not using Helm, we can use the InitContainers mechanism.
InitContainers are usually used to prepare an initial environment for an application container. However, we can use them to delay the launch of the main container until our environment is in a good condition.
A single Pod can have multiple InitContainers that execute sequentially and must end correctly (Exit code: 0) for the target container with the application to run. Note The restart Policy field allows you to control the initialization process if pod returns the code 1. If you do not set restartPolicy: Never, code 1 will be returned by the initiating container, it will restart it until it is properly started (Exit code: 0).
We can also configure multiple pod initialization containers. They all have to start correctly. They will run one by one. And each one must end correctly for the next one to start.
So let's use this mechanism to configure the initiating container for our root container with the testing service. This container will verify whether the service we are testing is already active by verifying with a script that returns 0 when it detects the correct status of the service that is to be tested.
Let's assume that the application we are testing is a Spring Boot application that provides endpoint/actuator/health from the Spring Boot Actuator module. It will be used to check whether the application has already been initiated. The service we are testing is provided by the Service app-service. The image from the testing service is app-tests.
To verify the status, we will use a script that checks every minute whether the application is properly launched and returns 0. If we do not find out within 5 minutes (5 attempts at 1-minute interval) that the application is running, it will return exit code 1.
Then the definition of Job launching the testing service may be as follows:
apiVersion: batch/v1
kind: Job
metadata:
name: test-job
spec:
template:
spec:
restartPolicy: Never
containers:
- name: app-tests
image: app-tests:latest
initContainers:
- name: wait-for-app
image: alpine-curl:latest
command: [
"sh",
"-c",
"NUMBER_OF_TRIES=0;
sleep 2m;
IS_UP=$(curl -f -s --connect-timeout 5 app-service:8080/actuator/health | grep -c '\"status\":\"UP\"');
while [[ \"$NUMBER_OF_TRIES\" -lt 5 ]] && [[ \"$IS_UP\" != 1 ]];
do
let \"NUMBER_OF_TRIES+=1\";
echo \"App not running, wait 1min, number of tries $NUMBER_OF_TRIES...\";
sleep 1m;
IS_UP=$(curl -f -s --connect-timeout 5 app-service:8080/actuator/health | grep -c '\"status\":\"UP\"');
done;
if [[ \"$IS_UP\" == 1 ]];
then echo App is up and running; exit 0;
else echo App is down; exit 1;
fi"]
Our app-test container has an initConteiner called wait-for-app defined, which will run before the app-test container and wait for the script to detect that the application has started successfully. In this case, it will finish working properly and the app-test container will be run and start testing the application when it is already initialized and available.
If the application is not initiated within 5 minutes (5 attempts with an interval of 1 minute), the script will return a value of 1. This will be a signal to Kubernetes that our initiating container ended "incorrectly". In our case restartPolicy is set to Never, so the proper container on the Pod will not be started and there will be no further attempts to initialize it.
Depending on the requirements and a specific case, using the parameters, you can adjust this mechanism to your own needs.