7. Build your application and containerize using Docker and Kubernetes

This is the 7th blog as part of the series Full Stack: Remastering Master Data Management into graph like data. Hope you enjoy the series and find it useful !!

Introduction

If you have followed or seen the above series, then you would see that we have tried to understand and learn new concepts in bits and pieces to build an overall application. Well, now we need to stitch this pieces together to achieve the main goal of our application: Data-lake.

Let's recap to see what we have done till now:

• Quarkus application - To have a basic controller and REST APIs to fetch data from any kind of database.
• Reactive Stream messaging in Quarkus - To create a reactive Kafka stream sink and source application to ingest any kind of records.
• Spark SQL Structured Streaming - To Stream and process the data from one Kafka topic, perform various kind of transformation or aggregation and dump it into another topic.
• Ingesting graph data in Neo4j - To ingest the transformed or aggregated data in the form of graph into Neo4j with the help of above Reactive streaming.
• GRAND stack to access data as GraphQL - To access the ingested data into Neo4j using GraphQL.
• GRAND stack for React UI - To write React UI code to use the GraphQL data to build visualizations or dashboard.

Dockerize your application

In today's world, in order to run an application in any kind of environment, the best way is to run as a docker container. So what if you can just configure a docker container with zero knowledge in docker. Both Quarkus and GRAND stack gives that power.

As you know, Quarkus is known for building container-ready application, it creates Dockerfile for JVM as well as native by default. Adding relevant extension provides power to build or run a docker image.

./mvnw quarkus:add-extension -Dextensions="container-image-docker"

Building Docker image:

./mvnw clean package -Dquarkus.container-image.build=true

Pushing a Docker image:

./mvnw clean package -Dquarkus.container-image.push=true

On the other hand, GRAND stack comes with a default docker-compose.yaml file. you can define all the configurations and run the container using:

docker-compose up --build

Deploying application in Kubernetes

Once we have our docker images built and ready, we can run the application inside Kubernetes cluster easily. Quarkus provides extensions to create automated default configurations defined for Kubernetes instance.

Add the following dependencies to your pom.xml:

<dependency>
   <groupId>io.quarkus</groupId>
   <artifactId>quarkus-kubernetes</artifactId>
</dependency>
<dependency>
   <groupId>io.quarkus</groupId>
   <artifactId>quarkus-container-image-jib</artifactId>
</dependency>

Then execute the following command:

./mvnw clean package

You will notice that once the command is executed, that along with the other files that are created, two files named kubernetes.json and kubernetes.yml in the target/kubernetes/ directory is also created.

The Kubernetes extension will create 3 Kubernetes objects:

• Service Account
• Service
• Deployment

The configuration and naming of these is based on some basic parameters that have to be added in application.properties:

quarkus.kubernetes.part-of=graph-based-data-lake
quarkus.container-image.registry=docker.io
quarkus.container-image.group=data-lake
quarkus.container-image.name=getting-started
quarkus.container-image.tag=1.0
quarkus.kubernetes.image-pull-policy=never
quarkus.kubernetes.service-type=NodePort

Deploy to Minikube

With Minikube, we will create the Container (Docker) Image in the Docker installation that is part of the Minikube VM. So after starting Minikube (minikube start) you need to point your local docker command to the Minikube environment. Now with the above configurations, lets try building and deploying the docker container:

./mvnw clean package -Dquarkus.kubernetes.deploy=true

Check that everything is started with:

kubectl get pod 
kubectl get deploy
kubectl get svc

Get the IP of the Minikube cluster:

minikube ip

Now you can test your configured services and REST APIs.

Source Code

You can find the source code in these two repos:

Cover Photo credit

Cover Photo by chuttersnap on Unsplash.