This is the 7th blog as part of the series Full Stack: Remastering Master Data Management into graph like data. Hope you enjoy the series and find it useful !!
If you have followed or seen the above series, then you would see that we have tried to understand and learn new concepts in bits and pieces to build an overall application. Well, now we need to stitch this pieces together to achieve the main goal of our application: Data-lake.
Let's recap to see what we have done till now:
- Quarkus application - To have a basic controller and REST APIs to fetch data from any kind of database.
- Reactive Stream messaging in Quarkus - To create a reactive Kafka stream sink and source application to ingest any kind of records.
- Spark SQL Structured Streaming - To Stream and process the data from one Kafka topic, perform various kind of transformation or aggregation and dump it into another topic.
- Ingesting graph data in Neo4j - To ingest the transformed or aggregated data in the form of graph into Neo4j with the help of above Reactive streaming.
- GRAND stack to access data as GraphQL - To access the ingested data into Neo4j using GraphQL.
- GRAND stack for React UI - To write React UI code to use the GraphQL data to build visualizations or dashboard.
Dockerize your application
In today's world, in order to run an application in any kind of environment, the best way is to run as a docker container. So what if you can just configure a docker container with zero knowledge in docker. Both Quarkus and GRAND stack gives that power.
As you know, Quarkus is known for building container-ready application, it creates Dockerfile for JVM as well as native by default. Adding relevant extension provides power to build or run a docker image.
./mvnw quarkus:add-extension -Dextensions="container-image-docker"
Building Docker image:
./mvnw clean package -Dquarkus.container-image.build=true
Pushing a Docker image:
./mvnw clean package -Dquarkus.container-image.push=true
On the other hand, GRAND stack comes with a default docker-compose.yaml file. you can define all the configurations and run the container using:
docker-compose up --build
Deploying application in Kubernetes
Once we have our docker images built and ready, we can run the application inside Kubernetes cluster easily. Quarkus provides extensions to create automated default configurations defined for Kubernetes instance.
Add the following dependencies to your pom.xml:
<dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-kubernetes</artifactId> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-container-image-jib</artifactId> </dependency>
Then execute the following command:
./mvnw clean package
You will notice that once the command is executed, that along with the other files that are created, two files named kubernetes.json and kubernetes.yml in the target/kubernetes/ directory is also created.
The Kubernetes extension will create 3 Kubernetes objects:
- Service Account
The configuration and naming of these is based on some basic parameters that have to be added in application.properties:
quarkus.kubernetes.part-of=graph-based-data-lake quarkus.container-image.registry=docker.io quarkus.container-image.group=data-lake quarkus.container-image.name=getting-started quarkus.container-image.tag=1.0 quarkus.kubernetes.image-pull-policy=never quarkus.kubernetes.service-type=NodePort
Deploy to Minikube
With Minikube, we will create the Container (Docker) Image in the Docker installation that is part of the Minikube VM. So after starting Minikube (minikube start) you need to point your local docker command to the Minikube environment. Now with the above configurations, lets try building and deploying the docker container:
./mvnw clean package -Dquarkus.kubernetes.deploy=true
Check that everything is started with:
kubectl get pod kubectl get deploy kubectl get svc
Get the IP of the Minikube cluster:
Now you can test your configured services and REST APIs.
You can find the source code in these two repos:
Cover Photo credit
Cover Photo by chuttersnap on Unsplash.