Log Shipping Using Fluent Bit and vSphere with Tanzu

April 20, 2021 Kendrick Coleman

One of the new features that came with the latest update of vSphere with Tanzu was the ability to use TKG Extensions. This powerful framework allows simplified deployment and management of multiple open source projects that are backed by VMware support, including alerting with Prometheus, visualization with Grafana, ingress with Contour, and logging with Fluent Bit. 

This post will walk through the setup and configuration of Fluent Bit using TKG Extensions on a Tanzu Kubernetes cluster. After the setup is complete, all the logs from the cluster will be forwarded to vRealize Log Insight using syslog, which can be used for log retention and analysis. Fluent Bit has the ability to forward logs to multiple collectors, such as Elasticsearch, HTTP, Kafka, and Splunk, so it’s highly configurable. 

The first step is making sure there is a target to collect logs. Deploying the VMware vRealize Log Insight Virtual Appliance is straightforward, or sign up for the VMware vRealize Log Insight Cloud. Once up, it’s ready to start collecting!

TKG Extensions require three prerequisite components—Extension Manager, Kapp Controller, and Cert Manager—each of which are located in the TKG Extensions bundle at /tkg-extensions/extensions/. It may appear that there are duplicate folders, so to ensure  successful deployment, check that you’re using the extensions folder. Then install the components and verify that each of them is running. 

cd /tkg-extensions/extensions/
kubectl apply -f tmc-extension-manager.yaml
kubectl apply -f kapp-controller.yaml
cd /tkg-extensions/cert-manager/
kubectl apply -f cert-manager/

Deploying Fluent Bit requires three easy steps. First, create the namespace and role objects. This establishes a home for where the Fluent Bit containers will run and ensures that several role-based access control objects are used.

cd /tkg-extensions/extensions/logging/fluent-bit/
kubectl apply -f namespace-role.yaml

The second step is to create a copy of a Fluent Bit configuration file, editing it with the values for the environment and applying the file as a Kubernetes secret. vRealize Log Insight, a syslog collector that listens on tcp/514, can use multiple RFC formats. The example below uses RFC 5424.

cp syslog/fluent-bit-data-values.yaml.example syslog/fluent-bit-data-values.yaml
vi  syslog/fluent-bit-data-values.yaml
kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=syslog/fluent-bit-data-values.yaml -n tanzu-system-logging

After the secret has been applied, the final step is deploying the Fluent Bit extension.

kubectl apply -f fluent-bit-extension.yaml

Shortly after the deployment, logs will start appearing in vRealize Log Insight. 

To examine what happens with an individual application, let’s create a simple NGINX deployment:

kubectl create deployment --image nginx my-nginx

In vRealize Log Insight, go to Interactive Analytics and perform a search for nginx to bring up all the logs needed to create the pod.

TKG Extensions provide an easy framework for getting started with some of the most popular open source applications in the Cloud Native Computing Foundation landscape. In the future, the framework will also include simplified lifecycle management of these existing extensions, and will package even more extensions—all while being supported by VMware. 

Make sure to read the documentation for the latest instructions on how to install and configure TKG Extensions. For more, check out the free containers, Kubernetes, and Spring on-demand labs and Tanzu products training.

Image courtesy of Billy Hyunh via Unsplash.


About the Author

Kendrick Coleman is a reformed sysadmin and virtualization junkie. His attention has shifted from hypervisors to cloud native platforms focused on containers. In his role as an Open Source Technical Product Manager, he figures out new and interesting ways to run open source cloud native infrastructure tools with VMware products. He's involved with the Kubernetes SIG community and frequently blogs about all the things he's learning. He has been a speaker at DockerCon, OpenSource Summit, ContainerCon, CloudNativeCon, and many more. His free time is spent sharing bourbon industry knowledge hosting the Bourbon Pursuit Podcast.

More Content by Kendrick Coleman
Kubernetes Guides and Software Development Practices from VMware Tanzu Labs
Kubernetes Guides and Software Development Practices from VMware Tanzu Labs

As with all the information on the public Tanzu Developer Center, content detailing our renowned practices ...

How Tanzu Act and Save the Children Rethought Nutrition Data
How Tanzu Act and Save the Children Rethought Nutrition Data

Save the Children partnered with VMware's Tanzu Act to improve its approach to nutritional causal analysis ...

SpringOne at VMware Explore 2023

Learn More