Tanzu Developer Center Blog
Thomas Kraus January 8, 2021
In a previous post, we discussed the advantages of running JupyterHub on Kubernetes. We also showed you how to install a local Kubernetes cluster using kind on your Mac, as well as how to install the JupyterHub Helm chart on a Kubernetes cluster.
In this post, we will focus on the experience of the developers, who are going to be leveraging our service to develop new models using scikit-learn or perform calculations and transformations of large datasets using pandas. To illustrate the value that Jupyter Notebooks and JupyterHub provide in a multiuser environment, we will clone a Git repository containing two example Jupyter Notebooks that we can work with.
Using JupyterHub
Each user that accesses JupyterHub will have their own workspace complete with a single-user Jupyter Notebook server, which uses the JupyterLab Interface. To demonstrate the capabilities of JupyterHub and Python, we will check out the following sample notebooks that we have written and executed:
- industry-revenue-analysis.ipynb – Analysis of historical financial data organized by industry that leverages the pandas library
- ml-stock-predictor-knn-v4.ipynb – Machine learning (ML) based on revenue data from public financial statements that leverages the
scikit-learnlibrary for Python
Note: Each time a user logs into the JupyterHub web page, an additional pod will be instantiated for that user and a 10GB …
Thomas Kraus January 7, 2021
Provisioning environments for data scientists and analysts to run simulations, test new models, or experiment with new datasets can be time-consuming and error-prone. Python is a popular choice for data science use cases, and one of the easiest ways to leverage Python is through Jupyter Notebooks. A web-based development environment for multiple languages, Jupyter Notebooks support the creation and sharing of documents that contain code, equations, visualizations, output, and markup text all in the same document. Because Jupyter Notebooks are just text files, they can be easily stored and managed in a source code repository such as GitLab or GitHub. JupyterHub, meanwhile, is a multiuser hub that spawns, manages, isolates, and proxies multiple instances of a single-user Jupyter Notebook server.
Kubernetes provides the perfect abstractions and API to automate consistent and isolated environments for data scientists to conduct their work. Combining these three things—Jupyter Notebooks, Python, and Kubernetes—into one powerful platform therefore makes a lot of sense.
In the first post in this two-part series, you will learn how to deploy a Kubernetes cluster using kind on a Mac, then how to install JupyterHub into that cluster. In the second post, we will show you how to use the data science and machine learning notebooks you have created on your newly deployed JupyterHub service …
Michael Coté November 23, 2020
Who will speak for the various, meaningless phrases and jargon that fills our ears? “Digital transformation,” for example. Year after year, surveys of Very Important People in the form of Gartner’s CIO Agenda Report and others show rising interest, even “do or die” desire for digital transformation. These efforts seem to always be behind: They’re either underfunded or in the process of getting more funding; skilled people are consistently hard to find. And the headwinds! Always with the macro-global headwinds.
But surely a company must transform in order to remain competitive. Indeed, if all these executives are craving “digital transformation” and complaining about how hard it is to achieve, it must be something very important, right?
Well, sort of.
Computers are awesome!
The problem with “digital transformation” is that it’s become an umbrella term to mean any spending on or change to IT. We need to implement remote working? Then we need digital transformation. Our goal is now better analytics? That means digital transformation! Upping our sales through Instagram? Roll in the digital transformation!
When a term is used for everything, it loses its meaning. In such cases, I like to replace “digital transformation” or whatever the phrase of the moment is with “Computers are awesome!” Doing so helps me remember that all people are talking about is using computers to conduct …
Brian McClain November 16, 2020
Three years ago, a colleague of mine wrote a post to help readers understand when to use RabbitMQ and when to use Apache Kafka, which many found to be very useful. While the two solutions take very different approaches architecturally and can solve very different problems, many find themselves comparing them for overlapping solutions. In an increasingly distributed environment where more and more services need to communicate with each other, RabbitMQ and Kafka have both come to be popular services that facilitate that communication.
It has been three years since that post was published, however, which in technology can be lifetime. We thought this would be a great opportunity to revisit how RabbitMQ and Kafka have changed, check if their strengths have shifted, and see how they fit into today’s use case.
What Are RabbitMQ and Apache Kafka?
RabbitMQ is often summarized as an “open source distributed message broker.” Written in Erlang, it facilitates the efficient delivery of messages in complex routing scenarios. Initially built around the popular AMQP protocol, it’s also highly compatible with existing technologies, while its capabilities can be expanded through plug-ins enabled on the server. RabbitMQ brokers can be distributed and configured to be reliable in case of network or server failure.
Apache Kafka, on the other hand, is described as a “distributed event streaming …
Tiffany Jernigan November 11, 2020
KubeCon North America is coming up soon! It will take place virtually November 17th-20th.
The schedule is chock-full of very interesting talks,from introductory overviews to advanced deep dives. When I first saw it, I ended up copying down as many talks as possible to share here because they are all just so good. But I figured I should probably curate a bit, so below you will find a list of my top recommendations, broken down, for the most part, by their respective Special Interest Group (SIG) names.
Talks with a 🌱 next to them are introductory/deep dive talks, and each SIG section header links to its respective SIG page.
Keynotes
If you can, definitely watch all of the keynotes. There is a complimentary pass just for the keynotes if you’re unable to attend the rest of the conference.
Sponsored
If you’re interested in seeing what cool things different companies are working on and/or are interested in, check out the sponsored sessions on Day 1.
API Machinery
Day 2
🌱 Admission Control, We Have a Problem - Ryan Jarvinen, Red Hat
This is an interactive session that will teach you how Admission Controllers play a critical role in securing Kubernetes APIs. You will be able to “implement basic input validation and testing of webhooks for the Admission Controller.”
Day 4
Scott Rogers November 4, 2020
It’s the day and age of mountains of microservices, running on various platforms, consuming multiple services from multiple providers. As applications become more and more distributed, they become more complex. Even splitting a monolith into multiple smaller microservices introduces several points of failure. What happens when the two services can’t reach each other over the network? What if one service relies on the other and it crashes? What about if the application slows to a crawl; where would you start looking to figure out why?
Rather than guessing and hoping, you can lean on properly instrumented observability. Being able to aggregate logs and metrics, as well as trace a request as it flows through various applications and services, is as achievable as ever. No matter your language, framework, or platform of choice, chances are you have some great options.
But first, let’s talk about why you should care about observability.
What Is Observability?
I think of observability as the ability to infer the correlation between (seemingly) disparate systems. That means bringing together metrics from many systems in a way that allows us to find answers to questions that speed up both MTTD (the mean time to detect an issue) and MTTR (the mean time to resolve an issue). By themselves, metrics such as CPU, memory, response time, error rates, and latency are valuable, but they will …
Sam Foo October 15, 2020
Octant is a tool designed to enable developers without a deep knowledge of Kubernetes to become productive as quickly as possible. This post will walk through an NGINX deployment with a faulty NGINX configuration to demonstrate how common pitfalls can be identified using Octant.
Before You Begin
Before you get started, here are the tools you’ll need:
- An existing cluster through KinD, Minikube, or a cloud provider
- An installation of Octant 0.16.1 or above
- Some knowledge of NGINX (helpful but not required)
Create an NGINX Deployment
Let’s start with a ConfigMap containing a basic NGINX configuration and set the mount path to /etc/nginx. A deployment will mount the volume containing the NGINX configuration for the container. Copy the YAML below.
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-conf
data:
nginx.conf: |
user nginx;
worker_processes 3;
error_log /var/log/nginx/error.log;
events {
worker_connections 10240;
}
http {
log_format main
'remote_addr:$remote_addr\t'
'time_local:$time_local\t'
'method:$request_method\t'
'uri:$request_uri\t'
'host:$host\t'
'status:$status\t'
'bytes_sent:$body_bytes_sent\t'
'referer:$http_referer\t'
'useragent:$http_user_agent\t' …VMware Pivotal Labs October 7, 2020
The transition to working remotely due to COVID-19 has proven to be quite a challenge—including for those of us at VMware Pivotal Labs. While remotely communicating and collaborating using digital tools is not new to us, doing so in an entirely distributed environment for the foreseeable future is.
The challenges have been especially daunting when we’ve had to apply the principles of Extreme Programming (XP), namely constant communication, intense collaboration, and ongoing reflection. On top of all that, we engage with new customers on a regular basis to kick off projects multiple times throughout the course of a year. In this new normal, our practitioners must not only quickly bootstrap burgeoning relationships and get up to speed on unfamiliar domains, but build customer-based teams that feature all these new ways of working and in the process, re-define a set of previously shared norms.
There are undoubtedly more challenges left for us to face in our new, all-remote set-up. In the meantime, however, we set out to identify the most pressing problems and come up with the best possible solutions to address them. The result is a series of remote working tips that we are thrilled to share with you.
Our tips
Our remote working tips cover a wide range of topics, from general considerations to specific advice for high-collaboration teams:
Building relationships – How to create …
Jorge Morales Pou September 22, 2020
One of the biggest challenges I face when developing applications that will run on Kubernetes is having a local environment that I can spin up at any time—one that won’t give me any problems, won’t cost me money when left on during the weekend or at night, and that I can be confident will have all the same functionality as my cloud-based environment. That’s why I use minikube for local development, as it’s the tool that gives me the best “developer experience” possible. None of the alternatives can really compare.
But minikube is not perfect. There are two things in particular that require some additional configurations.The first is that every time you create a minikube instance you get a different IP address, which becomes an obstacle when you want to recreate your environments. The second is that I prefer my minikube instances to have a registry, which like the services I choose to work with should also be secure. But while the minikube documentation provides instructions for how to secure a registry, it’s still a complicated process.
For both of these reasons, I’m going to explain how I set up my minikube instances so they can be used and recreated easily, and so they give me the ability to work with trusted secured services.
Create your own local CA
When working with secure services, I want the minikube instance to have secure routes and internal access. The easiest …
Ivan Tarin September 17, 2020
SpringOne 2020 just wrapped, and the self-paced workshops were a complete success! Moreover, all of your requests to continue providing these workshops beyond the conference have been heard. Their future home will be the Tanzu Developer Center. UPDATE: These workshops are available to try out now!
For those that missed SpringOne 2020, we’ll quickly recap what the workshops focused on and what they accomplished. Feel free to jump around if you need to; the recap is meant to be a quick read with plenty of pictures.
A total of 10 self-paced workshops covering a range of open source technology—from CI/CD with Tekton, several Spring technologies, and infrastructure technology like Kubernetes, Octant, and Carvel—were available at SpringOne 2020:
- Kubernetes Fundamentals
- Container Basics
- Getting Started with Octant
- Getting Started with Spring Boot on Kubernetes
- Spring Boot Probes on Kubernetes
- Spring Boot Skaffold on Kubernetes
- Spring Microservices
- Getting Started with Spring Cloud Gateway
- Getting Started with Carvel (formerly k14s)
- Tekton Fundamentals
Each of these workshops has an environment prepared and ready, which is quite refreshing when you are accustomed to spending 15-30 minutes setting up to follow a tutorial. The workshop environment is also native to the technology being used. For example, you can interact with actual Kubernetes clusters in a Kubernetes workshop or work …
Eric Tsiliacos August 19, 2020
You’re halfway through delivering your feature and you decide to take a look at your diff. Doing so gives you a sinking feeling in your stomach, because you see a lot more changes than you were expecting, some of which were refactorings, like renames or structural changes you wish were separated into their own commits. Teasing apart these smaller commits can be messy and would take too much effort at this point. But if you’d tried to preemptively break them apart at random, you’d have run the risk of overengineering your work or creating unnecessary changes.
Writing commit messages first can help pairs navigate between feature delivery and what could be smaller, more atomic commits. It’s analogous to working from “stories” off a backlog or trying to get the next failing test to pass. If you or your pair gets lost in the weeds, pointing back to the story provides a direct and egoless way to get back on track. In test-driven development, pairs stay focused by thinking about small, incremental, falsifiable functionality one failing test at a time. Writing commit messages first can help pairs articulate the space between a feature story and individual tests.
How to Write Commit Messages First
You can use a command line tool such as Stacker to keep track of your commit messages, or simply use pencil and paper. The first commit message is easy to write and provides a frictionless …
Brian McClain August 11, 2020
For quite a while now, I’ve kept an eye on RedMonk’s programming language rankings — which track the usage of all programming languages based on data from GitHub, as well as discussion about them on Stack Overflow —— to get insight into the various language trends. In the January 2020 update, something interesting happened: Python reached No. 2 on the list, taking over Java.
As RedMonk pointed out, “[T]he numerical ranking is substantially less relevant than the language’s tier or grouping. In many cases, one spot on the list is not distinguishable from the next.” However, it’s still interesting, especially as Python has continued to hold the No. 2 spot into the latest ranking after spending approximately seven years ranked third or fourth.
RedMonk isn’t alone in its findings, either. GitHub’s report, The State of the Octoverse also ranked Python as the second most-popular language used on that website, just behind JavaScript. Not only that, it also found Python remains among the top 10 fastest-growing languages in the community, despite already having a foothold with developers. In the JetBrains Python Developers Survey in 2019, it found that one of the most popular things developers use Python for is web development, with Flask and Django fighting for the top web framework.
At one time Python was my main language of choice. For the past few years, I’ve primarily been a Java …
Dan Dobrin July 30, 2020
Your cloud-native application has been designed, built, and tested locally, and now you’re taking the next step: deployment to Kubernetes.
Isolated environments can be provisioned as namespaces in a self-service manner with minimal human intervention through the Kubernetes scheduler. However, as the number of microservices increases, continually updating and replacing applications with newer versions, along with observing and monitoring the success/failure of deployments, becomes an increasing burden.
Deployment processes are performed while allowing for zero or some downtime, at the expense of increased resource consumption or support of concurrent app versions.
What options do you have to turn deployments into a predictable, repeatable, consistent, easy to observe process? Kubernetes with declarative deployments.
What Are Declarative Deployments in Kubernetes?
The concept of deployment in Kubernetes encapsulates the upgrade and rollback processes of a group of containers and makes its execution a repeatable and automated activity.
The declarative nature of a Kubernetes deployment allows you to specify how the deployed state should look, rather than the steps to get there. Declarative deployments allow us to describe how an application should be updated, leveraging different strategies while allowing for fine-tuning of the various aspects of the deployment process.
The …
Santiago Vasquez July 22, 2020
Throughout most of my career as a developer, I have written code using .NET (mostly C#). But lately, I have been spending more time with Spring, and I keep hearing comments about exciting changes in .NET around containers. I decided it was time to go back and check out what I had missed. This article highlights some of these changes, emphasizing the ones most relevant to containers and microservices; after all, I am part of the VMware Tanzu Portfolio.
Microsoft released .NET Core 3.0 on Sept. 23, 2019, and a couple of months later, on Dec. 3, 2019, version 3.1 followed. Version 3.0 had already reached its end of life, while version 3.1, with its LTS designation, will have support until Dec. 3, 2022 (more details here).
.NET Core 3.1 contains a tiny number of changes compared to version 3.0. These are mainly related to Blazor and Windows Desktop, in addition to the LTS designation. The bulk of significant changes were in version 3.0. I have selected a subset of items that I believe have a more significant impact on my day-to-day role at VMware Pivotal Labs. For the complete list of changes, go here and here.
From Container-Friendly to Container-Aware
Before version 3, running .NET Core in a container was not for the faint of heart. CoreCLR was inefficient when allocating GC heaps and quickly ran into Out-of-Memory situations. The new version of .NET Core has made significant …
Brian McClain July 17, 2020
When you’re first learning how Kubernetes works, or are developing code that leverages Kubernetes, you’re likely to find yourself looking to one of the many options available to run it locally. As with almost anything in technology, there are more options than you probably know what to do with, which can leave you asking yourself which one you should use. Minikube? Kind? Microk8s? Even Docker Desktop ships with the ability to spin up Kubernetes.
Consider a scenario in which you need to develop and test on Kubernetes locally. For example, Spring Cloud Kubernetes gives you tools such as service discovery, which enables you to look up Kubernetes services, as well as the ability to set properties in your code using ConfigurationMaps. This post will use a simple two-tier application that has a frontend (written in Spring) that looks up where the backend service is (also written in Spring) by looking up the service name that exposes it. The backend service presents a REST API that reports inventory information about an imaginary grocery store, and the frontend application visualizes it.
Since the whole point is to develop locally, you’ll deploy the backend, then get a shell into another pod that mounts a volume from your local machine containing the frontend code. This will allow you to rapidly iterate changes to the code without building and deploying a new container every time. …
Chris Splinter July 8, 2020
You can start to build modern, cloud native apps today using the latest innovations from Spring, Cassandra and Kubernetes. This blog will show you code samples and a fully working demo to get you up to speed in minutes with these open-source technologies.
Context
There’s no shortage of buzzwords, such as “digital transformation”, “cloud native” and “serverless,” being thrown around the internet. Peeling back a layer of the buzzword onion, we do see significant changes in the technology world that have inspired these terms. First, most companies are becoming technology companies as having a presence in the digital space grows as a requirement for survival. Second, the mad dash to the cloud is showing no signs of slowing down. Third, time to market for new applications matters more than ever.
So, how has this affected technology practitioners? Well, the developer population is multiplying rapidly and the pressures for faster delivery of these new digital experiences are getting more extreme by the day. It’s now a fundamental expectation that applications will be there whenever, wherever and however users want to engage. The cloud movement is creating complex architectures that span on-premises and cloud environments, not to mention the pure amount of data coming through these services is exploding. Sounds fun!
Open-source technology …
Ivan Tarin June 29, 2020
Understanding the way containers communicate will make your life easier in many ways. Technologies like Kubernetes and Docker abstract away details that make containers special, but can also abstract away intuition and understanding. Without that knowledge, challenges arise—educated problem-solving adds confidence to decision-making!
In this post, we will demystify containers and cover some networking basics by explaining how to create two rudimentary containers and connecting them with a virtual network so they can talk to each other. The host machine, which is the machine where the network lives, views this network as if it were completely external. So, we will connect the network to the host. We’ve also included a bonus section about connecting the network to the internet so your containers can reach Google. You do not need a Linux machine to run through the exercises.
Containers and Networks: What Are They?
A container can be considered synonymous with a Linux network namespace. Keep this in mind. Essentially, a container is a namespace.
Each container runtime uses a namespace differently. For example, containers in Docker get their own namespace, while in CoreOS’ rkt, groups of containers share namespaces, each of which is …
Ray Chuan Tay June 23, 2020
If you’re like me and you’ve worked with Git for some time, you might have a couple of commands committed to your memory—from git commit for recording your changes, to git log for sensing “where” you are.
I have found git checkout to be a command that I reach for pretty frequently, as it performs more than one operation. But a single command doing more than one thing might produce a suboptimal user experience for someone learning Git. I can almost picture an XKCD strip:
Learner: What do I run to change the branch I’m on?
You: Usegit checkout <branch>.
Learner: What can I run to discard changes to a file?
You: Use…git checkout <file>.
Learner: OK…
Even if you have the commands memorized, there have likely been times when you had to pause after typing a git checkout command while you tried to match it with the operation you had in mind (e.g., “I just typed git checkout … to do X, but I thought git checkout does Y, does this really do what I want?")
Let’s take a look at what git checkout can do, and an alternative (or two) that can make for a friendlier user experience in Git.
Quick, what does git checkout do?
Perhaps you were trying something out and made some changes to the files in your local Git repository, and you now want to discard those changes. You can do so by calling git checkout with one …
Tyler Britten May 29, 2020
While running software in containers is very popular, it can be a little confusing to figure out the best way to get your code into a container. Now that the industry is mostly unified on Open Container Initiative (OCI) Standard container image formats, they can be built in any number of ways.
Building via Dockerfiles is the most commonly used approach, but there are also other tools that can make it easier with less learning upfront and some other advantages.
Dockerfiles
If you’re not familiar with the specification for Dockerfiles, you can find it here. The basic layout looks something like this:
FROM debian:latest
ADD my-app-file /app/
CMD /app/my-app-file
The first thing we need is a starting point, and in this case, we’re using a debian image, and the latest version. There are also ones that are language-specific like python or golang and ones tied to specific distributions.
The next lines include whatever steps we need to prepare the image, and the last line tells the image what command to run when the image is executed. There are a lot of variations of this but these are the basics. How can we make it better? Well a real application that is a bit more complicated would make this easier. Here’s a very simple golang http server application:
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", func (w http. …
David Dieruf May 21, 2020
Your app is destined for the cloud, but it’s going to meet some challenges along the way. First stop is the always fun whiteboarding session(s). Then come the sticky notes, which inevitably yield a backlog. Only when those two steps are complete does the Zen art of coding begin.
But ask yourself this: While whiteboarding the app’s design, how often is the developer’s local environment considered? Probably never. In fact, I bet during design a local environment doesn’t even make into the afterthoughts. Instead, it’s just one of those “figure it out” things.
Take, for example, the design of a microservice. Most likely it’s going to be dependent on an external configuration, like Spring Config. Ever consider how a developer is going to test on a config server locally? Do they have access to local containerization, like Docker? Or are they left to waste countless hours rigging up environment variables, only to find a totally different schema when the app is pushed to its platform?
I’ve been there and done that. It’s frustrating, wasteful, and breaks one of the most important of the 12 factors: No. 10, parity between environments. It’s also a personal favorite, and one I’ve been known to be ornery about.
The goal is not about following a good or bad design (although there’s plenty of room for bad decisions). It’s about …
Bryan Friedman May 7, 2020
Developers and VMware. The pairing might not make sense to you at first. As an application developer, maybe you have only limited experience working with VMware software. It’s probably just the place where your software runs on-premises. Or it’s the thing you get access to a couple of days after putting in that infrastructure request ticket.
But make no mistake, VMware Tanzu is for you. If you are a developer working in a large enterprise, or even a small- to medium-sized business, you are now being asked to build “modern” applications or to “modernize” your existing apps. VMware Tanzu is a collection of software created expressly to help you with this application modernization effort. It brings together innovations via VMware’s acquisitions of Heptio, Bitnami, Wavefront, and Pivotal with lots of open-source DNA from projects like Kubernetes and Cloud Foundry. And don’t worry, this site isn’t about infrastructure software like vSphere or NSX. Instead, it covers the topics you need to know to write modern software: Spring, .NET, Python, RabbitMQ, Kafka, CI/CD platforms, and much more.
App modernization
So now you see how a VMware-hosted developer site can be focused on app modernization. But what exactly is meant by “app modernization”? Is it simply the latest in a long line of technology fads that just means more work for you?
It can sometimes feel like that. Change is hard, …