Grails Profile: Netflix puts Lipstick on Apache Hadoop's Pig and More

July 18, 2013 Adam Bloom

Netflix has been doing quite a bit of work with open source software. In fact, they maintain 34 public repositories on GitHub. Since their development team’s focus on OSS stood out from many other Grails users like Disney, Lyris, Qualcomm, and ValueClick, we wanted to highlight a few of their recent Grails projects.


Lipstick built for a Pig

Certainly, the development team came up with quite a funny name, but the Lipstick project certainly doesn’t look like a joke. In this case, Lipstick does make a Pig look better.

For those working with Apache Hadoop, you are probably also familiar with Pig. If you aren’t familiar, we recently published an article that compares MapReduce, Hive, Pig, and Cascading with the well-known word-count example. Basically, Pig provides a higher level of abstraction for MapReduce programming in a syntax similar to SQL yet can also be extended with functions written in Java, Python, JavaScript, Ruby, or Groovy.

In a nutshell, Lipstick is a Grails app that provides a graphical view of a Pig workflow.

Screen Shot 2013-06-21 lipstick

As with most languages, complex applications can become hard to follow—making it difficult for developers to get work done. With Lipstick, it’s easier to maintain, debug, and monitor MapReduce jobs running Pig because we don’t have to sift through so many logs. The service runs on top of all Pig scripts and, in real time, reports status of each step in the workflow as it runs. As seen above, there is a useful user interface that goes well beyond a mouse-over for more info, and the app runs on Tomcat and MySQL.

ICE for Cloud Spending and Usage

ICE is another Grails project by Netflix, and it is built to provide an analytical view of cloud resources from a usage and cost perspective. It allows users to look at trends, organize reports into groups, and make better decisions.

There are three parts to ICE. First, the processor transforms the AWS programmatic billing access data into readable information. Two, a reader renders this data to the user interface. Then, the UI shows the graphs and tables. With ICE, you get visibility into accounts, regions, services, and usage types along with metrics on cost and use.


As you can see from the screen above, there are numerous filters that let you slice and dice interactively—you can see the cost per hour over time in chart views and table views. There are several other views that you can check out on the ICE project homepage.

More Netflix Open Source Projects and a Prize

There are a few other Netflix projects we think are worth checking out like:

  • Asgard—This project provides web interface and capabilities for automating deployments and managing clouds stacks. Asgard was the first Grails project at Netflix, and most of the other Netflix projects follow patterns from Asgard.
  • Blitz4j—Allows you to convert log4j appenders to use an asynchronous model for scale.
  • Genie—Provides a set of REST-ful APIs to run Hadoop, Hive, and Pig jobs as well as manage multiple Hadoop resources.
  • Pytheas—A framework for rapidly building dashboards to explore data.
  • Zuul—An API gateway processing system using functional Groovy scripts to dynamically maintain a smart routing fabric for requests

The Cloud-Prize contest might be the most interesting one to point out to Grails developers for now. When this contest ends in mid-September of 2013, some talented developer is going to win $10K USD, $5K USD in AWS credits, a trip to Vegas, and a ticket to the Amazon user conference, not to mention some solid notoriety. Basically, you need to build something cool or modify/improve one of their projects. Check out the Cloud-Prize contest page on GitHub for more information.

How to learn more about Grails

About the Author


Reading the Ruby Source to Understand Rails Idiosyncrasies
Reading the Ruby Source to Understand Rails Idiosyncrasies

Suppose somewhere within our Rails application, we have something like the following line of code. (10.days...

QML Timer Workaround in BB10
QML Timer Workaround in BB10

BlackBerry’s Cascades framework allows developers to add a lot of functionality in QML that would otherwise...

SpringOne 2021

Register Now