Headless UI Testing with Go, Agouti, and Chrome

November 17, 2016

Foreword

Acceptance tests are a fantastic way to ensure - from a user perspective - that your application works correctly. A popular UI acceptance test library is Agouti. This post will detail how to write a small Agouti acceptance test and run it in a containerized environment; one which mimics a CI environment.

In this blog post we’ll be using the following:

Go - testing language
Ginkgo and Gomega - test framework and matchers
Agouti - our webdriver client and UI testing library
ChromeDriver - webdriver
Xvfb - a virtual framebuffer X server to allow chrome to run headlessly
concourse.ci - CI server, whose tasks run on Ubuntu Docker containers

Note: All code is assumed to live at $GOPATH/src/website (feel free to change website to anything)

Writing a simple web app

Let’s create a simple main.go:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("<html><body>Hello World</body></html>"))
	})
	fmt.Println("Starting on :8080")
	fmt.Println(http.ListenAndServe(":8080", nil))
}

We can now go run main.go and navigate to localhost:8080 and see our UI!

Writing a simple test

First, let’s get our dependencies:

go get github.com/onsi/ginkgo/ginkgo
go get github.com/onsi/gomega
go get github.com/sclevine/agouti

Then, let’s set up our suite - website_suite_test.go:

package main_test

import (
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"

	"github.com/onsi/gomega/gexec"
	"github.com/sclevine/agouti"
	"os"
	"os/exec"
	"testing"
)

var (
	agoutiDriver   *agouti.WebDriver
	websiteSession *gexec.Session
)

func TestWebsite(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, "Website Suite")
}

var _ = BeforeSuite(func() {
	agoutiDriver = agouti.ChromeDriver()
	Expect(agoutiDriver.Start()).To(Succeed())

	startWebsite()
})

var _ = AfterSuite(func() {
	Expect(agoutiDriver.Stop()).To(Succeed())
	websiteSession.Kill()
})

func getPage() *agouti.Page {
	var page *agouti.Page
	var err error
	if os.Getenv("TEST_ENV") == "CI" {
		page, err = agouti.NewPage(agoutiDriver.URL(),
			agouti.Desired(agouti.Capabilities{
				"chromeOptions": map[string][]string{
					"args": []string{
						// There is no GPU on our Ubuntu box!
						"disable-gpu",

						// Sandbox requires namespace permissions that we don't have on a container
						"no-sandbox",
					},
				},
			}),
		)
	} else {
		// Local machines can start chrome normally
		page, err = agoutiDriver.NewPage(agouti.Browser("chrome"))
	}
	Expect(err).NotTo(HaveOccurred())

	return page
}

func startWebsite() {
	command := exec.Command("go", "run", "main.go")
	Eventually(func() error {
		var err error
		websiteSession, err = gexec.Start(command, GinkgoWriter, GinkgoWriter)
		return err
	}).Should(Succeed())
}

And finally our test - website_test.go:

package main_test

import (
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
	. "github.com/sclevine/agouti/matchers"
)

var _ = Describe("Website", func() {
	It("Displays hello world", func() {
		page := getPage()
		defer page.Destroy()

		Expect(page.Navigate("http://127.0.0.1:8080/")).To(Succeed())
		Eventually(page).Should(HaveURL("http://127.0.0.1:8080/"))
		Eventually(page.Find("body")).Should(HaveText("Hello World"))
	})
})

We’ll need to make ChromeDriver available on our $PATH before we can run this on our local machine. Once that’s done, run ginkgo to see this work.

Setting up your CI environment

Concourse uses Docker containers to run tasks, so let’s create a container that can run our test. Create the following Dockerfile:

FROM ubuntu:14.04

ENV LANG="C.UTF-8"

# install utilities
RUN apt-get update
RUN apt-get -y install wget --fix-missing
RUN apt-get -y install xvfb --fix-missing # chrome will use this to run headlessly
RUN apt-get -y install unzip --fix-missing

# install go
RUN wget -O - 'https://storage.googleapis.com/golang/go1.7.linux-amd64.tar.gz' | tar xz -C /usr/local/
ENV PATH="$PATH:/usr/local/go/bin"

# install dbus - chromedriver needs this to talk to google-chrome
RUN apt-get -y install dbus --fix-missing
RUN apt-get -y install dbus-x11 --fix-missing
RUN ln -s /bin/dbus-daemon /usr/bin/dbus-daemon     # /etc/init.d/dbus has the wrong location
RUN ln -s /bin/dbus-uuidgen /usr/bin/dbus-uuidgen   # /etc/init.d/dbus has the wrong location

# install chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get update
RUN apt-get -y install google-chrome-stable

# install chromedriver
# NOTE: this is a relatively old version.  Try a newer version if this does not work.
RUN wget -N http://chromedriver.storage.googleapis.com/2.25/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip
RUN chmod +x chromedriver
RUN mv -f chromedriver /usr/local/bin/chromedriver

Let’s build our Dockerfile: docker build . -t my-ci-container

Running your test headlessly

Let’s now enter the container and run test scripts as if we were the concourse task:

First, enter the container with our app directory mounted:

docker run -it -v $(echo $GOPATH/src/website):/gopath/src/website my-ci-container

Let’s run the following in the container:

export GOPATH=/gopath
export PATH=$PATH:$GOPATH/bin
go get github.com/onsi/ginkgo/ginkgo
go get github.com/onsi/gomega
go get github.com/sclevine/agouti
service dbus restart
cd $GOPATH/src/website
xvfb-run TEST_ENV=CI ginkgo -v

Conclusion

We’ve now created an acceptance test to test a very small server, and had it run locally in a docker container. This blog post ends here, but in a real world environment you might use the Dockerfile we’ve created as the task image for a concourse.ci pipeline task, and use the commands that we ran manually as the pipeline task script.

Debugging

Chances are, something will go wrong along the way. Here’s what we’ve used to debug this stack:

Flow

test -> Agouti -> ChromeDriver -> google-chrome -> X server (via xvfb-run)

test->Agouti: Tests use Agouti

Agouti->ChromeDriver: Agouti talks to ChromeDriver over HTTP

ChromeDriver->google-chrome: ChromeDriver spins up google-chrome instances for each ‘session’ (in Agouti this is NewPage)

google-chrome->X server: The X server acts as the ‘gui’ for google-chrome. It can be started with Xvfb or by wrapping the chromedriver call with xvfb-run

Debugging ChromeDriver

ChromeDriver can manually be run with the chromedriver command. Helpful tips:

chromedriver --verbose reveals a lot of info, including the command it runs google-chrome with!
Agouti can spin up ChromeDriver itself with agouti.ChromeDriver().Start()
Agouti can spin up ChromeDriver with --verbose by crafting your own NewWebDriver command (.ChromeDriver() is a very light wrapper)
You can spin up your own ChromeDriver (ex. chromedriver --verbose), and point Agouti at it with agouti.NewPage("http://127.0.0.1:9515", ...) (9515 is the default port). The command NewPage is basically a curl -XPOST http://127.0.0.1:9515/session -d '{"desiredCapabilities": {}}'

Debugging google-chrome

Google-chrome can manually be run with the google-chrome command. Helpful tips:

curl -XPOST http://127.0.0.1:9515/session -d '{"desiredCapabilities": {}}' to create your own session - this is analogous to Agouti’s NewPage. When run after a chromedriver --verbose, it will give you a bunch of useful info.
Each curl -XPOST http://127.0.0.1:9515/session -d '{"desiredCapabilities": {}}' starts a google-chrome process. Each of these processes is a page!
Change how the google-chrome process is spun up by adding capabilities. For instance, you might want to have chromedriver spin up google-chrome processes with the --disable-gpu and --no-sandbox options (see below)! This would be:
```
curl -XPOST http://127.0.0.1:9515/session -d '{"desiredCapabilities": {"chromeOptions": {"args": ["disable-gpu", "no-sandbox"]}}}'
```

Here are some common errors that you might see if you were to try to run the google-chrome command:

Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation
not permitted
Illegal instruction

google-chrome is trying to do sandbox-y things that a container won’t let it do. Run with --no-sandbox option: google-chrome --no-sandbox

root@6a59557360dd:/gopath/src/website# [1120/020402:ERROR:nacl_helper_linux.cc(311)] NaCl helper process running without
a sandbox!
Most likely you need to configure your SUID sandbox correctly

This cryptic error probably means that google-chrome is having trouble connecting to the X server. Run xvfb-run google-chrome --no-sandbox, or start X server manually (xvfb-run wraps this):

Xvfb :99 &
export DISPLAY=:99
google-chrome --no-sandbox

libGL error: failed to load driver: swrast
[1308:1332:1120/020823:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process.

This container has no GPU. Run with --disable-gpu option: xvfb-run google-chrome --disable-gpu --no-sandbox

[1141:1152:1120/020700:ERROR:bus.cc(434)] Failed to connect to the bus: Failed to connect to socket
/var/run/dbus/system_bus_socket: No such file or directory

The dbus service hasn’t started. You can:

Check status with service --status-all
Restart dbus with service dbus restart (be sure to service --status-all after)
Modify /etc/init.d/dbus with log statements and the like to further debug

Xlib:  extension "RANDR" missing on display ":99".

You can ignore this error.

If all else fails, try upgrading to a newer version of ChromeDriver in your Dockerfile. The latest release can be identified here: http://chromedriver.storage.googleapis.com/LATEST_RELEASE

Don't mix goroutines and namespaces: Part 1

Update: fixed! In Go 1.10, goroutines work correctly with linux namespaces. I’m leaving the post up in cas...

Leveraging NSX's Features with BOSH's vSphere CPI

VMWare NSX is a network virtualization platform (frequently paired with the vSphere IaaS (Infrastructure as...

Headless UI Testing with Go, Agouti, and Chrome

Foreword

Writing a simple web app

Writing a simple test

Setting up your CI environment

Running your test headlessly

Conclusion

Debugging

Flow

Debugging ChromeDriver

Debugging google-chrome

Previous

Next

Headless UI Testing with Go, Agouti, and Chrome

Foreword

Writing a simple web app

Writing a simple test

Setting up your CI environment

Running your test headlessly

Conclusion

Debugging

Flow

Debugging ChromeDriver

Debugging google-chrome

Previous

Next

Most Recent

When writing a Java Spring web application that uses an OAuth2 single sign-on (SSO) service for authentication, testing can be difficult, especially if the SSO service is provided by a third...

My co-worker Belinda Liu turned to me and said, “I don’t like these tests at all; they’re hard to follow, and I’m not sure what they’re testing.” I looked at the tests that I had spent much of...

0. Abstract HAProxy is an optional load balancer included in the canonical open source Cloud Foundry deployment. Its intended use is on IaaSes (Infrastructures as a Service) that do not offer...

Scaling the Loggregator API So you’ve used this article to correctly scale Dopplers in your Loggregator system. Even so, you notice that you’re still experiencing log loss. It could be that your...

Why care about Dopplers You might be wondering what a Doppler is (and why you care about it). Doppler VMs are a core component of log and metrics transport; one that you probably won’t care about...

Studying the experience of Pair Programmers This is the raw data (after anonymization, and after the removal of freeform fields, out of an abundance of caution, so as not to leak any intellectual...

Studying the experience of XP Teams This is the raw data (after anonymization, and after the removal of freeform fields, out of an abundance of caution, so as not to leak any intellectual...

Pivotal Application Service for Windows introduced the -s windows stack name in PASW 2.4, reducing the operator and developer need to concern themselves with specific Windows Server versions. From...

Abstract “How much faster will my VM’s disks be if I upgrade my ZFS-based (Z File System) NAS to 10 GbE?” The disks will be faster, in some cases, much faster. Our experience is that sequential...

The Spring framework has grown and changed at a massive pace over the last years. It has evolved from XML configured beans to annotation based beans, from synchronous to a non-blocking and...

Overview In a previous post I explained how you could create several components to build a Netflix stack for local development. Now, I want to explain how Pivotal Cloud Foundry makes this much...

Overview A couple of recent projects I have been on have started our engagement with the Netflix stack described here, and because I wanted to have a way to quickly prototype, I set up this demo. ...

Introduction One way to extend the Kubernetes platform is by building custom controllers that operate on custom resources. We can leverage custom resources to enhance the cluster with features for...

Introduction The kubelet exposes many useful metrics that can be used for a variety of purposes. These metrics are already being scrapped by components like the Metric Server. The metrics from the...

Abstract Smartphone authenticator apps such as Google Authenticator and Authy implement software tokens that are “two-step verification services using the Time-based One-time Password Algorithm...

(This blog is the fourth installment of a four-part series) The Operator Pattern The Operator Pattern stipulates a process that is registered with the Kubernetes system layer, listening to...

(This blog is the third installment of a four-part series) Kubernetes can automatically provision “remote persistent” volumes with random names Several types of storage volumes have built-in...

(This blog is the second installment of a four-part series) By default, all containers are free from limits but subject to eviction By default, Kubernetes places very few limits on a container. A...

Kubernetes is available across all public clouds nowadays, including Pivotal’s own PKS, which runs in the cloud and can also be run “on prem”, on the premises of an enterprise. Kubernetes promises...

Abstract By using tcpdump to troubleshoot an elusive error, we uncovered a man-in-the-middle (MITM) ssh proxy installed by our information security (InfoSec) team to harden/protect a set of...