End to end testing in kubernetes

Something that I continue to be interested in is the possibility of end to end testing using kubernetes. The project has come a long way since its launch on github a little over a year ago, and it is interesting to see where it might head next. The purpose of this post is to explore the focus of my own particular fascination with it – the possibility of end to end testing complex workflows built of multiple microservices.

Currently kubernetes does have an end to end test suite, but the purpose of this seems largely to be for self service of the kubernetes project. However, there are the seeds of something with broader utility in the innards of the beast. Some contenders downstream have started building ways to render the possibility of writing end to end tests for applications easier, such as fabric8-arquillian for openshift applications. More recently, about twenty days ago, there was a pull request merged that looks like it might go a step further towards providing a more user-friendly framework for writing such tests. And the documentation for the system is starting to look promising in this regard, although a section marked examples therein is still classified as ‘todo’.

But why this obsession with end to end testing this way? Why is this so important? Why not simply as a tester write browser tests for an application?

The problem is that browser tests do not capture a lot of what is going on behind the scenes, but really only the surface layer, the user facing component of an application. To truly get more immediate feedback about whether or not a particular part of a workflow is broken, one needs something that can plug into all the backend machinery. And, in the world of the modern stack, that essentially means something that can plug into containers and trigger functionality / or monitor things in detail at will.

Kubernetes is of course not the only technology that has promise here. The more fundamental building block of Docker is starting to become quite versatile, particularly with the release of Docker 1.9, and the Docker ecosystem (Docker compose, swarm, and machine) may yet have a role to play as a worthy competitor, in the quest for the vaunted golden fleece of the end to end testing champion merino.  Indeed, anything that might well provide a practical and compelling way to end to end test an application built out of multiple containers is of interest to me, and although I am currently sitting on the kubernetes side of the fence, I could well be convinced to join the docker platform crowd.

My current thinking of a system that provides great end to end testing capability is the following: it should use docker (or rkt) containers as the most atomic components. One should have the ability to hook into these containers using a container level or orchestration service level API and activate various parts of the micro service therein which have been built in such a way as to facilitate a proper end to end test. There is however a problem here though, and it is due to the need for the security of the system to be watertight. So altering production code is understandably not particularly desirable here.

Instead, it would be great to somehow be able to hook into each docker container in the pathway of an application, and read the logs along the way (this can be done today). Also, however, it would be great to be able to do more. One idea might be to write code in the ‘test’ folder of the application next to the unit tests, that will only be run when the testing flag is set to true. Or, alternatively, maybe one could add api endpoints that are only opened in the staging environment, and are switched off in production (although this still of course may introduce unacceptable risks). If one had these additional endpoints, one could then read the state of particular variables as data flows through the application in the test, and check that everything is functioning as it should be.

The alternative to directly debugging code in this way of course is to simply log the value of a variable at a particular point, and then read the logs. Maybe this is, indeed, all that is required. Together of course with write tests for all of the standard endpoints of each container, and checking that they are working as they should be. There might be an elegant way of writing helper libraries to log something out to a docker log file for purposes of testing only, and then expose this information cleanly via some form of endpoint.

Anyway, the area of end to end testing multi container applications is quite an interesting one and I am keen to see how it develops and eventually matures. Together with monitoring tools like Prometheus, there is a large amount of potential for good work to be done in this direction and I am excited about where things might eventually lead with this.


Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: