Search This Blog

Thursday, May 1, 2014

KVM and Docker LXC Benchmarking with OpenStack


Linux containers (LXCs) are rapidly becoming the new "unit of deployment" changing how we develop, package, deploy and manage applications at all scales (from test / dev to production service ready environments). This application life cycle transformation also enables fluidity to once frictional use cases in a traditional hypervisor Virtual Machine (VM) environment. For example, developing applications in virtual environments and seamlessly "migrating" to bare metal for production. Not only do containers simplify the workflow and life cycle of application development / deployment, but they also provide performance and density benefits which cannot be overlooked.

At the forefront of Linux Container tooling we have docker -- a LXC framework / runtime which abstracts out various aspects of the underlying realization by providing a pluggable architecture supporting various storage types, LXC engines / providers, etc.. In addition to making LXC dead easy and fun, docker also brings a set of capabilities to the table which make containers more productive including; automated builds (make files for LXC images), versioning support, fully featured REST API + CLI, the notion of image repositories and more.

Before going any further, let me reiterate some of the major benefits of Linux Containers from a docker perspective: 

  • Fast
    • Runtime performance at near bare metal speeds (typically 97+ percent or bare metal -- a few ticks shaven off for bean counters).
    • Management operations (boot, stop, start, reboot, etc.) in seconds or milliseconds.
  • Agile
    • VM-like agility -- it's still "virtualization".
    • Seamlessly move between virtual and bare metal environments permitting new development workflows which reduce costs (e.g. develop on VMs and move to bare metal in the "click of a button" for production).
  • Flexible
    • Containerize a "system" (OS less the kernel).
    • Containerize "application(s)".
  • Lightweight
    • Just enough Operating System (JeOS); include only what you need reducing image and container bloat.
    • Minimal per container penalty which equates to greater density and hence greater returns on existing assets -- imagine packing 100s or 1000s of containers on a single host node.
  • Inexpensive
    • Open source -- free -- lower TCO.
    • Supported with out-of-the-box modern Linux kernels.
  • Ecosystem
    • Growing in popularity -- just checkout the google trends for docker or LXC.
    • Vibrant community and numerous 3rd party applications (1000s of prebuilt images on docker index and 100s of open source apps on github or other public sources).
  • Cloudy
    • Various Cloud management frameworks provide support for creating and managing Linux Containers -- including OpenStack my personal favorite.

Using google-foo (searching), it's fairly easy to find existing information describing the docker LXC workflow, portability, etc.. However to date I haven't seen many data points illustrating the Cloudy operational benefits of LXC or the density potential gained by using a LXC technology vs. a traditional VM. As a result I decided to provide some Cloudy benchmarking using OpenStack with docker LXC and KVM -- the topic of this post and a recent presentation I posted to slideshare.

Update 05/11/2014

The benchmarks have been rerun and v2.0 of the results is now available on slideshare (embedded below). An overview of the revision changes are available in the presentation and copied below for your convenience:

  • All tests were re-run using a single docker image throughout the tests (see my Dockerfile).
  • As the result of an astute reader, the 15 VM serial “packing” test reflects VM boot overhead rather than steady-state; this version clarifies such claims.
  • A new Cloudy test was added to better understand steady-state CPU.
  • Rather than presenting direct claims of density, raw data and graphs are presented to let the reader draw their own conclusions.
  • Additional “in the guest” tests were performed including blogbench.

In accordance with the v2.0 results, this post has been updated to remove direct claims of density factors or statistical references to performance benefits; I leave such inferences to the readers.

Please read the FAQ and disclaimer in the presentation.

OpenStack benchmarking with docker LXC

As luck would have it my favorite Cloud framework, OpenStack, provides some level of integration with docker LXC. Specifically there has been a nova virt driver for docker LXC (which includes a glance translator to support docker based images) since the Havana time-frame and now in Icehouse we have heat integration via a plugin for docker. The diagrams below depict the high level architecture of these components in OpenStack (courtesy of docker presentations).

docker nova virt driver and heat plugin for OpenStack

Rather than taking a generic approach to the Cloudy benchmarks, I set out with a specific high level use case in mind -- "As an OpenStack Cloud user, I want a Ubuntu based VM with MySQL" -- and sought to answer the question "why would I choose docker LXC vs KVM". Honing the approach in this fashion allowed me to focus in on a more constrained set of parameters and in particular to use a MySQL enabled VM / Container for the tests.

To drive the Cloudy tests from an OpenStack perspective, OpenStack project rally fits the bill perfectly. Rally allowed me to drive the OpenStack operations in a consistent, automated fashion while collecting operational times from a user point of view. Thus, rally was used to drive 3 sets of tests:
  • Steady state packing of 15 VMs on a single compute node (KVM vs. docker LXC).
  • Serial packing of 15 VMs on a single compute node (KVM vs. docker LXC).
  • Serial soft reboot of VMs on a single compute node (KVM vs docker LXC).
  • VM boot and snapshot the VM to image (KVM vs docker LXC).

While rally was used to drive the tests through OpenStack and collect Cloudy operational times, additional metrics were needed from a compute node perspective to gain incite as to the resource usage during the Cloudy operations. I used dstat to collect various system resource metrics at 1 second intervals (in CSV format) while the Cloudy benchmarks were running. These metrics where then imported into a spread sheet where they could be analyzed and graphed. The result was a baseline set of data reflecting the average operational times through OpenStack as well as compute node resource metrics collected during those tests.

The presentation in full can be found here:

I've also made the raw data results available below on github. Feel free to use them however you see fit. Note that I didn't have the bandwidth to format / label all the data clearly so if you have questions let me know and I'll do my best to describe / update.

Note that although the performance and density factor look promising using the docker integration in OpenStack, these components are still under heavy development and thus lack a full set of feature parity. I believe we will see these gaps closed moving forward, so I would encourage you to put on your python hat and help out if you can.

Finally, I will be discussing these results as well as providing a quick overview on realizing linux containers at some upcoming conferences:

If you attend either of these conferences I would encourage you to stop by my sessions and chat containers. 

Parting thoughts for docker image creators

Docker uses a SIGTERM signal to stop your container's processes. By default bash ignores SIGTERM and hence if your container's CMD is bootstrapped using a bash script (or invocation) you will experience longer than necessary stop times for your container unless explicitly trapping / handling the SIGTERM. I would encourage docker image developers to build awareness of this fact and develop their images with SIGTERM handling to ensure optimal container stop times.


  1. Told you so -

  2. Your load avg graphs thrash any credibility you might have had.

    1. Unfortunately your comment does not provide enough details for me act upon. Should you care to elaborate on your concern I'd be happy to revisit the data, rerun the tests, etc. to double check any potential errors in the data.

      Moreover you can find raw data collect from the system using dstat linked in the blog post above.

    2. I assume they are referring to the fact that load average is not a percentage. The load is the number of running or runnable processes, and needs to be scaled by the number of CPUs to get a % utilisation.

  3. FYI -- comments about your posting @

  4. Your findings look interesting. But, to make more out of your data you should combine the graphs together. For example in slide 14 you compare "Compute Node CPU" on KVM and Docker, but the difference could be more apparent if you combined the graphs. That way you could exhibit the differences between the too approaches and we as readers can see it clearly. The same applies for the graphs in slides 15 and 16. Also, in slide 24 you can compare "usr" between Docker and KVM and "sys" between the same.

    I didn't have time to go through the whole presentation, but it looked good so far.

    1. Thanks for the feedback... You can find a comparison of user CPU and memory both on the same graph on page 49 and 50 of the presentation.

      One problem with combining the graphs together is that the graph lines become a bit dense making the results hard to differentiate. I will consider such an approach if I publish subsequent iterations of the deck.

      The data in its raw format is provided in this blog post so that readers can make such comparisons as they wish. Moreover the raw data contains additional points such IOs, interrupts, context switches, etc..

  5. This is great. Thank you!

    With your 'image that responds to SIGTERM' you should also see faster serial VM boots, since your testing process seems to have been to issue 'destroy'.

    Also, if you could, I suggest updating the architecture diagrams to specify the Heat plugin as DockerInc::Docker::Container. Many of our older slidedecks had the wrong path set (the path changed when it went upstream). Newer decks the right module path! Thanks

    - Eric Windisch, @Docker

    1. Eric, v2.0 of the benchmark results have been posted to slideshare which update the docker OpenStack integration diagrams as per your comment. This blog post has also been amended to address as well.

  6. Your 26x better CPU number is bogus because you are comparing slopes of one line that is essentially flat with one that has a clear slope to it. By removing the first few data points from each line (where the data is flat, and appears to be pre-test run) the docker line slope can be made to be negative or arbitrarily small, so the comparison goes to infinity.

    What does appear to be happening there is some kind of CPU build-up for LXC that isn't happening for docker. Understanding why the behavior is different would be useful. The average CPU usage difference is more like 5x and appears to be measuring the startup cost of the container, not the running cost.

    1. Thank you for your comments.

      I will rerun the cloudy portions of the test taking into consideration your comment above and publish the results. In particular I will run an additional test:
      - boot 15 VMs in serial
      - wait 5 minutes for the CPU to stabilize
      - tear down the 15 VMs

      I believe the test above should provide greater incite as to the stable CPU usage.

      These tests are presented in full disclosure in an attempt to educate the community and therefore I welcome feedback.

    2. Adrian,
      v2 of the benchmarks have been completed and uploaded to slideshare. In addition this post has been updated to remove direct claims of density factors -- that is an exercise left to the reader.

      Additional comments / feedback is welcome & appreciated.

  7. Thanks for review, it was excellent and very informative.
    thank you :)

  8. Thank you. You have given to the community freely and transparently. Your contribution deserves to be recognized.

  9. Your network test is not good enough because you tested only with 1Gbps NIC and both vm and container reached the goal, how about 10/40Gbps?
    What was the the CPU & Memory usage during the test?

    1. I believe other parties have published similar benchmarks since this initial post. Perhaps you'll find the details you're looking for in their results.

  10. Very helpful article, your test results may just save my skin.


  11. Would be great info!
    How about Open VZ with Docker?

  12. Like the article, especially "major benefits of Linux Containers".

  13. Awesome article. Linux containers (LXCs) has long way to go.

  14. thanks for sharing,hope it will be helpful for too many people that are searching for this topic
    Google Cloud Platform Training
    GCP Online Training
    Google Cloud Platform Training In Hyderabad

  15. Poker online situs terbaik yang kini dapat dimainkan seperti Bandar Poker yang menyediakan beberapa situs lainnya seperti , kemudian,, dan yang paling akhir yaitu Jangan lupa mendaftar di panenqq silakan dicoba ya boss