Scott Dowdle's blog
Odin and the OpenVZ Project announced the beta release of a new version of Virtuozzo today. This is also the next version of OpenVZ as the two are merging closer together. See their release announcement.
There will eventually be two distinct versions... a free version and a commercial version. So far as I can tell they currently call it Virtuozzo 7 but in a comparison wiki page they use the column names Virtuozzo 7 OpenVZ (V7O) and Virtuozzo 7 Commercial (V7C). The original OpenVZ, which is still considered the stable OpenVZ release at this time based on the EL6-based OpenVZ kernel, appears to be called OpenVZ Legacy.
Odin had previously released the source code to a number of the Virtuozzo tools (mailing list post) and followed that up with the release of spec-like source files used by Virtuozzo's vztt OS Template build system. The plan is to migrate away from the OpenVZ specific tools (like vzctl, vzlist, vzquota, and vzmigrate) to the Virtuozzo specific tools although there will probably be some overlap for a while.
The release includes source code, binary packages and a bare-metal distro installer DVD iso.
Bare Metal Installer
I got a chance to check out the bare-metal installer today inside of a KVM virtual machine. I must admit that I'm not very familiar with previous Virtuozzo releases but I am a semi-expert when it comes to OpenVZ. Getting used to the new system is taking some effort but will all be for the better.
I didn't make any screenshots yet of the installer... I may do that later... but it is very similar to that of RHEL7 (and clones) because it is built by and based on CloudLinux... which is based on EL7.
What is CloudLinux? CloudLinux is a company that makes a commercial multi-tenant hosting product... that appears to provide container (or container-like) isolation as well as Apache and PHP enhancements specifically for multi-tenant hosting needs. CloudLinux also offers KernelCare-based reboot-less kernel updates. CloudLinux's is definitely independent from Odin and the CloudLinux products are in no way related to Virtuozzo. Odin and CloudLinux are partners however.
Why is the distro based on CloudLinux and does one need a CloudLinux subscription to use it? Well it turns out that Odin really didn't want to put forth all of the effort and time required to produce a completely new EL7-clone. CloudLinux is already an expert at that... so Odin partnered with CloudLinux to produce a EL7-based distro for Virtuozzo 7. While CloudLinux built it and (I think) there are a few underlying CloudLinux packages, everything included is FOSS (Free and Open Source Software). It DOES NOT and WILL NOT require a CloudLinux subscription to use... because it is not related to CloudLinux's product line nor does it contain any of the CloudLinux product features.
The confusion was increased when I did a yum update post-install and if failed with a yum repo error asking me to register with CloudLinux. Turns out that is a bug in this initial release and registration is NOT needed. There is a manual fix of editing a repo file in /etc/yum.repos.ed/) and replacing the incorrect base and updates URLs with a working ones. This and and other bugs that are sure to crop up will be addressed in future iso builds which are currently slated for weekly release... as well as daily package builds and updates available via yum.
More Questions, Some Answers
So this is the first effort to merge Virtuozzo and OpenVZ together... and again... me being very Virtuozzo ignorant... there is a lot to learn. How does the new system differ from OpenVZ? What are the new features coming from Virtuozzo? I don't know if I can answer every conceivable question but I was able to publicly chat with Odin's sergeyb in the #openvz IRC channel on the Freenode IRC network. I also emailed the CloudLinux folks and got a reply back. Here's what I've been able to figure out so far.
Why CloudLinux? - I mentioned that already above, but Odin didn't want to engineer their own EL7 clone so they got CloudLinux to do it for them and it was built specifically for Virtuozzo and not related to any of the CloudLinux products... and you do not need a subscription from Odin nor CloudLinux to use it.
What virtualization does it support? - Previous Virtuozzo products supported not only containers but a proprietary virtual machine hypervisor made by Odin/Parallels. In Virtuozzo 7 (both OpenVZ and Commercial so far as I can tell) the proprietary hypervisor has been replaced with the Linux kernel built-in one... KVM. See: https://openvz.org/QEMU
How about libvirt support? - Anyone familiar with EL7's default libvirtd setup for KVM will be happy to know that it is maintained. libvirtd is running by default and the network interfaces you'd expect to be there, are. virsh and virt-manager should work as expected for KVM.
Odin has been doing some libvirt development and supposedly both virsh and virt-manager should work with VZ7 containers. They are working with upstream. libvirt has supposedly supported OpenVZ for some time but there weren't any client applications that supported OpenVZ. That is changing. See: https://openvz.org/LibVirt
Command line tools? - OpenVZ's vzctl is there as is Virtuozzo's prlctl.
How about GUIs or web-based management tools? - That seems to be unclear at this time. I believe V7C will offer web-based management but I'm not sure about V7O. As mentioned in the previous question, virt-manager... which is a GUI management tool... should be usable for both containers and KVM VMs. virsh / virt-manager VZ7 container support remains to be seen but it is definitely on the roadmap.
Any other new features? - Supposedly VZ7 has a fourth-generation resource management system that I don't know much about yet. Other than the most obvious stuff (EL7-based kernel, KVM, libvirt support, Virtuozzo tools, etc), I haven't had time to absorb much yet so unfortunately I can't speak to many of the new features. I'm sure there are tons.
About OS Templates
I created a CentOS 6 container on the new system... and rather than downloading a pre-created OS Template that is a big .tar.gz file (as with OpenVZ Legacy) it downloaded individual rpm packages. It appears to build OS Templates on demand from current packages on-demand BUT it uses a caching system whereby it will hold on to previously downloaded packages in a cache directory somewhere under /vz/template/. If the desired OS Template doesn't exist already in /vz/template/cache/ the required packages are downloaded, a temporary ploop image made, the packages installed, and then the ploop disk image is compressed and added to /vz/template/cache as a pre-created OS Template. So the end result for my CentOS 6 container created /vz/template/cache/centos-6-x86_64.plain.ploopv2.tar.lz4. I manually downloaded an OpenVZ Legacy OS Template and placed it in /vz/template/cache but it was ignored so at this time, I do not think they are compatible / usable.
The only OS Template available at time of writing was CentOS 6 but I assume they'll eventually have all of the various Linux distros available as in the past... both rpm and deb based ones. We'll just have to wait and see.
As previously mentioned, Odin has already released the source code to vztt (Virtuozzo's OS Template build system) as well as some source files for CentOS, Debian and Ubuntu template creation. They have also admitted that coming from closed source, vztt is a bit over-complicated and not easy-to-use. They plan on changing that ASAP but help from the community would definitely be appreciated.
How about KVM VMs?
I'm currently on vacation and only have access to a laptop running Fedora 22... that I'm typing this from... and didn't want to nuke it... so I installed the bare-metal distro inside of a KVM virtual machine. I didn't really want to try nested KVM. That would definitely not have been a legitimate test of the new system... but I expect libvirtd, virsh, and virt-manager to work and behave as expected.
Despite the lack of perfection in this initial release Virtuozzo 7 shows a lot of promise. While it is a bit jarring coming from OpenVZ Legacy... with all of the changes... the new features... especially KVM... really show promise and I'll be watching all of the updates as they happen. There certainly is a lot of work left to do but this is definitely a good start.
I'd love to hear from other users to find out what experiences they have.
Congrats Odin and OpenVZ! I only wish I had a glass of champagne and could offer up a respectable toast... and that there were others around me to clank glasses with. :)
This article is translated to the Bosnian language by Vlada Catalic.
Linux Weekly News had a write-up in their Weekly Edition last week... of Lennart Poettering's talk (Containers with systemd) at LinuxCon Japan 2015. That article should be available freely later this week... but I found a recording of what appears to be the same talk at a different event from April 2015. Here are the slides. Enjoy!
For those with iFrame issues, here's the direct link:
Here's some documentation on that stuff if you are looking for it.
Somehow I missed this when it was first posted (Feb. 24th, 2015) from the Collaboration Summit 2015... but here it is... Jon Corbet's most recent Kernel Report. Enjoy!
LinuxCon Japan is happening this week so hopefully an updated report soon?
If you haven't seen the pilot episode of USA's new series, "MR. ROBOT"... you can see the whole thing on YouTube... until they decide to take it down. Anyway, there was an exchange about GNOME vs. KDE and Linux is even mentioned. Here's the 1 minute clip. Enjoy!
I guess they haven't seen Plasma 5 yet. It's da'bomb!
If you didn't notice, Fedora 22 was released today. Today I refreshed the Fedora 22 OS Template I made for OpenVZ and uploaded it to contrib. For fun, I thought I'd build a MATE Desktop GUI container right in front of your eyes... and then connect to it via x2go.
Installing a desktop environment in a container can be fraught with danger for the uninitiated. The problem? Well, it always drags in NetworkManager, a graphical login manager, and various other packages / services that aren't really appropriate for a container. With a handful of systemd statements though, it is an easy fix. Watch and I'll show you how. Enjoy!
For those with iFrame issues, here's a direct link to the webm video:
You can pretty much use the same recipe for other desktop environments. The only thing you want to avoid are desktop environments that require accelerated 3D because those won't work over x2go. Which desktops use that? GNOME and Plasma 5... Cinnamon probably... and if you were on Ubuntu, Unity. XFCE, MATE, OpenBox, LXQT, etc work fine... although I haven't tried them all.
Since I'm such a big container fan (been using them on Linux since 2005) and I recently blogged about Docker, LXC, and OpenVZ... how could I pass up posting this? Some Canonical guys gave a presentation at the recent OpenStack Summit on "LXD vs. KVM". What is LXD? It is basically a management service for LXC that supposedly adds a lot of the features LXC was missing... and is much easier to use. For a couple of years now Canonical has shown an interest in LXC and has supposedly be doing a lot of development work around them. I wonder what specifically? They almost seem like the only company who is interested in LXC.. or at least they are putting forth a publicly noticeable effort around them.
Why Should You Care?
If Canonical can actually deliver on their LXD roadmap it is possible that it will be a suitable substitute for OpenVZ. The main "problem" with OpenVZ is that it is not in the mainline kernel, whereas LXC is. In practice you have to purposefully make an OpenVZ host (currently recommended on RHEL6 or clone) but with LXC/LXD any contemporary Linux system should be able to do full-distro containers... aka containers everywhere for everyone.
How About a Roadmap
Where is LXD now? Well, so far it seems to be mostly a technology preview available in Ubuntu 15.04 with the target "usable and production ready" release slated for the next Ubuntu LTS release (16.04)... which if you weren't familiar with their numbering scheme is 2016 April.
That's about a year away, right... so what do they still have left to do? If you go to about 23:30 in the video you'll get to the "Roadmap" section. They have work to do on storage, networking, resource management and usage reporting, and live migration. A bit of that falls within the OpenStack context... integrating with various OpenStack components so containers will be more in parity with VMs for OpenStack users... but still, that's quite a bit of work.
The main thing I care about absolutely being there is isolation and resource management which are really the killer features of OpenVZ. So far as I can tell, LXD does not offer read-only base images and layering like Docker... so that would be an area for improvement I would suggest. BTW they are using CRIU for checkpointing and live migration... thanks Parallels/OpenVZ!
Certainly LXD won't really make it no matter how good it is until it is available in more Linux distributions than just Ubuntu. In a video interview a while back (which I don't have the link handy for at the moment) Mark Shuttleworth stated that he hopes and expects to see LXD in other distributions. One of the first distros I hope to see with LXD is Fedora and that's the reason I tagged this post appropriately.
Broadening the Echosystem
Historically I've been a bit of an anti-Canonical person but thinking more about it recently and taking the emotion out of it... I do wish Ubuntu success because we definitely need more FLOSS companies doing well financially in the market... and I think Red Hat (and OpenVZ) will have an incentive to do better. Competition is good, right? Anyway, enjoy the video. BTW, everything they tout as a benefit of LXD over KVM (density, speed of startup, scalability, etc) is also true of OpenVZ for almost a decade now.
For those with iFrame issues, here's the YouTube link: LXD vs. KVM
Containers Should Contain
Let's face it, Docker (in its current form) sucks. Why? Well, ok... Docker doesn't totally suck... because it is for applications and not a full system... but if a container doesn't contain, it isn't a container. That's just how language works. If you have an airplane that doesn't fly, it isn't an airplane, right? Docker should really say it is an "Uncontainer" or "Uncontained containers"... or better yet, just use a different word. What word? I'm not sure. Do you have any suggestions? (Email me: email@example.com)
What is containment? For me it is really isolation and resource control. If a container doesn't do that well, call it something else. OpenVZ is a container. No, really. It contains. OpenVZ didn't start life using the word container. On day one they were calling them "Virtual Environments" (VEs). Then a year or two later they decided "Virtual Private Server" (VPS) was the preferred term. Some time after switching to VPS, VPS became quite ambiguous and used by hosting companies using hardware virtualization backends like Xen and VMware (KVM wasn't born yet or was still a baby). Then OpenVZ finally settled on the word "container".
If you want a fairly good history of the birth and growth of OpenVZ over the years, see Kir's recent presentation.
Hopefully LXD will live up to "container" but we'll have to wait and see.
One of the Virtuozzo folks sent a link to an OpenVZ survey that I filled out. It requires a Google account. I do have one but I try to avoid using it as much as possible.
Just wanted to share my answers to the, "What features are absent in OpenVZ from your point of view?" question.
1) Base images and layering like that of Docker. Docker mostly sucks but the ease and speed of deployment is amazing. The OpenVZ container creation tools... can they be adapted to use a pre-existing ploop image as a read-only base image?
2) Application containers. While I don't have a personal need for them quite yet I can definitely see how they are handy for developers as well as those into fleet computing.
3) qcow2 disk images are very popular with KVM. It isn't clear to me what benefits ploop offers over qcow2 or vice-versa. It would be nice if OpenVZ could use or convert qcow2 disk images.
4) Better OS Template tools. OpenVZ's vzpkg tool bit-rotted because there weren't enough developer resources to keep it alive. As a result OpenVZ's official OS Templates have been being built with the proprietary Virtuozzo tools for some time. I understand that is changing in the not too distant future with the public release of more of Virtuozzo's tools. I'm not familiar with those so I don't know how good they are... but yes, more attention to OS Template creation and management tools is needed. This is especially true if and when OpenVZ adds application containers and/or disk layering features.
5) Better integration with LXC in the mainline kernel. I think LXC and Docker could be a stepping stone to OpenVZ / Virtuozzo... if the OpenVZ tools worked reasonably well with LXC in the mainline kernel... and it was clear to the user what features they could gain if they moved up to OpenVZ and/or Virtuozzo.
6) An entry-level web panel. OpenVZ Web Panel seems somewhat popular but I've always been turned off by its reliance on Ruby... and unsure of its security-related testing. The recent Packt Publishing book, "OpenVZ Essentials" by Mark Furman spends half of the book covering OpenVZ Web Panel. It would be nice if OWP was adopted in some way or replaced with something similar to offer an entry-level web-based management system like VMware does with ESXi. If considered, I'd strongly recommend that there is a clear differentiation between the features in the entry-level web-panel and those commercially offered. I know a few companies are selling OpenVZ compatible web-interfaces... like SolusVM, Proxmox VE, etc.
7) More modern kernel support... but that is in-the-works.
I've been busy lately trying to learn more about Docker. I'm not much of a fan of "application containers" and still prefer a full-blown "distro container" like that provided by LXC (good) or OpenVZ (better)... but I have to admit that the disk image / layering provided by Docker is really the feature everyone loves... which provides almost instantaneous container creation and start-up. If OpenVZ had that, it would be even more awesome.
OpenVZ certainly has done a lot development over the past couple of years. They realized that simfs just wasn't cutting it and introduced ploop storage... and then made that the default. ploop is great. It provides for instant snapshots which is really handy for doing zero-downtime backups. I wonder how ploop differs these days from qcow2? I wonder how hard it would be to add disk layering features like Docker to OpenVZ with ploop snapshots?
Applications Containers In the Beginning
Ok, so Docker has taken off but I really can't figure out why. I mean Red Hat introduced OpenShift some time ago. First it was a service, then a product, and lastly a open source product that you can deploy yourself if you don't need support. A couple of years ago I attended an OpenShift presentation and at that time it provided "Gears" which were basically chrooted processes with a custom SELinux policy applied... and cgroup resource management? Something like that. While (non-OpenVZ) containers don't contain, with the SELinux added, OpenShift gears seemed to be secure enough.
OpenShift offered easy deployment via some git-based scheme (if I remember correctly) and a bunch of pre-packaged stacks, frameworks, and applications called "cartridges" which I see as functionally equivalent to the Docker registry.. It didn't have the disk image layering and instant startup of Docker so I guess that's was a minus.
These days I guess OpenShift is going to or has shifted to using Docker.
Docker Crawls Before It Can Walk
Docker started off using aufs but that was an out-of-tree filesystem that isn't going to make it into mainline. Luckily Red Hat helped by adapting Docker to use device mapper-based container storage... and then btrfs-based container storage was added. What you get as default seems to depend on what distro you install Docker on. Which of the three is performant and which one(s) sucks... again that depends on who you talk to and what the host distro is.
Docker started off using LXC. I'm not sure what that means exactly. We all know that LXC is "LinuX native Containers" but LXC seems to vary greatly depending on what kernel you are running and what distro you are using... and the state of the LXC userland packages. Docker wised up there and decided to take more control (and provide more consistency) and created their own libcontainer.
The default networking of Docker containers seems a bit sloppy. A container gets a private network address (either via DHCP or manually assigned, you pick) and then if you want to expose a service to the outside world you have to map that to a port on the host. That means if you want to run a lot of the same service... you'll be doing so mostly on non-standard ports... or end up setting up a more advanced solution like a load balancer and/or a reverse proxy.
Want to run more than one application / service inside of your Docker container? Good luck. Docker was really designed for a single application and as a result a Docker container doesn't have an init system of its own. Yeah, there are various solutions to this. Write some shell scripts that start up everything you want... which is basically creating your own ghetto init system. That seems so backwards considering the gains that have been made in recent years with the switch to systemd... but people are doing it. There is something called supervisor which I think is a slight step up from a shell script but I don't know much about it. I guess there are also a few other solutions from third-parties.
Due to the complexity of the networking and the single-app design... and given the fact that most web-services these days are really a combination of services that are interconnected, a single Docker container won't get you much. You need to make two or three or more and then link them together. Links can be private between the containers but don't forget to expose to the host the port(s) you need to get your data to the outside world.
While there are ways (hacks?) that make Docker do persistent data (like mapping one or more directories as "volumes" into the container or doing a "commit"), Docker really seems more geared toward non-persistent or stateless use.
Because of all of these complexities, which I really see as the result of an over-simplified Docker design, there are a ton of third-party solutions. Docker has been trying to solve some of these things themselves too. Some of Docker's newer stuff has been seen by some (for example CoreOS) as a hijacking of the original platform and as a result... additional, currently incompatible container formats and tools have been created. There seems to be a new third-party Docker problem solver start-up appearing weekly. I mean there are a ton of add-ons... and not many of them are designed to work together. It's kind of like Christianity denominations... they mostly believe the same stuff but there are some important things they disagree on. :)
Application Containers Are Real
Ok, so I've vented a little about Docker but I will admit that application containers are useful to certain people... those into "livestock" virtualization rather than "pet" virtualization aka "fleet computing". Those are the folks running big web-services that need dozens, hundreds or thousands of instances of the same thing serving a large number of clients. I'm just one one of those folks so I prefer the more traditional full-distro style of containers provided by OpenVZ.
Working On Fedora 22
I've already blogged about working on my own Fedora 22 remix but I've also made a Fedora 22 OpenVZ OS Template that I've submitted to contrib. Yeah, it is pre-release but I'll update it over time... and Fedora 22 is slated for release next week unless there are additional delays.
Like so many OpenVZ OS Templates my contributed Fedora 22 OS Template doesn't have a lot of software installed and is mainly for use as a server. For my own use though I've added to that with the MATE desktop, x2goserver, Firefox, LibreOffice, GIMP, Dia, Inkscape, Scribus, etc. It makes for a pretty handy yet light desktop environment. It was a little tricky to build because adding any desktop environment will drag in NetworkManager which will overpower ye 'ole network service and break networking in the container upon next container start. So while building it "vzctl enter" access from the OpenVZ host node was required. With a handful of systemctl disable / mask commands it was in working order again. Don't forget to change the default target back to multi-user from graphical... and yeah, you can turn off the display manager because you don't need that since x2go is the access method of choice.
BTW, there was a libssh update that broke x2go but they should have that fixed RSN.
Multi-purpose OS Templates
I also decided to play with LXC some on my Fedora 22 physical desktop. I found a libvirt-related recipe for LXC on Fedora. Even though it was a little dated it was very helpful.
The yum-install-in-chroot method of building a container filesystem really didn't work for me. I guess I just didn't have a complete enough package list or maybe a few things have changed since Fedora 20. I decided to re-purpose my Fedora 22 OpenVZ OS Template. I extracted it to a directory and then edited a few network related files (/etc/sysconfig/network, removed /etc/sysconfig/network-scripts/ifcfg-venet*, and added an ifcfg-eth0 file). I also chroot'ed into the directory and set a root password and created a user account that I added to the wheel group for sudo access.
After a minute or so for the minor modifications (and having left the chroot'ed environment) I did the virt-install command to create a libvirt managed LXC container using the new Fedora 22 directory / filesystem... and bingo bango that worked. I also added some GUI stuff and just like with OpenVZ I had to disable NetworkManager or it broke networking in the container. Anyway... running an LXC container is a like OpenVZ on a mainline kernel... just without all of the resource management and working security. Baby steps.
Containers Taken Too Far?
While hunting down some videos on Docker I ran into RancherVM. What is that? To quote from their description:
RancherVM is a new open source project from Rancher Labs that makes it simple to run KVM inside a Docker container.
What they heck? Run KVM VMs inside of Docker containers? Why would anyone want to do that? Well, so you can embed KVM VM disk images inside of Docker images... and easily deploy a KVM VM (almost) as easily as a Docker container. That kind of makes my head hurt just thinking about running a Windows 7 Desktop inside of a Docker container... but someone out there is doing that. Yikes!
Thanks to Vlada Catalic for translating this article into Bosnian.