Saturday at SCaLE was fun. I staffed the OpenVZ booth most of the day... from 10 AM until 6 PM. I did get a few breaks and a break for lunch. I had a number of people stop by the booth who mentioned they were using Proxmox VE and really liking it. Quite a few people had no idea what OpenVZ was and so I would start off back asking them if they use any virtualization products and the answer was usually yes. If they were a hobbyist user, their answer was usually VirtualBox. If they were a business type person their answer was usually VMware or KVM. I was ssh'ed into a couple of remote machines that were OpenVZ hosts so I was able to show what containers looked like by doing a pstree on the host and showing multiple init processes.
A guy from IllumOS dropped by the booth to ask if anyone was running KVM virtual machines inside of OpenVZ containers. Not to my knowledge. He said they were using Zones isolate KVM VMs on IllumOS. He seemed to have some concern that KVM wasn't secure/isolated enough and that users might be able to break out... and that the zones would keep everything safe. I mentioned sVirt for KVM but I've not used it myself.
Kir gave his presentation at 3PM on Checkpoint and Restore In Userspace (CRIU) which is a sub-project of OpenVZ. He said it went well and he filled his time but he didn't get a chance to actually show a demo... which was a shame because he had a nice video that showed it from beginning to end. Hopefully I can get him to share that video online RSN.
Kir also mentioned to me the commercial containers product that Parallels has to do Windows containers on Microsoft Windows... and that it was a monumental product with a high level of Windows internals knowledge on display... and that they were trying to work with Microsoft to share information... but that Microsoft didn't seem to interested. They are toying with the idea of possibly releasing OpenVZ for Windows but it seems unlikely. Containers on Windows has to bypass some kernel anti-patching technology Microsoft has so it can install the container functionality and then it has to re-enable it to keep the bad guys out. I'm not much of a Windows person and I definitely don't know Windows internals at all... but it was interesting even when dumbed down for me. :)
For lunch I thought I'd head over to Carls Jr. again but that place was packed... mostly with SCaLE attendees... so I ended up walking about a mile to a Burger King. That was a nice bit of exercise. Speaking of exercise, I decided that I wasn't going to use the elevator and take the stairs... which is a bit of exercise because as you may recall my room is on the 12th floor. I did that about three times down and up on Saturday.
A long time OpenVZ user named John Wenger from the L.A. Co-op stopped by to visit with Kir but he was away from the booth at the time. Late in the afternoon a guy from the Zenoss booth stopped by to say that he was going to give a demo using ZenPack (or something like that) that started up a few OpenVZ containers and showed off the monitoring capabilities of Zenoss. I commented on his "Bring IT" tee-shirt. After I had been talking with him for about 10 minutes it turned out that he was Daniel Robbins... who I have talked to a number of times on the #openvz IRC channel. Perhaps you recognize his name. Daniel was the creator of the Gentoo Linux distribution. These days he works on Funtoo. I jokingly asked him to sign my arm but he said I was weird. :)
Maybe it is because I'm in the L.A. area and I watch a lot of TV but one guy stopped by the booth who looked a lot like Hugh Laurie. If you don't know who that is, look it up. Anyway, he didn't know who Hugh Laurie was and hadn't been told he looked like him before. If only I had taken a picture. Anyway.
A few people reported that Java-based apps (like Jenkins I think) didn't perform well inside of a container and tended to spike I/O usage that would make the machine unresponsive for periods of time. Told them to check bugzilla.openvz.org and jump on an existing bug report if one existed or to file a new bug.
I got a chance to walk around both exhibit rooms and take pictures of all of the booths... but once I got them copied off of the SD card to my netbook I discovered that the vast majority of them were terribly blurry. I guess I was in too big of a hurry and wasn't holding the camera still long enough. The lighting in the place isn't that great but I didn't want to be pulsing a flash in everyone's eyes. Anyway, I dumped that set of pictures... so sorry, no pictures today like I had promised. I will make a concerted effort to take all new pictures.
I went by the OLPC booth and saw Caryl Bigenho. As you may recall, her and her husband Ed have a summer home outside of Bozeman and have visited every year. Caryl gave a presentation to the campus on the OLPC about two or three years ago... and she is the one that suggested we apply for a lending lab which netted us 10 OLPCs to share with the Montana community. I ended up going out to dinner at Denny's Restaurant (that is about half a mile from the Hilton) with Caryl, Ed, and a friend of theirs named Tony. Tony told me some about various OLPC deployments he has helped with in a few different countries in Africa. That was very interesting.
At 9PM I attended a Raspberry Pi Birds-of-a-Feather (Bof) get together. Caryl Bigenho lead it and asked everyone who had one or more Raspberry Pis and what they were doing with them. One guy who I think had an Australian accent but said he visited England at least once a month and that his son was involved with the MagPi magazine Kickstarter project. He said his son was now working on another Kickstarter project for an add-on power button for the RPi... and he showed a prototype microSD card adapter for the RPi that was different in that it was NOT the normal SDcard size. It was specifically made for the RPi SDcard slot so you could put a microSD card in it, plug it into the RPi and it does not stick out from the slot and is flush with the side of the board. I won't go into the various projects that people were doing because it was a very broad list (ham raid, First Robotics projects, co-location for RPi as a hosting server, etc). There were a couple of people from the Tiny-Core Linux booth and they said they had recently gotten Tiny-Core going on it and that it was the smallest and fastest Linux distro yet. They are showing it at their booth for anyone who wants to stop by and see it. Caryl mentioned that she had recently acquired an SDcard with Sugar (the OLPC learning environment) on it for the RPi. She hadn't had a chance yet to try it out but hoped to later in the day on Sunday.
There is a huge amount of interest in the RPi and even though other competitors may come and go in the space, the RPi has the numbers and add-ons and the people excited about it... that it should remain a viable platform for at least a few years.
This is my first time at the Southern California Linux Expo (SCALE) and it is their 11th year. You see, just a few days ago I had no plans to attend but then Kirill Kolyshkin contacted me via IRC asking if I was available to attend to help staff the OpenVZ booth. If you haven't heard of Kir before, he is the OpenVZ Project leader who is employed by Parallels. Having never attended SCALE I was very excited about going and checked with my two bosses (the wife and work) to see if I could go. Luckily both gave the thumbs up.
SCALE is actually FRIDAY - SUNDAY. Turns out that the Exhibit floor doesn't open until SATURDAY. Living in Montana my flight took me through Denver and by the time I got to Los Angeles and had ridden a free shuttle bus to the Hilton it was about 3PM PST.
I was so excited about going to SCALE, I had trouble sleeping Thursday night and add jet lag to that... I didn't have a whole lot of energy and went to bed around 8PM. Between hitting the hotel and going to bed though, I did do a few things.
1) I stopped by the exhibit floor to see how everyone was getting along setting up their booths. The exhibit floor is actually in two large rooms on the ground floor of the Hilton. One room is smaller than the other and the OpenVZ Booth (#93) is in the smaller of the two. I found the OpenVZ booth and I saw that Kir had already gotten it set up with a nice OpenVZ/CRIU banner and a large flat screen monitor.
2) I then checked in and got my room keys at the Hilton. I'm on the 12th floor. I went up to the room and got my netbook on the network to check my email and get on IRC. In IRC I saw Larry Cafiero. Larry is a SCALE promoter and PR person who is trying to work with a few of us in Montana to get a Montana Linux Fest in the not too distant future. Larry asked me to go to the Catalina C room to visit with him to touch base. I have visted with Larry several times at various shows (LFNW and UTOSC) when he was still associated with the Fedora Project. We chit chatted about the trip thus far and when I might start concentrating more on a Big Sky Linux Fest.
3) I attended Robyn Bergeron talk entitled, Managing the release and life cycle of an open source software project in a community. She jokingly said it as the longest presentation title ever.
4) Then I went to Carls Jr across the street from the Hilton and had one of their turkey burgers.
5) Then I went back to the room and the phone rang. It was Kir. His room is on the 3rd floor. He said he was doing some last minute work on his CRIU presentation (that is on Saturday at 3PM in the Century CD room) but that he wanted to get together to discuss the latest happenings in the OpenVZ world and what he has been working on... so I'd be better prepared for the booth. So, I went down to his room and we talked for about a hour or so. I got a gigantic brain dump worth of information. Turns out Kir (and his wife and two children) moved from the Moscow Russia area to the Seattle area sometime in December. Parallels has a small office there. Also in the same office is Linux kernel hacker and Parallels CTO of Server Virtualization, James Bottomley. You might have heard that James has been working lately on a secure boot setup for The Linux Foundation. Anyway, Kir mentioned that James wanted him to attend Matthew Garrett's Saturday morning keynote entitled, The Secure Boot Journey. Kir also wants to attend a presentation on Linux Native Containers (LXC) and of course he has his own presentation at 3PM. That means I'm going to be at the OpenVZ both for quite a while by myself. That's ok. Kir said that if there were any presentations I wanted to go to on Sunday, I could. I haven't really looked at the schedule yet.
Then Kir's wife and children (a boy and a girl) got back to the room after having toured around Hollywood. It just so happens that the Kolyshkin family had driven down from Seattle to L.A. which is quite the drive (about 1,000 miles or more?) although believe it or not, still shorter than some of Kir's flights over from Moscow to the US for various trade shows. They invited me to dinner but I was still full from the turkey burger and declined.
I went back to the room only to notice the Fedora Project had pushed out a considerable number of Fedora 18 updates including the 3.7.9 kernel and Firefox 19. I got my netwook all updated. I ssh'ed into my MontanaLinux build hosts at work and rebuilt with all of the updates. While that progressed I watched a little American Pickers on the History channel. It wasn't too long after that that I drifted of into the world of slumber.
I just woke up about 4AM PST and began writing this blog post. Getting up so early, I have about 5.5 hours before the opening of the OpenVZ booth. Wooo hooo.
I definitely have a lot to share from the talk I had last night with Kir but I'll wait until later to do so. Hmmm, I guess I do have to turn the light on in the room to get the coffee pot going. :)
I didn't bring my video camera because I thought I'd be stuck in the booth the whole time but I did bring a camera so expect lots of pictures from Saturday and Sunday. Assuming I have Internet connectivity (we as supposed to) at the booth, I'll be on IRC "live from the SCALE booth" just for the fun of it.
Oh, I guess I missed Jono Bacon's presentation late Friday about the Ubuntu Phone. In all honesty, I had no interest in attending.
It seems I've had a lot of questions about OpenVZ container migration lately on the #openvz IRC channel on the Freenode IRC network. While I made a silent screencast on that topic a few years ago, I thought it was time for a refreshed one so here it is. Enjoy.
What is an OpenVZ container? It is a form of virtualization where you can create a type of a virtual machine called a container that is basically a strongly isolated chroot environment with device and resource management features.
What is migration? It is the ability to easily move a container from one physical OpenVZ host to another. Live / online migration allows for no downtime and maintains existing network connections. Offline migration stops the container on the original host and starts it up on the destination host and as a result the containers uptime is reset and existing network connections are dropped. Watch the screencast for all of this in action.
You can also download this directly if desired. right-click, save link as:
openvz-vzmigrate.webm (12.8 MB)
In the last post I mentioned that I migrated from CentOS 4.9 to Scientific 6.1... and that certain aspects of this Drupal 4.7.x site were broken because of an incompatibility with PHP 5.3.x.
Downgrading a distro
Well, I decided to move from Scientific Linux 6.1 to Scientific Linux 5.7. EL5 offers both PHP 5.1.x and PHP 5.3.x and Red Hat announced a few weeks ago that they are extending the support lifecycle of both RHEL5 and RHEL6 from 7 years to 10 years. Migrating back to EL5 fixes the issues (knock on wood) that I was having with Drupal... but yet I can easily move to PHP 5.3.x at some point in the future if I so desire.
Doing EL major version upgrades
Two friends of mine happened to have CentOS 4.9 OpenVZ containers as well. They also run a number of services I'm less familiar with and weren't really versed enough with Linux to migrate their data like I did. In an effort to help them out, I looked into how to upgrade from EL4 to EL5. That really IS NOT supported or recommended but I thought I'd give it a try and see how it went. If it failed, I'd roll back to the original system. If it succeeded I'd keep it. After much work I *THINK* I figured it out. At least it worked for me in the particular situation I was dealing with. I started off with a page on the CentOS wiki about Upgrading from 4.4 to 5. I did not do a boot media based upgrade (I'm working with containers) so I did it strictly with rpm and yum.
I followed the instructions but they were written some time ago and were a bit outdated. So the first container I did took the longest because I was finding my way. Basically this happens in a few steps.
- Install the EL5 repos
- Manually download the core packages recommended and install them.
- Hopefully when you are done rpm is still working. If yum is broken, manually install a few more packages to make it work.
- With a working yum, upgrade everything else
- Turn off any new services that happen to be on by default that you don't want
- Find any stray packages left over from the previous release
- Fix your service configs by comparing your original service configs with the new ones
Read on to find out more of the nitty gritty details.
I posted a contributed OpenVZ OS Template today. The contributed OS Template is Scientific Linux 6 32 bit and it was contributed by Vic from powerpbx.org (email@example.com).
I asked him to share information about he created it and this is what he replied back with via email:
I have no plans to create a x86_64 version or provide regular updates to the x86 version at this time. The only reason I created the x86 version is because I needed a RHEL (or clone) v6 template for my own use. It is easy enough to update/modify/copy by someone else now that this version is out there.
I created it using this procedure and rsync from VMWARE to OpenVZ. Then I manually went through all the installed packages and took out as much as I could to get the size down. When in doubt I compared to the installed packages in a CentOS 5 template.
Yum would not remove kernel so I had to do a "rpm -e --nodeps kernel"
In the newly rsync'ed OpenVZ container I create a file called "vz.repo" in /etc/yum.repos.d with the following text:
then "yum install vzdev vzdummy-apache vzdummy-jre-el5 vzdummy-kernel-el5"
Could not get "vzdummy-glibc" to work. It caused the template to not load on reboot. Someone smarter than me will have to figure that one out. Perhaps vzdummy-glibc needs to be updated for RHEL 6.
Additional things I ran into that appear to be RHEL v6 specific are as follows.
You must comment out "console" in /etc/init/rc.conf and /etc/init/rcS.conf
You must also delete or rename tty.conf and start-ttys.conf.
I noticed Kir's blog post about the updated vzctl today. Cool! Finally I can create Fedora 14 containers... and the container restart mechanism has been fixed up too.
I downloaded the beta OS Template that the OpenVZ Projects offers for Fedora 14, created a container, did all of the updates, removed the samba* packages, added a few packages I wanted (mc, screen, links), and modified the httpd.conf so it is more like factory. Then I disabled a few services that aren't really needed... after all, who needs xinetd running when it it doesn't have any services configured? Then I stopped the container, cleaned up the container filesystem some, and tar.gz'ed it up and uploaded it as a contrib OS Template.
I did this for both the 32-bit and 64-bit OS Templates. Enjoy!
I don't usually repost mailing list messages but just got this one in my inbox from the OpenNode folks. Since I'm a big virtualization geek, I'm sharing. Haven't heard of OpenNode? Here's a brief description before I get to the status update email:
OpenNode is a open source server virtualization solution providing easy to use (CentOS / RHEL based) bare-metal ISO installer and supporting both OpenVZ container-based virtualization and emerging KVM full virtualization technology on the same physical host.
So, OpenNode is a lot like Proxmox VE except OpenNode is based on CentOS and uses libvirt, virt-manager, and other Red Hat standard tools.
Just wanted to mention a few news items from the OpenVZ Project.
Updated vzctl - vzctl 3.0.24 has been released. Even though the version number only changed from 3.0.23 to 3.0.24 there are a ton of changes, fixes and some feature additions. Of special interest is the --swappages option as well as being able to refer to a container by its name rather than requiring the CTID with vzmigrate. Over all it was a long overdue, much appreciated update.
Updated Official OS Templates - The last wiki notice is dated April 27th but looking today at the dates on the OS Templates they appear to have been updated May 27th. One thing to note is that there are now OS Templates for Ubuntu 10.04 which I'm sure Ubuntu folks will be happy about.
Beta Fedora 13 OS Templates - And speaking of OS Templates, Kir just released Beta OS Templates for Fedora 13. On the day Fedora 13 was released I tried creating my own OS Templates by taking Fedora 12 containers and updating them but ran into a snag. With Fedora 13 a lot of new stuff has been added to the init setup and some of it causes a container to just hang during startup. I was glad to see the beta OS Templates released. I created containers from them, made my own changes, and then uploaded those to the contrib section.
As luck would have it, later in the afternoon the Fedora Project released a whole bunch of updates and among them was a new initscripts package. I suspected that when I upgraded my container whatever changes the OpenVZ folks had made to the init setup that made it work in a container would be wiped out and I was correct as upgrading the initscripts package did make the container get stuck in the init process upon container reboot. I ended up filing two bugs: 1566 and 1567. I joyfully await their resolution.
2.6.32 devel kernel - There have been a few releases of the 2.6.32 devel kernel and it appears to be making progress. While there have been a number of OpenVZ devel kernels that died on the vine, 2.6.32 should be different mainly because it is the kernel in the upcoming Red Hat Enterprise Linux 6, the upcoming Debian 6, and in Ubuntu 10.04. I have no guess as to when it'll be marked stable. My guess would be sometime after RHEL 6 is released.
***Please note that any URLs mentioned (and the information they contain) in this posting are time sensitive and will surely be outdated not long after posting.
Shorewall and Proxmox VE Cluster Configuration
This is a follow up article describing how to use Proxmox VE and Shorewall together. This article focus on using Shorewall within your Proxmox cluster. If you have not read the first article I recommend that you do so, it will aid your understanding with what is going in this one.
Network Layout and Shorewall Configuration
We are going to be using a bridging configuration. This is what Proxmox VE uses with by default. Bridging allows for easy migration of hosts without having to re-configure the firewall each time a machine is migrated.