Skip to content


Fedora 20 Virtualization Test Day Report

So, it was Fedora Virtualization Test Day last Tuesday and I actually went down and took the occasion for some good testing of Xen on the next Fedora release (Fedora 20, codename Heisenbug). Fedora is going to ship Xen 4.3 (and there are not many other mainstream distribution doing that), so it is very important to try to make sure it will be as good as possible for Fedora users!

A lot of information on how to (well, how you should have… but it’ll be for next time ;-P) participate  to such event are available on our Wiki. What I am up to, here, is reporting how some of the tests I did that day went. Hopefully, this would give an idea of where we stand, regarding the integration of Xen in Fedora, as well as how well Xen itself works with Fedora’s default virtualization toolstack, i.e., libvirt.

Setting up the testing environment

Well, you at least need a Fedora 20 installation, in order to test Xen on Fedora 20. For the details, have a look at the already mentioned wiki page. Here I’m only going to say that I decided to go for a PXE-boot based install, which I did by downloading the following files:

 $ wget http://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/Fedora/x86_64/os/images/pxeboot/vmlinuz
 $ wget http://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/Fedora/x86_64/os/images/pxeboot/initrd.img

and by preparing an appropriate entry in my PXE server configuration (usually a file called pxeconfig.cfg):

label fedora-20btc1-amd64-s
    KERNEL fedora/20/x86_64/Beta-TC2/vmlinuz
    APPEND initrd=fedora/20/x86_64/Beta-TC2/initrd.img repo=http://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/Fedora/x86_64/os/ console=ttyS0,115200n8 text serial

Screenshot from 2013-10-08 15_19_37

Mind the console=ttyS0,115200n8 text serial in case you want to run the install on a serial console, like I’m doing in this case.

On a second test box, I did a proper graphical install (still via PXE). No big difference, really, just follow the guided procedure, then grab a coffe` and wait for this screen (on the right) to happear.

Installing Xen and rebooting into Dom0

After finishing installing the host, we need to install Xen, libvirt and some libvirt related tools. It’s all described in this other Wiki page, so let’s skip it here…. Just follow that instructions and reboot into the following:

Fedora release 20 (Heisenbug)
Kernel 3.11.2-301.fc20.x86_64 on an x86_64 (hvc0)

odyn login: root
Password:

# cat /etc/fedora-release 
Fedora release 20 (Heisenbug)
# sudo uname -a
Linux odyn.uk.xensource.com 3.11.3-301.fc20.x86_64 #1 SMP Thu Oct 3 00:57:21 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
# virt-what 
xen
xen-dom0

Well, I guess we can make a note of the first important fact of the Test Day:

  • Fedora 20 works quite nicely and straightforwardly as Xen Dom0!

Creating guests

Ok, let’s move forward to creating some guest. In Fedora, when you want to create and install a guest, especially a Fedora guest, you do it with virt-install. Period. So, let’s do that, a Fedora 20 PV guest, on a Fedora 20 Dom0, with virt-install, installed from the serial console too. It’s actually more easily done than said:

 # lvcreate -nf20_64 -L10G /dev/fedora_odyn
 # virt-install --paravirt --name f20_64 --ram 2048 --vcpus 4 -f /dev/fedora_odyn/f20_64 --network bridge=virbr0 --location http://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/Fedora/x86_64/os/ --graphics none
=====================================================================
Installation

 1) [x] Installation source               2) [!] Timezone settings
        (http://dl.fedoraproject.org/pu          (Timezone is
        b/alt/stage/20-Beta-TC1/Fedora/          not set.)
        x86_64/os/)                       4) [!] Set root password
                                                 (Password is not set.)
 3) [!] Install Destination               6) [!] Software selection
        (No disks selected)                      (GNOME Desktop)
 5) [!] Create user
        (No user will be created)
 7) [x] Network settings
        (Wired (eth0) connected)
  Please make your choice from above ['q' to quit | 'c' to continue |
  'r' to refresh]: 
[anaconda] 1:main* 2:shell  3:log  4:storage-log  5:program-log

Let’s now head to my second test box, and do something similar, which lead us where this screenshot shows:

Screenshot from 2013-10-08 16_50_31

I also created another PV guest and an HVM guest there, with similar procedures. From all this, we can reasonably assess the following:

  • Fedora 20 works fine both as a PV and HVM Xen guest.

Playing with the guests with virsh

Now, what about, seeing what we have running:

# virsh list
Id Name State
----------------------------------------------------
39     F20-HVM                               running
40    fedora20                               running

Pausing and resuming both the PV and HVM guests:

# virsh suspend F20-HVM
Domain F20-HVM suspended
# virsh suspend fedora20
Domain fedora20 suspended
# virsh list
Id Name State
----------------------------------------------------
39      F20-HVM                               paused
40     fedora20                               paused

# virsh resume F20-HVM
Domain F20-HVM resumed
# virsh resume fedora20
Domain fedora20 resumed
# virsh list
Id Name State
----------------------------------------------------
39     F20-HVM                               running
40    fedora20                               running

And, finally, saving-&-restoring one of them

# virsh save fedora20 /tmp/savefile
Domain fedora20 saved to /tmp/savefile
# virsh list
Id Name State
----------------------------------------------------
39      F20-HVM                              running

# virsh restore /tmp/savefile
Domain restored from /tmp/savefile
# virsh list
Id Name State
----------------------------------------------------
39      F20-HVM                              running
41     fedora20                              running

I also tried importing and cloning a VM, as described here and here, and it all worked.

Any issues, then? There indeed was one. Basically, it looks like reaching the guest’s PV console via virsh does not work, while it is fine with xl console:

# virsh console fedora20
Connected to domain fedora20
Escape character is ^]
error: internal error: cannot find character device (null)

# xl console fedora20
Fedora release 20 (Heisenbug)
Kernel 3.11.2-301.fc20.x86_64 on an x86_64 (hvc0)

fedora20 login: root
Password:
[root@fedora20 ~]#

And I will of course report that to the appropriate mailing list/bugzilla.

What’s there, what’s missing

The previous section reveals that not only Xen is straightforward to install and works quite well on Fedora 20 as a Dom0, and that Fedora 20 works quite well as a Xen PV or HVM guest. It also shows how the basic VM lifecycle of a Xen guest, in Fedora 20, can be handled nicely enough with libvirt and the related tools (virt-install, virt-manager, virt-viewer, etc.). That of course does not exclude the possibility of using Xen’s default command line toolstack (xl).

The only two relevant missing features, at the time of writing, in the libvirt libxl driver are:

  • PCI Passthrough
  • live migration

Yes, big ones, I know. However, consider the following:

  1. that does not mean that PCI Passthrough and migration does not work on Xen on Fedora at all. They do work via the xl toolstack, they are just not available via libvirt;
  2. this is going to be solved soon, as the libvirt libxl driver maintainer Jim Fehling reported recently on xen-devel. In fact, this is the patch series for PCI Passthrough, and this is the patch series implementing live migration, and there are pretty good chances that both these patches make it in libvirt before Xen 4.4 release time (so, not in time for Fedora 20, but still not bad at all).

So, stay tuned since, as Jim says, “Slowly, with each libvirt release, the libxl driver is improving”.

Conclusions

Fedora really does a great job with these Test Days. All of it: the planning, the managing, the reporting… An example that many other project should look at and try to follow (and actually, Xen Project is trying, as we also started having Xen Test Days).

Participating to the latest Fedora Virtualization Test Day has been really nice, although we need to do a better job in convincing more Xen folks to be there and do some Xen specific testing. Anyway, I am really glad to have had the chance to verify how well Xen 4.3 will work on Fedora 20.

It is actually quite important that we get a good Xen on Fedora test coverage, at least as far as running the next release of Fedora as a Xen DomU is concerned. In fact, being a functional Xen guest is one of the release blockers for a Fedora release, as in, if release X does not work as a Xen guest, it can’t be released!

Since testing Xen on Fedora is, for the most part, testing Xen integration with libvirt, what about producing some libvirt test-cases for OSSTest? That would be very cool, and we are already working on it. Another interesting thing would be to also have OSSTest could try to build and run Xen on various distro (as host), instead than using only Debian, as it is doing right now. This is a bit more tricky than the above, but we are thinking at how to do that too (standalone mode could, perhaps, help).

Posted in Community, Uncategorized, Xen Case Study, Xen Hypervisor.

Tagged with , , , , , .


Xen Project User Summit Videos Available

xen_user_summit_bgFor those community members, who could not attend the Xen Project User Summit in New Orleans, we now published all the videos on our video stream. For your convenience, we also created a portal page that links to all recordings of the user summit. Again, a big Thank You to our speakers and to Russell Pavlicek for organizing the event.

Event Recordings and other information on Events

events-on-videoIn future, we will be recording all Xen Project events such as user and developer summits. You will be able to find recordings of Xen Events on the events page in a box marked Xen Events On Video. Also, if you want to know if there are any Xen talks at an event you are planning to attend, why not check out our events calendar?

Don’t Forget: Xen Project Developer Summit is around the corner

A quick reminder: our Developer Summit is just round to corner! We have a great line up of topics covering the latest developments in server and cloud to new frontiers in virtualization in mobile, automotive and embedded!

Posted in Events.

Tagged with , .


Fedora 20 Virtualization Test Day is today!

Fedora Logo

Yes, today (Tuesday, October 8th) is one of the Fedora 20 Test Days, more specifically, Virtualization Test Day.

Specific information regarding testing Xen on the new Fedora can be found in this Wiki page. For attending and participating, join us now on IRC at #fedora-test-day (Freenode)!

Fedora 20 will be one of the first mainstream distros shipping Xen 4.3, so come and help us making sure it will work great for you and all Fedora users!!

Posted in Announcements, Community, Events, Xen Development.

Tagged with , , , , , .


Xen Project Developer Summit Line-up Announced

xendevsummit

Last week, we announced the program for the Xen Project Developer Summit on the Xen Mailing lists. This year, we have a fantastic line-up covering topics from Xen Development, Cloud Computing, Xen on Mobile Devices, Graphics Virtualization and new and interesting use-cases for Xen. Half of the available spaces are already gone: if you were planning to come please register quickly.

More Information

You can find more information about the event in the following locations:

  • Event Website
  • Accommodation :  Note that the room block had to be already extended once
  • Visas: If you need a visa, please register first and then fill out a visa request

Posted in Announcements, Events.

Tagged with .


OSSTest Standalone Mode Step by Step

Xen has long history and many features. Sometimes even experienced developers cannot be sure whether their new code is regression-free. To make sure new code doesn’t cause regression, Ian Jackson developed a test framework called OSSTest. In order to make this framework usable for individual ad-hoc testing, standalone mode was introduced. Recently I played with it and found it useful to share my experience with the community.

Basic Requirements

To run OSSTest in standalone mode, you need to have at least two machines: one is controller, which runs OSSTest itself and various other services, the other is test host, which is used to run test jobs.

For the controller you need to have several services setup:

  • DNS / DHCP, use to translate IP < -> hostname
  • Webserver, provide test box accessible web space, OSSTest will expose configurations / overlays via HTTP and detect test host status via webserver’s access log.
  • PXEboot / Tftp, used to provide Debian installers to test host.
  • NTP, optional

For the test host you need to:

  • enable PXE boot
  • connect to PDU if necessary

Step by step setup

git clone OSSTest tree, master branch, we assume you run everything inside osstest.git directory.

Have a look at README / TODO which contains useful information. You can create a global config file in ~/.xen-osstest/config. In standalone mode you can also create standalone.config in osstest.git directory to override global configurations.

In the configuration you can specify DNS name, name server etc.

DnsDomain uk.xensource.com
NetNameservers 10.80.248.2 10.80.16.28 10.80.16.67
HostProp_DhcpWatchMethod leases dhcp3 dhcp.uk.xensource.com:5556
TftpPath /usr/groups/netboot/

DebianNonfreeFirmware firmware-bnx2
DebianSuite squeeze
DebianMirrorHost debian.uk.xensource.com
DebianPreseed= < <’END’
d-i clock-setup/ntp-server string ntp.uk.xensource.com
END

Debian is the de-facto OS in OSSTest, you can find more info in Osstest/Debian.pm. Here we use Squeeze to build binaries and run test jobs, because there’s a bug in Wheezy’s Xen-tools which breaks xen-image-create, which eventually breaks ts-guest-start. Patches to fix that problem have been posted but not yet merged. Hopefully some day in near future we can use Wheezy to run build jobs and test jobs.

Configure test host in OSSTest config file. There can be multiple test hosts in the config file, but I only have one. Here is what I have:

TestHost kodo4
HostProp_kodo4_DIFrontend newt
HostProp_kodo4_PowerMethod xenuse
HostProp_kodo4_Build_Make_Flags -j16
HostFlags_kodo4 need-firmware-deb-firmware-bnx2

There is detailed explanation of what those paramters mean in README. An interesting one is PowerMethod, which in fact points to a module in OSSTest that controls power cycle of the test box. Have a look at Osstest/PDU for all supported modules. Use manual.pm if your test host is not capable of doing power cycle automatically.

Before actually running OSSTest, you need to make some directories yourself.

  • logs: used to store tarballs from build-* jobs
  • public_html: expose this directory via HTTP server to the test host
  • $TFTPROOT/osstest: OSSTest will write PXE boot information for test hosts here

Make sure test host is able to access contents in public_html, then you can have “WebspaceUrl http://YOUR.CONTROLLER/public_html” in your OSSTest config. Test host will try to fetch all sort of things from there.

Next step will be setting “WebspaceLog /PATCH/TO/WEBSERVER/ACCESS.LOG”. OSSTest watches webserver access log. When test host / guest go to fetch things via HTTP OSSTest gets to know their status. I use Apache2 so I’ve set WebspaceLog to /var/log/apache2/access.log which just works.

Have Debian PXE installers ready. Remember the “DebianSuite” option in your config file? In order to make OSSTest fully functional you will also need to place Debian PXE installers in the right place. You can grab Debian’s PXE installers from any of the Debian archives around the world. And put them under the TFTP you just set up. I would recommend having at least amd64 and i386 in place. Make sure installers are accessible from the test host before proceeding.

By now we’re all set! Next step:

./standalone-reset

This will reset everything in standalone mode and create standalone.db, which includes test jobs and runtime variables. You can peek what’s inside that database with sqlite3.

The first job to run should be a build-* job. That can: 1) verify your setup is correct; 2) generate a bunch of runtime variables for subsequent test-* jobs.

./sg-run-job build-amd64 # WARNING: this will wipe out your test box

If the job pass, you’re all set. You can play around with other jobs. The default setting of jobs is to wipe out test box on every run. If you don’t want that you need to specify OSSTEST_HOST_REUSE=1 as stated in README.

An example of what I run:

./sg-run-job build-amd64-pvops
OSSTEST_HOST_REUSE=1 ./sg-run-job test-amd64-amd64-xl

If you only want to run a specific testcase, you can try OSSTEST_JOB=$JOBNAME ./ts-XXX host=$TESTHOSTNAME.

Customized tree / revisions

By default OSSTest always fetches trees and revisions from Xenbits. You can easily override them with standalone.config.

Say if I want to test a specific revision of Xen, I have:

export REVISION_XEN=c5e9596cd095e3b96a090002d9e6629a980904eb

in my standalone.confg.

You can look at make-flight to know all the interesting environment variables. (Sorry, no document yet)

Writing new testcases

If you’re interested in writing new testcase, you can do that in two simple steps:

  1. write ts-my-test-case script, you can use any existing testcase as template (they are prefixed with “ts-”)
  2. modify sg-run-job, which has the information for which testcases to run for a specific job

Do have a look as osstest/TestSupport.pm, in which you can find lots of helpers to accomplish your task.

Writing new test job

The script responsible for making jobs is cs-job-create. When you run OSSTest in standalone mode, it is probably more useful to modify make-flight. You also need to modify sg-run-job to link your new job with testcases.

Hope the above information can help you get started with OSSTest. If you have any problem, do come to Xen-devel and ask.

Some readers may recall the recent announcement of open-sourcing XenRT, and wondering about the relationship between OSSTest and XenRT. Long-term, we expect that XenRT will mature as an open development project and eventually displace OSSTest. But that is not likely to happen for another six months to a year. Developing OSSTest has benefits for the current development, and we hope that getting people involved in writing test cases for OSSTest will be of service in helping people write test cases for XenRT when it becomes available. So we have decided to continue to develop OSSTest until XenRT is ready to replace it.

Have fun! :-)

Posted in Xen Development.


Xen Project Alive and Well at LinuxCon/CloudOpen North America

The Xen Project was well represented at LinuxCon North America and CloudOpen North America.  Sponsored by the Linux Foundation, the two co-located conferences featured a number of Xen-related talks, as well as the first Xen Project User Summit (which will be discussed at length in a post to follow).

Lars Kurth gave a well-received talk about the lessons we’ve learned from the Xen Project over the years.  It included a number of insights from the past decade, and a practical tool called “The Community Funnel Model” which can be used to analyze activity within an Open Source community.

I gave a talk about securing a cloud using Xen.  It identified many of the advanced security features included in the Xen hypervisor.  Thanks to George Dunlap for assembling most of the material, and Stefano Stabellini for the ARM information.

The slide decks for these talks can be found on XenProject.org.  As the recordings for the sessions become available, we intend to post them there as well.

Lars (who in addition to being the Xen Project Community Manager is also the unofficial community photographer) shared a number of pictures on the Xen Project Google+ page.  Albums include the Mardi Gras-style party and pictures from the Xen Project User Summit.  See if you can pick out your friends (especially difficult if they happen to be wearing the festive Mardi Gras gear).

However, some of the most important outcomes of the event could be found in the press coverage.  ServerWatch produced a piece called, “How Xen Virtualization Got Its Zen Back: LinuxCon“.  Lars was featured in a Cloudcast focusing on the latest news from the Xen Project.  And when eWeek put together its slideshow of LinuxCon luminaries, Lars made the deck as well!

And stay tuned for a summary of the Xen Project User Summit in the next few days.  Slide decks from many of the talks are already on the XenProject.org website.

Posted in Events.


QEMU vs. qemu-traditional update

Here is an update about feature completeness of QEMU compared to the old qemu-traditional.

But first, what is the difference between QEMU and qemu-traditional?

QEMU is the software that can be found at qemu.org, we can also call it QEMU upstream. It’s where all new features are supposed to land.

What we call “qemu-traditional” is the fork of QEMU that has been used by Xen. It became harder and harder to maintain and to upgrade with recent version QEMU, so we could not benefit from some of the new features that have been developed in QEMU, and also any bug that we would have found in the fork can be hard to fix upstream because the code would be very different.

So, we took everything that was needed from qemu-traditional to run QEMU with Xen and integrate them in a modern QEMU. Up to now, few features were missing to be able to use QEMU, but now, all the main features are in and QEMU became the default for most usage.

So what’s new in 4.3?

An important feature to be able to live-migrate a guest is a way to be able to track memory that has been modified during a live-migration. The feature first appears in Xen 4.2.2.

We also added the support to CPU hotplug for HVM as it was one of the missing features to get QEMU closer to qemu-traditional and have it as default.

Missing feature?

We are still missing a few things with QEMU upstream, so far, the limited support for VGA passthrough has not been upstream. Another missing feature would be the use of QEMU in a stub-domain instead of using qemu-traditional as it is right now. This last one is planned to be fixed in hopefully 4.4.

There is also patchs to fix the suspend resume cycle of an HVM guest that should be applied soon.

Beside those missing features, there are more works going one to enhance the support of many QEMU features that are not usable right now with the tool stack libXL, like using USB redirection or USB passthrough or even one day supporting QXL.

Posted in Xen Development.

Tagged with , .


Peek Preview : Xen Project Developer Summit, October 24-25, Edinburgh

xendevsummit

The CfP for the Xen Project Developer Summit finished on Friday. I wanted to thank our Program Management Committee for putting in the effort to put together our program in record time. This was no easy task: we had nearly 50 extremely high quality submissions this year. Despite restricting talks to 30 minutes, we still had to reject many excellent submissions. We have 31 talks in the program, from 14 different organizations.

Program Highlights

This year’s Xen Project Developer Summit program is exceptional: it shows that the Xen Project is gathering momentum outside its traditional use-cases of server virtualization, client virtualization and cloud. We have talks and demos showing Xen running on mobile devices, showing Xen as virtualization platform for In-Vehicle Infotainment systems and as basis for network appliances. Just take a look at some of the talks scheduled:

  • HVM Dom0: Any unmodified OS as Dom0 Xiantao Zhang of Intel
  • “Unlimited” Event Channels by David Vrabel of Citrix Systems
  • Test-as-a-Service and XenRT by Alex Brett of Citrix Systems
  • XenGT: A software based Intel Graphics Virtualization Solution by Haitao Shan of Intel
  • Dual-Android on Nexus 10 by Lovene Bhatia of Samsung
  • Enabling Fast, Dynamic Network Processing with ClickOS by Joao Martins of NEC
  • Continued…

Posted in Events.

Tagged with .


Xen 4.1.6.1 released

I am pleased to announce the release of Xen 4.1.6.1. This is available immediately from its git repository xenbits.xen.org (tag RELEASE-4.1.6.1) or from the Xen Project download pages.

Note that 4.1.6 did not get released, as a build issue was found late in the release process, when the 4.1.6 version number was already irreversibly applied.

Xen 4.1.6.1 is a maintenance release in the 4.1 series and contains: We recommend that all users of Xen 4.1.5 upgrade to Xen 4.1.6.1.

This release fixes the following critical vulnerabilities:

  • CVE-2013-1918 / XSA-45: Several long latency operations are not preemptible
  • CVE-2013-1952 / XSA-49: VT-d interrupt remapping source validation flaw for bridges
  • CVE-2013-2076 / XSA-52: Information leak on XSAVE/XRSTOR capable AMD CPUs
  • CVE-2013-2077 / XSA-53: Hypervisor crash due to missing exception recovery on XRSTOR

Continued…

Posted in Announcements.

Tagged with , .


Xen 4.2.3 released

I am pleased to announce the release of Xen 4.2.3. This is available immediately from its git repository xenbits.xen.org (tag RELEASE-4.2.3) or from the Xen Project download pages.

This release fixes the following critical vulnerabilities:

  • CVE-2013-1918 / XSA-45: Several long latency operations are not preemptible
  • CVE-2013-1952 / XSA-49: VT-d interrupt remapping source validation flaw for bridges
  • CVE-2013-2076 / XSA-52: Information leak on XSAVE/XRSTOR capable AMD CPUs
  • CVE-2013-2077 / XSA-53: Hypervisor crash due to missing exception recovery on XRSTOR
    Continued…

Posted in Announcements.

Tagged with .