Operating systems are generally written assuming that they are in direct control of the hardware. So when we run operating systems in virtual machines, where they share the hardware with other operating systems, this can sometimes cause problems. One of the areas addressed by a recently proposed patch series is the problem of spinlocks on a virtualized system. So what exactly is the problem here, and how does the patch solve it? And what is the effect of the patch when the kernel is running natively?

Spinlocks and virtualization

Multiprocessor systems need to be able to coordinate access to important data, to make sure that two processors don’t attempt to modify things at the same time. The most basic way to do this is with a spinlock. Before accessing data, the code will attempt to grab the spinlock. If code running on another processor is holding the spinlock, the code on this processor will “spin” waiting for the lock to be free, at which point it will continue. Because those waiting for the spinlock are doing “busy-waiting”, code should try to hold the spinlock only for relatively short periods of time.

Now let’s consider how this looks on a virtualized system. In this case, the VM has virtual cpus (vcpu) which share the physical cpus with virtual cpus from other VMs. Typically the total number of virtual cpus across all VMs exceeds the number of physical CPUs; in some cases, such as cloud environments, there may be several times as many vcpus as physical cpus.

To accomplish this, the hypervisor scheduler gives timeslices of physical processor time to the vcpus, similar to the way that operating system schedules processes. If the system has more virtual cpus wanting to run than physical processors to run them, some of them will be preempted to let others run. This allows the VMs to share the physical cpu resources effectively, but it breaks a hidden assumption in the spinlock algorithm: that the kernel code is not preempted while holding a spinlock.

Now, suppose that vcpu A grabs a spinlock, and before it finishes, is preempted by the scheduler. And then suppose vcpu B tries to grab the spinlock. Now B, instead of spinning for the short period of time that A needed the spinlock for, will be spinning until vcpu is scheduled again — which may be anywhere from several milliseconds to hundreds of milliseconds, depending on how busy the system is. B is now using the cpu but accomplishing nothing. It’s burning up its VM’s share of CPU time, and keeping other VMs with useful work to do from running. Worse yet, it may even be that the reason A is not running is that the hypervisor’s scheduler is trying to give priority to B — so B is actively keeping A from finishing the work that it needed to do with the spinlock.

The situation gets even worse with a more advanced form of spinlock called a ticketlock. Ticketlocks have a lot of advantages for large systems, including reduced cacheline bouncing and more predictable wait time. (See this LWN article for a complete description.) The key attribute for this discussion is that ticketlocks essentially make a first-come first-served queue: if A has the lock, then B tries to grab it, and then C, B is guaranteed to get the lock before C. So now, if C is spinning waiting for the lock, it must wait for both A and B to finish before it can get the lock.

The result is that on a moderately loaded system, the vast majority of the cpu cycles are actually spent waiting for ticketlocks rather than doing any useful work. This is called a “ticketlock storm”. (It was documented for the first time by Thomas Friebel in his presentation at XenSummit 2008.)

Fixes to the vCPU starvation

In order to fix this starvation problem, in 2008, Jeremy Fitzhardinge developed a Linux kernel patch. His approach is to offer an intermediate layer for the spinlocks to allow paravirt backends to redefine the full serie of spinlock operations. Then he wrote a XEN-specific implementation which:

• For the fast case (uncontented) uses the traditional spinlock single-byte lock approach, overriding the ticketlock logic
• If a vCPU cannot get the lock after a specific amount of iterations (probabilly because of lock contention), it adds itself on a per-cpu list and blocks on an event channel
• On unlock the per-cpu list is walked and the first vCPU in line is awaken

In other words, after spinning for a certain amount of time, the code will assume that the vcpu holding the lock is not running; and instead of continuing to spin, will yield the cpu so that other work can get done.

This case doesn’t penalize native Linux while still giving all the wanted benefits. However, the paravirtualized spinlocks introduced a performance regression when using as native. So a specific kernel option, CONFIG_PARAVIRT_SPINLOCK, was introduced to include them. That way, distros could choose whether to take the additional spinlock indirection overhead when running a kernel natively (CONFIG_PARAVIRT_SPINLOCK=y), or the risk ticketlock storm when running a kernel as a VM (CONFIG_PARAVIRT_SPINLOCK=n).

While this approach has proven to work well, it has also shown some problems. First of all the indirection layer adds some overhead, even if not excessive, on a critical path. Second, this model imposes some code duplication.

In order to address these issues, Jeremy worked on a new approach based on existing ticketlock implementation. More specifically, the fastpath is left untouched and some PVOPs are added to the slow paths. These PVOPs are responsible for doing the following:

• The __ticket_lock_spinning() PVOP is used in the lock case and it is invoked if a CPU has been spinning for a certain, specific, threshold. Once invoked, __ticket_lock_spinning() marks the spinlock as in slowpath state and blocks the vCPU.
• The __ticket_unlock_kick() is invoked in the release slow path case and kicks the next vCPU in line awake. When the lock gets uncontented
  the slowpath bits gets cleared. Also, the bit signaling for the slow-path is stored in the lock tail ticket at the LSB, which means the number of CPUs ticketlocks can handle is reduced by a factor of 2, which is someway important on little tickets.

This allows paravirtualized ticketlock to share most of the code with native ticketlocks without the usage of any additive layer. It also removes the need for distros to choose between slower native performance and potentially catastrophic virtualized performance.

These patches have been heavilly tested in the past and recently they have been rebased to mainline Linux and further enhached by IBM engineers Srivatsa Vaddagiri and Raghavendra K T.

Benchmarks of the new approach

One of the key things the Linux maintainers care about when considering this kind of functionality is the effect it will have when running the kernel native. In order to support patch inclusion in mainline Linux, I (Attilio) wanted to show evidence that the patches were not introducing a performance regression in native configuration with real-world workloads. In order to reproduce real-world situations I used mainly 3 tools:

1. pgbench
2. pbzip2
3. kernbench

pgbench is a tool benchmarking PostgreSQL behaviour. It runs the same sequence of SQL commands repeteadly and calculates the average transactions per seconds. The sequence of commands can be customized, but by default pgbench uses five SELECT, UPDATE and INSERT for every transaction.

For my test I used a postgresql 9.2-devel version (mainline) as a backend, which has a lot of important scalability and performance improvements over the stable version, aiming for a larger output and more performing results. Also, I used a stock installation, with only a simple configuration change.

In order to have a full characterization of the scalability, I ran the workload with different sets of threads (ranging from 1 to 64), 10 times each, and used some warmup runs in order to load all the database in memory and thus avoid subsequent actual I/O operations when real measuration is taken. The script used for collecting datas is available here.

pbzip2 is a parallel implementation of the bzip2 file compressor utility using threads. I used this in order to emulate a CPU-intensive, multithreaded, application. The compressed file was a 1GB recipient created from /dev/urandom and all the I/O was performed on tmpfs in order to reduce floaty effects. Again, for a scalability characterization, the workload has been tried with several sets of threads and the used is here.

kernbench is a script comparing Linux kernel compile times, taking into account several number of make jobs. I went with the following invokation:

• kernbench -n10 -s -c16 -M -f

which basically means running the test for 10 times with 1 thread, 8 and 16 threads. Again, in order to reduce the I/O effect I used a tmpfs volume to do all the I/O.

The tests were performed on top of a vanilla Linux-3.3-rc6 (commit 4704fe65e55fb088fbcb1dc0b15ff7cc8bff3685), with a monholitic configuration. It was important also to estimate the impact of CONFIG_PARAVIRT_SPINLOCK option on this benchmark in order to eventually consider its removal.
The tests done, then, involved 4 different kernels which in turn had on and off Jeremy’s patch and CONFIG_PARAVIRT_SPINLOCK.

Below is information related to the system and machine used:

• Machine is a XEON x3450, 2.6GHz, 8-ways system
• System version, a Debian Squeeze 6.0.4
• gcc version 4.4.5

All the results have been chartered with ministat, a tool calculating fundamental statistical properties of data sets provided in files. They are summarized, divided by number of threads and kernel configurations, in the following links:

• http://xenbits.xen.org/people/attilio/jeremy-spinlock/pgbench-9.2-total.bench
• http://xenbits.xen.org/people/attilio/jeremy-spinlock/pbzip2-1.1.1-total.bench
• http://xenbits.xen.org/people/attilio/jeremy-spinlock/kernbench-0.50-total.bench

As you can easilly verify, ministat made an average calculation of the 10 retrieved values for every case, then compared the results and calculates difference and standard deviation for every compare.

The final result is that the patch doesn’t seem to introduce any performance penalty for these 3 workloads. Futhermore, it seems that the option CONFIG_PARAVIRT_SPINLOCK can be removed at the present time (although it likely needs to be re-evaluated with older CPUs than XEON x3450.)

XCP Benefits for Ubuntu Users:

• Ubuntu users can get started with Xen and stay up-to-date through the native Ubuntu package management system and update service
• Ubuntu users are now able to make use of enterprise virtualization functionality such as Host Pools, Storage Repositories, Performance Monitoring, Host Plugins, Guest VM Templates and Open vSwitch integration (for a detailed list see the XCP feature matrix).
• Ubuntu users can interact with Xen through the powerful Xen API (XAPI), with its host of integrations and tools.
• Ubuntu users can now customize XCP to their needs, whereas traditionally XCP appliances only utilized pre-defined configurations. For example users can now choose the latest versions of the Xen Hypervisor, Open vSwitch, Python other software in their Ubuntu Dom 0.

For more information see the XCP-API package and documentation. This is just the beginning: XCP in Ubuntu will evolve with the help of our user and developer communities. You can influence the direction of XCP in Ubuntu by attending the XCP design and Xen design session at Ubuntu Developer Summit – Q.

Supporting Quotes:

“The inclusion of XCP into Ubuntu lets users keep up to date with Xen with a single command.” said Paul Oh, Director of Business Development at Canonical. “The combination of Xen and Ubuntu offers a compelling combination of performance and security for the Cloud that Ubuntu users will find very compelling.”

“The collaboration between the Xen and Ubuntu communities was a great example of the power of open source. XCP-XAPI packages in Ubuntu will enable Ubuntu users to build Xen based clouds more easily.“ said Mark Heath, VP of Products for XenServer at Citrix. “As we continue to integrate and optimize Xen and XCP with Ubuntu, we look forward to meeting our users needs and helping to evolve better solutions for cloud.”

“I am pleased that project Kronos, which was started five months ago, has delivered XCP into Ubuntu Server 12.04 LTS. Until now, it was only possible to use XCP in Linux appliances within a tightly controlled environment. In Ubuntu Server 12.04 LTS we changed how users interact with XCP, providing much more flexibility and enabling anybody to use Ubuntu as a XCP Dom0 kernel.” said Lars Kurth, Community Manager, Xen.org. “I am also very excited about the possibilities this opens to build CloudStack, Eucalyptus, OpenNebula and OpenStack powered clouds on Ubuntu and XCP-XAPI.“

Further Information:

• Xen Documentation on Ubuntu
• XCP-API package on Ubuntu
• XCP-XAPI documentation on Ubuntu
• XCP-OpenStack documentation

tl;dr: If you previously had ‘dom0_mem=2G’ as a command line option to Xen, change this to ‘dom0_mem=2G,max:2G’.  If that didn’t help, read on.

What Changed?

When a domain is started, it is provided with:

1. A number of pages of memory. These are contiguous in the guest’s pseudo-physical address space and start at address 0.
2. The maximum number of pages the domain is allowed to have (the maximum reservation).  This may be more than the initial number of pages to allow the the guest to balloon up (increase its number of pages).
3. The e820 memory map.  For dom0, the physical e820 map is used so dom0 can access all BIOS data areas and PCI device MMIO regions.

The principle differences between the new and old kernels are:

1. The new kernel allocates page tables and struct page’s for all the pages up to the maximum reservation, the older kernels only allocate enough for the current number of pages.  On systems with lots of RAM relative to the number of pages used by dom0 this can result in lots of memory wasted.
2. The new kernel tries harder to release pages back to Xen that are unusable (because they are behind reserved regions or holes in the memory map).

Give Me Back My Memory!

To stop the kernel from wasting memory on page tables etc. for memory it will never use:

1. Use Xen dom0_mem=max:LLL command line option to set the maximum reservation to LLL (the M and G suffixes can be used for for MiB and GiB) .  e.g., If you previously had dom0_mem=XXXG, change this to dom0_mem=XXXG,max:XXXG.

This requires Xen 4.1.2 or later (which sets the maximum reservation based on the command line) and Linux 3.0.5 or later (which makes use of it)..

To recover the memory released during boot:

1. Ensure the balloon driver is enabled in the kernel (CONFIG_XEN_BALLOON).
2. Set the balloon driver’s target to the desired amount of RAM. Either with the mechanism provided by your toolstack or directly with (XXX is the amount of RAM in KiB):

   echo XXX > /sys/devices/system/xen_memory/xen_memory0/target_kb

I’m Lazy, Make it So

A series of patches by Konrad Rzeszutek Wilk will eliminate the need for the balloon driver.  With these patches, the otherwise unusable pages will be moved instead of released.  These patches are expected in 3.5.

Having to deal with a Non-Uniform Memory Access (NUMA) machine is becoming more and more common. This is true no matter whether you are part of an engineering research center with access to one of the first Intel SCC-based machines, a virtual machine hosting provider with a bunch of dual 2376 AMD Opteron pieces of iron in your server farm, or even just a Xen developer using a dual socket E5620 Xeon based test-box (any reference to the recent experiences of the author of this post is purely accidental ). Just very quickly,  NUMA means the memory accessing times of a program running on a CPU depends on the relative distance between that CPU and that memory. In fact, most of the NUMA systems are built in such a way that each processor has its local memory, on which it can operate very fast. On the other hand, getting and storing data from and on remote memory (that is, memory local to some other processor) is quite more complex and slow. Therefore, while hardware engineers bump their heads against cache coherency protocols and routing strategies for the system BUSes to be put on such machines, the most urgent issue for us, OS and hypervisor developers, is the following: how can we couple scheduling and memory management so that most of the accesses for most of our tasks/VMs stay local?

NUMA and Xen

The Xen hypervisor already deals with NUMA in a number of ways. For example, each domain has its own “node affinity”, which is the set of NUMA nodes of the host from which memory for that domain is allocated (in equal parts). That becomes very important as soon as many domains start running memory-intensive workloads on a shared host. In fact, as soon as the majority of the memory accesses become remote, the degradation in performance is likely to be noticeable. An effective technique to deal with this architecture in a virtualization environment is virtual CPU (vCPU) pinning. I mean, if a domain can only run on a subset of the host’s physical CPUs (pCPUs), it is very easy to turn all its memory accesses into local ones, isn’t it? Actually, that is exactly what Xen does by default: at domain creation time, it constructs the domain’s node affinity basing on what nodes these pCPUs belongs to… Provided the domain specifies some pinning for its vCPUs in its config file, with something like the below (assuming CPUs #0 to #3 belongs to the same NUMA node):

…
vcpus =  ’4′
memory =  ’1024′
cpus = “0-3″
…

This is quite effective, to the point that the (old) xend-based toolstack, if there is no vCPU pinning in a domain config file, tries to figure out on its own where to put it, and pin its vCPUs there! That could be seen as something reasonable to do, and surely brings performance benefits. However, vCPU pinning is quite unflexible, as the VM in question won’t for any reason be allowed to run outside from that set of pCPUs. This means, no matter whether it is the most (artificially-)intelligent of the toolstacks or the most careful of the sysadmins setting it up, you are in constant danger of under/bad utilizing your hardware resources you payed a lot of money for.

As of now, the new xl-based toolstack does not have anything like that.  Yes, there is the possibility of dealing with NUMA by partitioning the system using cpupools (available in the upcoming release of Xen, 4.2), as explained in Xen 4.2: cpupools. Again, this could be “The Right Answer” for many needs and occasions, but  has to to be carefully considered and manually setup by hand. What would be nice to have is a self-configuring solution automatically jumping into the game and maximizing the overall system performances.

Some numbers or, even better, some graphs!

Let’s give it a break to all this talking and try looking at what we are discussing more concretely. So question is: can we run some memory-intensive benchmark within a bunch of competing VMs (on a NUMA host) and see if this local/remote memory accessing thing really matters? Well, actually, yes we can. Unfortunately, no matter how bad you are hoping the benchmarker to be the research lab guy with login credentials for an SCC platform… He actually is the poor Xen developer with the 2 Socket Xeon. Plots showing what happened on a 2 nodes, 16 cores system is what we can show then. For now.

The elected benchmark was SpecJBB2005, under the assumption that it will generate quite a bit of stress on the memory subsystem, which turned out to be the case. Host was the 16 CPUs, 2-NUMA nodes, Xeon based system with 12GB RAM (2GB of which reserved for Dom0). Linux kernel for dom0 was 3.2, Xen was xen-unstable. Guests had 4 vCPUs and 1GB of RAM each. Numbers come from running the benchmark on an increasing (1 to 8 ) number of Xen PV-guests at the same time, and repeating each run 5 times for each of the VMs configurations below:

• default is the defaul Xen and xl behaviour without any vCPU pinning at all;
• pinned means VM#1 was pinned on NODE#0 after being created. This implies its memory was striped on both the nodes, but it can only run on the fist one;
• all memory local (best case) means VM#1 was created on NODE#0 and kept there. That implies all its memory accesses are local, and we thus call it the the best case;
• all memory remote (worst case) means VM#1 was created on NODE#0 and then moved (by explicitly pinning its sCPUs) on NODE#1. That implies all its memory accesses are remote, and we thus call it the worst case.

In all the experiments, it is only VM#1 that was pinned/moved. All the other VMs have their memory “striped” between the two nodes and are free to run everywhere. The final score achieved by SpecJBB on VM#1 is reported below. As SpecJBB output is in terms of “business transactions per second (bops)”, higher values correspond to better results.

First of all, notice how small the standard deviation is for all the runs: this just confirms SpecJBB is a good benchmark for our purposes. The most interesting lines to look at and compare are the red and the blue ones. Evidence is there that things can improve quite a bit, even on such a small box, especially in presence of heavy load (6 to 8 VMs). Let’s also look at the percent increase in performance of each run with respect to the worst case (all memory remote):

This second graph makes even more clear how NUMA placement and scheduling is accountable for a ~10% to 20% (depending on the load) impact on performance. The default Xen behavior is certainly not as bad as it could be: default almost always manage in getting ~10% better performance than the worst case. Also, although pinning can help in keeping performance consistent, it doesn’t always yield an improvement (and when it does, it is only by few percent points). There is a ~10% performance increase to gain (and even more, in heavy loaded cases), if we manage in getting default to be close enough to all memory local, and that should be the way to go!
The full set of results, with plots about all the statistical properties of the data can be found here.

Are we on the case then?

We sure are! Preliminary patches have been posted to the xen-devel mailing list, and the results of some (preliminary as well, of course) benchmarks are available on this Wiki article. A new blog post expanding on their aims, features, and performances will follow. In the meanwhile, should you feel like wanting to help with testing, benchmarking and fixing things, please, jump in!

And the moral of the story is…

Yes, my dear NUMA pieces of hardware out there, the Xen.org community is staring at you right in the eyes, and we will get the best out of you, no matter how hard it will be… “Lower your shields and surrender. Resistance is futile“!

Xen @ Ubuntu Developer Summit, May 7-11, Oakland, CA

The Xen and XCP teams will be participating in the Ubuntu Developer Summit in Oakland, CA. The agenda for technical discussions is not yet quite tied down: when it is we will let you know.

Xen @ Build a Cloud Day, May 10, San Francisco, CA

Learn how to build an open source cloud cloud with CloudStack, RightScale, Ubuntu, Xen and Zenoss at the free Build a Cloud Day. Build a Cloud Days are about learning, best practices and industry insights into building elastic, scalable and profitable open source clouds. The event is held in conjunction with Citrix Synergy 2012 in San Francisco and as a bonus you will get a free pass to the Synergy Cloud Keynotes and Solutions Expo.

Xen Day Fortaleza 2012, May 30, Brazil

Xen Day Fortaleza is a community organized evening of lectures focussing on virtualization with Xen and cloud management with Cloudstack. The event will be held on May 30, 2012 in Fortaleza, CE, Brasil. It is a free event: register here.

Xen Summit, August 27-28, San Diego, CA

Program Management Committee and CFP

First I wanted to welcome the Xen Summit Program Committee, which this year will be: David Nalley (Cloudstack.org), Donald D Dugger (Intel), Ian Campbell (Citrix), Konrad Rzeszutek Wilk (Oracle) and Lars Kurth (Xen.org).

I also wantet to let you know that the community asked me to extended the deadline for the Call for Participation: the deadline now is midnight June 15, 2012 PDT. More information at the XenSummit event page.

Xen Dev Day

You should also note that we are contemplating to hold a Kernel Summit like Developer event called Xen Day alongside Xen Summit. The event would be invite. A community vote is open until Friday, Apr 27 regarding time and date preferences.

Hotel Bookings

It is now possible to book the hotel for Xen Summit. Several events will be held in the same hotel in the same week: Xen Summit, LinuxCon, Linux Plumbers Conference, Linux Kernel Summit and CloudOpen. If you plan to attend Xen Summit and any of the other events, it will be possible for you to book your hotel for the entire week for USD$170/night through any of the event sites. Note that rooms must be booked before 17:00 PST, July 27, 2012 and that there are only limited rooms available, which might fill up fast.

The core idea behind cpupools is to divide the physical cores on the machine into different pools. Each of these pools has an entirely separate cpu scheduler, and can be set with different scheduling parameters. At any time, a given logical cpu can be assigned to only one of these pools (or none). A VM is assigned to one pool at a time, but can be moved from pool to pool.

There are a number of things one can do with this functionality. Suppose you are a hosting or cloud provider, and you have a number of customers who have multiple VMs with you. Instead of selling based on CPU metering, you want to sell access to a fixed number of cpus for all of their VMs: e.g. a customer with 6 single-vcpu VMs might buy 2 cores worth of computing space which all of the VMs share.

You could solve this problem by using cpu masks to pin all of the customer’s vcpus to a single set of cores. However, cpu masks do not work well with the scheduler’s weight algorithm — the customer wont’ be able to specify that VM A should get twice the cpu as VM B. Solving the weight issue in a general way is very difficult, since VMs can have any combination of overlapping cpu masks. Furthermore, this extra complication would be there for all users of the credit algorithm, regardless of whether they use this particular mode or not.

With cpu pools, you simply create a pool for each customer, assign it the number of cpus that customer is paying for, and then put all of that customer’s VMs in the pool. That pool has its own complete cpu scheduler; and as far as that pool’s scheduler is concerned, the only cpus in existence are the one inside the pool. This means all of the algorithms regarding weight and so on work exactly the same, just on a restricted set of cpus.

Additionally, this means that each customer can request different scheduling parameters for their VMs (for example, the timeslice or ratelimit parameters we talked about last week), or even completely different schedulers, including the experimental credit2 scheduler, and the real-time SEDF scheduler.

Cpupools have the potential to increase security as well: they limit the interaction between different customers to physically separate cpus. Sometimes information about cryptographic keys can be pieced together just by knowing cache patterns or the amount of time spent on certain operations; having VMs from different customers run on phsyically separate cpus removes this vector of attack with very little effort.

Of course, all of the above can be useful even if you’re not a cloud provider: your realtime workloads can run in a pool with the SEDF scheduler, your latency sensitive workloads can run in a pool with a short timeslice, and your number-crunching workloads can run in a pool with a really long timeslice.

One of the particulary convenient commands that Jürgen implemented is the cpupool-numa-split command. This command will automatically detect the NUMA topology of the box you’re on, create a single pool for each NUMA node, and put all of the cpus in the corresponding pool. Then when you create VMs, you specify the pool you wish them created in, and all of the memory allocated will be local NUMA accesses.

The details of the interface for cpupools is still undergoing some cleaning up in the last few weeks before the 4.2 release, so I don’t want to go into details. There will be an introduction with examples on the Xen.org wiki page before the release, as well as documentation in the man pages and in the command-line help.

For a list of items that need work, check out the community maintained TODO and wishlist. Of course, you can work on anything you like: the list just provides suggestions.

How do I participate?

• Join us on IRC: freenode channel #xendocday
• Tell people what you intend to work on (to avoid doing something somebody else is already working on)
• Fix some documentation
• Help others
• And above all: have fun!

See you on IRC!

• Re-engineered how tools can perform hypercalls – by using a standard interface (/dev/xen/privcmd instead of using /proc/xen/privcmd)
• Backends (netback, blkback) can now function in HVM mode. This means that a device driver domain can be in charge of a device (say network) and a subset of the network (netback). What is exciting about this it allows for security by isolation – so if one domain is compromised it does not affect the other domains. Both Qubes and NSA Research Center have been focusing on this functionality and it is exciting to see components of this goal taking shape!
• My pet peeve: graphics not working. The Xen architecture makes an interesting sandbox for making drivers compatible with other platforms. This is due to the fact that the when device drivers DMA data, the “bus” address (so the address used on PCI chipsets) is different from the “physical” (the address used by the CPU) address. This mapping is common on other platforms besides x86 – SPARC, or PPC. The x86 architecture usually has a 1:1 mapping (so “phys” == “bus”) – and both nouveau and radeon drivers weren’t using th full gamma of the PCI API to take advantage of that. The TTM DMA pool driver does that and now PCI (example: ATI ES1000) and PCIe cards can function properly under Xen – the remaining piece is to hookup this to the AGP API, but it not clear if there is that much huge demand for that.
• Thanks for the XenDoc day a lot of documentation was added to the kernel and the XenBus code was cleaned up.

Also some pretty serious bug fixes were put in:

• A fix in the spinlock code: if you build the kernel for fewer than 256 CPUs corruption of the spinlock occurs and a crash will ensure.
• We fixed a long-uptime bug when using the radeon or nouveau driver. The user would see: “ WARNING: at arch/x86/xen/mmu.c:475 xen_make_pte” and things started to mysteriously break.
• Hardening of the XenBus code to deal with bad frontends.

That is it for now. The Linux 3.4 merge window has opened. Expect to see another blog post from in the near future, with detail on what will go in Linux 3.4. The good news is that some of these items have been requested by many of you for some time!

A thank you to the contributors of this release

Annie Li (7):

• Annie Li (7):
• xen/granttable: Introducing grant table V2 stucture
• xen/granttable: Refactor some code
• xen/granttable: Grant tables V2 implementation
• xen/granttable: Keep code format clean
• xen/granttable: Improve comments for function pointers
• xen/granttable: Support sub-page grants
• xen/granttable: Support transitive grants

Bastian Blank (5):

• xen: Add privcmd device driver
• xen: Add xenbus device driver
• xen: Add xenbus_backend device
• xen/privcmd: Remove unused support for arch specific privcmp mmap
• xen/xenbus-frontend: Make error message more clear

Daniel De Graaf (10):

• xen/gntalloc: Change gref_lock to a mutex
• xen/gnt{dev,alloc}: reserve event channels for notify
• xen/event: Add reference counting to event channels
• xen/events: prevent calling evtchn_get on invalid channels
• xen/gntalloc: release grant references on page free
• xen/gntalloc: fix reference counts on multi-page mappings
• xenbus: Support HVM backends
• xenbus: Use grant-table wrapper functions
• xen/grant-table: Support mappings required by blkback
• xen/netback: Enable netback on HVM guests

David Vrabel (3):

• xen: document balloon driver sysfs files
• xen: document backend sysfs files
• x86: xen: size struct xen_spinlock to always fit in arch_spinlock_t

Ian Campbell (3):

• xen/xenbus: Reject replies with payload > XENSTORE_PAYLOAD_MAX.
• xenbus: maximum buffer size is XENSTORE_PAYLOAD_MAX
• xen/xenbus: don’t reimplement kvasprintf via a fixed size buffer

Jan Beulich (2):

• Xen: consolidate and simplify struct xenbus_driver instantiation
• xenbus_dev: add missing error check to watch handling

Jeremy Fitzhardinge (1):

• Xen: update MAINTAINER info

Jerome Glisse (8):

• drm/ttm: remove userspace backed ttm object support
• drm/ttm: remove split btw highmen and lowmem page
• drm/ttm: remove unused backend flags field
• drm/ttm: use ttm put pages function to properly restore cache attribute
• drm/ttm: test for dma_address array allocation failure
• drm/ttm: merge ttm_backend and ttm_tt V5
• drm/ttm: introduce callback for ttm_tt populate & unpopulate V4
• ttm: fix agp since ttm tt rework

Julia Lawall (1):

• xen-gntalloc: introduce missing kfree

Kay Sievers (1):

• xen-balloon: convert sysdev_class to a regular subsystem

Konrad Rzeszutek Wilk (24):

• xen/blk[front|back]: Squash blkif_request_rw and blkif_request_discard together
• xen/blk[front|back]: Enhance discard support with secure erasing support.
• xen/blkback: Move processing of BLKIF_OP_DISCARD from dispatch_rw_block_io
• swiotlb: Expose swiotlb_nr_tlb function to modules
• drm/ttm: provide dma aware ttm page pool code V9
• drm/radeon/kms: enable the ttm dma pool if swiotlb is on V4
• drm/nouveau: enable the ttm dma pool when swiotlb is active V3
• xen/xenbus-frontend: Fix compile error with randconfig
• xen/xenbus: Fix compile error – missing header for xen_initial_domain()
• drm/ttm/dma: Only call set_pages_array_wb when the page is not in WB pool.
• drm/ttm/dma: Fix accounting error when calling ttm_mem_global_free_page and don’t try to free freed pages.
• x86/PCI: Expand the x86_msi_ops to have a restore MSIs.
• xen/pciback: Move the PCI_DEV_FLAGS_ASSIGNED ops to the “[un|]bind”
• xen/pciback: Fix “device has been assigned to X domain!” warning
• xen/pciback: Expand the warning message to include domain id.
• xen/mmu: Fix compile errors introduced by x86/memblock mismerge.
• xen/balloon: Move the registration from device to subsystem.
• ttm/dma: Remove the WARN() which is not useful.
• xen/granttable: Disable grant v2 for HVM domains.
• xen/bootup: During bootup suppress XENBUS: Unable to read cpu state
• xen/smp: Fix CPU online/offline bug triggering a BUG: scheduling while atomic.
• xen/pci[front|back]: Use %d instead of %1x for displaying PCI devfn.
• xen/setup: Remove redundant filtering of PTE masks.
• xen/pat: Disable PAT support for now.

Li Dongyang (1):

• xen-blkback: convert hole punching to discard request on loop devices

Maxim Uvarov (1):

• xen: Make XEN_MAX_DOMAIN_MEMORY have more sensible defaults

Stefano Stabellini (1):

• xen pvhvm: do not remap pirqs onto evtchns if !xen_have_vector_callback

Tejun Heo (1):

• memblock: Fix alloc failure due to dumb underflow protection in memblock_find_in_range_node()

Thomas Meyer (1):

• xen-blkfront: Use kcalloc instead of kzalloc to allocate array

Tony Luck (1):

• xen/ia64: fix build breakage because of conflicting u64 guest handles

Note that there is a roadmap session on XenAPI support in at the OpenStack Design Summit later today: XenAPI (XenServer/XCP) Folsom Roadmap at 2pm, PST. If you are at the Design Summit and care about Xen support, why not drop in and meet Ewan Mellor, John Garbutt or Renuka Apte.

Of course, project Kronos is almost completed, which will help Xen support in OpenStack. You can find information about XCP-XAPI in Debian in the Debian package repository (for docs see Debian XCP wiki and this tutorial). XCP-XAPI support in Ubuntu is near complete: we are waiting for the XCP-XAPI packages to be synced to Ubuntu (see ticket #962184). Documentation can be found in the manpages and the XCP-XAPI package description.

Information about the virtualization test day can be found here. We will have some people hanging out on IRC at #fedora-test-day and you can also get in touch with us via the usual channels.

We have some Xen-specific information here.