Skip to content


Will you give Xen a ride … or will Xen give you a ride?

And by “a ride”, we actually mean a ride. Like this:

 

8275748195_4a18513755_z

Like, will Xen run in your car?  Well, it appears it will!

It all started with ARM Support

In fact, Xen Project developers started woking on supporting the ARM architecture (with hardware virtualization capabilities) a couple of years ago. The goal was simple: as soon as ARM server are available, it must be possible to run Xen Project software on them. That goal has been achieved, but that is another story!

It is well known that processors employing the ARM architecture are powering already the vast majority of the so called Embedded Systems, ranging from phones, tablets and smart TVs up to cars or even airplanes. But does that mean that at some point we will start to see virtualization capable chips in cars? And if yes, when? The answers to these questions are “Yes” and “really really really soon”! In fact, the Xen Project Hypervisor is uniquely placed to support this new range of use-cases. Its isolation and security features, flexible virtualization mode and architecture, not to mention driver disaggregation and the fact that it now supports ARM (and does it with only ~90K lines of code), make it a perfect fit for the embedded world.

Some Recent ‘History’

Mobile and embedded virtualization on ARM has a long history within the Xen Project, with research projects such as Samsung’s ARM PV port and the Embedded Xen effort. However these projects were mainly research focused. With ARM support becoming a part of the Xen Project Hypervisor last year and various market factors coming together, Xen Project based products are now on the horizon. Last autumn was pivotal in generating momentum for this concept. A number of companies showed real demos and prototypes at our 2013 Developer Summit, such as

  • The Xen Project Hypervisor running on a Nexus 10 (slides and video)
  • The Xen Project Hypervisor powering an in-vehicle infotainment (IVI) system, and other systems on the TI Jacinto 6 automotive platform designed for cars (slides and video).

Since then, momentum has built within the community – as can be seen on xen-devel mailing list discussions – to port embedded OSes to the Xen Hypervisor  (some examples: FreeRTOS, Erika and QNX). Contributions and patches for making The Xen Project Hypervisor work better in such environments started to arrive too, from individuals, research institutions and small and big companies. Among the companies, GlobalLogic Inc., a full-lifecycle product development services company, has made the largest contribution so far, but we must also mention DornerWorks, GaloisUniversity of Washington and Evidence (in collaboration with the University of Modena).

A summary of the past and ongoing activities of this kind is below:

What about now?

On Monday (we told you: “really really really soon” :-D ), The Xen Collaborative Project and The Linux Foundation announced a new Embedded and Automotive initiative. Artem Mygaiev, AVP Development at GlobalLogic, will serve as the Embedded and Automotive Project Lead.

The Embedded and Automotive team within The Xen Project intends to build a platform around the Xen Hypervisor that enables using it for all the non-data center use cases (automotive, internet TV, mobile, etc.) by providing a community focal-point within the Xen Project community as well as within the wider open source community.

The team plans to:

  • develop and upstream necessary changes to The Xen Project Hypervisor and Linux
  • implement new drivers (such as GPU, HID, …), protocols, capabilities and functionality that are needed for a complete automotive/embedded/mobile virtualization stack
  • upstream all necessary changes to support such functionality in operating systems that are needed for these use-cases (e.g. Android, Linux, etc.)

For the occasion, Alex Agizim, CTO of Embedded Systems at GlobalLogic, which also is a member of The Linux Foundation Automotive Grade Linux Steering Committee, said:

With ARM support, Xen Project technology is a perfect fit for embedded systems and automotive use. For example, our Nautilus platform, based on The Xen Project virtualization, enables ourin-vehicle infotainment (IVI) and auto manufacturing partners to quickly and cost-effectively develop hybrid Android/Linux-based systems. Using Nautilus, developers are able to run multiple sandboxed OSes on a single System-on-Chip (SOC). This provides superior functionality and security for both infotainment and operational functions within a car.

The latest demo of GlobalLogic‘s Nautilus Platform has been shown at the latest edition of the Automotive Linux Summit, in Tokyo. Check out the video and slides. We also heard about further use cases for Xen Project Software at this week’s Developer Summit. The rate of innovation in our community in this area is staggering: fasten your seat belts! We will tell you about these more in an upcoming event report. All this activity is also creating many benefits for the cloud and traditional server use use-cases. Certification will lead to quality improvements across shared components. Realtime scheduling can be used for graphics and gaming use-cases in the cloud and for Network Function Virtualization. And so on, and so on, …

Learn More

GlobalLogic, in partnership with The Linux Foundation, will present a free webinar at 9 a.m. PDT, Wednesday, August 27, 2014, titled “Virtualization in the Automotive Industry.” Register today to learn how Xen Project technology adds reliability and security when adopting virtualization for automotive software development.

Vendors and individual developers interested in collaborating on embedded, automotive and mobile use cases are encouraged to join the new Xen Project subproject at http://xenproject.org/developers/teams/embedded-and-automotive.html.

Posted in Announcements, Community, Incubated Project, Linux, Xen Case Study, Xen Development, Xen Hypervisor, Xen Summit, Xen Training.

Tagged with , , , , , , , .


Time to Register for Xen Project User Summit on Sept 15, 2014

Great Session Lineup Awaits Attendees in New York City!

xpus-join_us-2014

It’s time to make your travel plans to New York City for the September 15 arrival of the Xen Project User Summit!

The Lighthouse Executive Conference Center will play host to the only full day user-centric Xen Project event on this year’s calendar.  If you are a power user, a new user, or you just want to see what Xen Project can possibly do for you, you’ll want to be there.

Attendees will find an excellent selection of talks waiting for them at this year’s event.

Is Your Head in the Clouds?

We have a number of terrific cloud-related talks on the schedule!

Listen to SUSE‘s Peter Linnell talk about Xen, OpenStack, and the SUSE Cloud.  And are you thinking about unikernels in the cloud?  Adam Wick of Galois, provider of HaLVM, will expound on the virtues of tiny VMs providing a single-purpose operating environment (imagine many of the advantages of Linux Containers but with the security of a real hypervisor).  And Don Marti from Cloudius Systems will talk about using the OSv unikernel to create a C and Java environment directly on the hypervisor.

Or is Security Your Focus?

Then you’ll want to hear about the Zazen security framework as described by Steven Maresca of Zentific.  Plus, there’s my talk discussing the Advanced Security Features of Xen Project, most of which are easy to use, but most users don’t even seem aware of them.

Is Your Mind on the Future?

That’s good, because Oracle‘s Konrad Wilk, the Release Manager for the upcoming 4.5 Xen Project Release, will give us the run down of the features to expect.  There’s an impressive list of improvements in the works, so Konrad will help us to understand what is likely to make the cut.  And our own Community Manager Lars Kurth from Citrix will give us a healthcheck on the project itself, including fascinating information about how diverse our development ecosystem has become in the past few years.

Or Maybe Distribution Support is Your Thrust?

You’ll want to hear Johnny Hughes of the CentOS Project discuss the Xen4CentOS effort with a quick and easy cookbook for restoring our favorite hypervisor to the CentOS world.  He will also address how the effort has given birth to the new CentOS Virtualization SIG with an eye on making CentOS a key platform for all types of virtualization technologies.

Could XenServer be Your Concern?

If so, then you will want to hear XenServer Community Manager Tim Mackey discuss where XenServer has gone in the year since it became Open Source — and, more importantly, where it is expected to go from here.  And you will definitely want to hear from Olivier Lambert of Vates as he discusses the Xen Orchestra project, an interface for controlling XenServer and XAPI installations from a web-based interface.

Or High Availability?

Then you’ll definitely want to hear Will Auld of Intel discuss the COLO project.  Still in development, COLO aims to create lock-step VMs to deliver non-stop availability during a VM failure.  This is critical capability when you have a VM which needs to keep running no matter what.

Or Deployment?

Don’t miss Grant McWilliams‘ talk about deploying XAPI-based security devices.  It is actually an amazing tale describing the design and deployment of some amazing devices using the Xen Project Hypervisor to make magic happen.

Then You Need to Register Today!

For just $79 — less than the price of a good dinner in Manhattan — you can get all this, plus the User Jam Session.  User Jam is your time to give feedback about the project and the presentations given.

And tell a coworker to sign up, too!  We have two tracks most of the day, so bring a friend who can go to the sessions you can’t.

See the full schedule and register on the Linux Foundation website:

And we hope to see you in New York City on September 15!

Posted in Announcements, Community, Events, Xen Summit.


Customers Call the Shots — Verizon Cloud Adds Business Value with Quality of Service

dslutz

Both businesses and consumers rely on public clouds for a range of tasks and activities from collaboration and video streaming to gmail and Netflix. New companies are born with just a dozen employees, a laptop and an Internet connection practically overnight. This is all thanks to cloud computing.

It’s no surprise that in the next six years, almost 90 percent of new spending on Internet and communications technologies, a $5 trillion global business, will be on cloud-based technology, according to industry analyst firm IDC. Cloud applications will also account for 90 percent of total mobile data traffic by 2018, according to the Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2013–2018.

The benefits for users are almost too numerous to count, but most IT professionals agree that cloud computing epitomizes constant change. Its ability to provide ubiquitous, on-demand access to a shared pool of networks, servers, storage, and services whenever and wherever they are needed is creating both market opportunity and market upheaval.

To temper the turbulence, capitalize on the opportunities and best prepare for any number of cloud unknowns, several of the world’s largest public providers including Amazon Web Services, Rackspace, IBM/SoftLayer and Verizon Terremark rely on Xen Project virtualization. Open source Xen Project software offers superior IT efficiencies, workload balancing, hyperscalability and tight security by running VMs on a cloud service.

While today the media is focusing on price wars and the possible commoditization of infrastructure as a service (IaaS), cloud providers like Verizon Terremark are innovating with novel Quality of Service agreements and new levels of automation. In his talk in Chicago at our Xen Project Developer Summit, Verizon Terremark’s Don Slutz will present an overview of the Verizon Cloud architecture based on Xen.

“It’s the core foundation of the Verizon Cloud, allowing our users to run any type or size workload they’d like to. Xen is critical to Verizon. Competing solutions were either too cost prohibitive or lacked the security controls that Xen has,” Don said.

Verizon Terremark is a long-time advocate of open standards and is more actively involved than ever before in the open source ecosystem. Verizon sponsors and participates in Xen Project software, invests in CloudStack and most recently joined the Cloud Foundry Foundation, hoping to see the cloud market mature quickly and provide businesses with cloud-based offerings that address specific needs like performance, choice, cost and flexibility.

For the past three years, Don has worked on integrating and designing Xen for the Verizon Cloud architecture along with seven full-time engineers. Today, clients are fully deployed on Verizon’s IaaS based on Xen. A focal point of his talk will be Verizon’s Quality of Service (QoS) goals with CPU, memory, network and disk performance.

“Often clouds end up requiring far too much support personnel, which we are trying to rectify. With our QoS agreement, we allow users to set the performance parameters their business requires and guarantee that Verizon will back these up at all times. Instead of focusing on speed or load size, we’ll guarantee certain CPU, memory, network or disk performance. This is really unique in the industry,” he added.

In addition to delivering workload efficiency, security and cost savings to its cloud customers, Verizon is also giving back to the Xen Project community.

“We’re working to make Verizon Cloud a high capacity service that allows people to move existing VMs easily onto it it,” Don said. “Our goal is to add enough VMWare support so that a guest can be exported from VMWare and automatically run without any changes on Xen.”

Verizon’s VMWare code is currently in review and in the past year has contributed 40 change sets that totals 4,300 lines of code.

Proof that demand for cloud services is growing and spurring more change, Don will also address Verizon’s design goals to move from three to seven data centers in the near future. If you’re interested in learning more, be sure to register today for the Xen Project Developer Summit to hear Don present on Tuesday, August 19 from 9 to 9:45 a.m.

About Don Slutz
Currently, Don works for Verizon Terremark enhancing Xen, which is the basis for Verizon Cloud. He got started early (1970) in computers because of his father Dr. Ralph J. Slutz and spent 16 years at Prime Computer in operating systems. He has extensive networking, performance, and testing experience.

Posted in Events, Xen Case Study.

Tagged with , , , , .


2 weeks to Xen Project Developer Summit – Chicago 18-19, IL, USA

Only last week, the Xen Project team was at OSCON where we launched Mirage OS 2.0 (event report to follow soon, but in the meantime check out the following sessions Nymote and Mirage, Floss Weekly on Mirage OS and Community War Stories) and now our Developer Summit is just round the corner. As we have seen tremendous community growth in the last 12 months (>30%) and the most feature reach Xen Project Hypervisor release coming up soon, I thought I’d share what you can expect.

xpds14
(click image to go to event website)

What to expect?

Xen Project Developer Summits are packed with highly technical content where the core developers of the Xen Project community come together to discuss the evolution of the Xen Project. The conference is a mixture of talks and interactive sessions in un-conference format (which we call BoFs). Newcomers and those who are interested in the progress and future of the Xen Project, it’s sub projects (Hypervisor on ARM and x86, Upstreams and Downstreams, Embedded and Automotive variants, Cloud Operating Systems such as Mirage OS) usually will get tremendous value from attending the event.

Besides roadmap, feature updates and developer topics, this year features a few themes:

  • Network Function Virtualization
  • Security
  • Performance and Scalability
  • Cloud Operating Systems
  • Topics that are important for automotive/embedded/mobile use-cases, such as Real-time virtualization, certification and ARM support

Why not check out the agenda or watch last year’s sessions to get a sense of what is coming. Note that BoF’s and discussion groups will be published next week.

How to get the most out of the Summit?

Our developer events are designed to help you make connections and to participate. A good way to network are our evening social event and to network during the breaks. Another great way to get the most out of the summit is to submit a BoF/discussion groups about a topic you care about or to participate in a BoF/discussion group. BoF submissions are open until August 11 and the BoF schedule will be published the week before the event. Most of our talks will have an extensive and interactive Q&A portion, which is another way to engage.

Posted in Announcements, Events.

Tagged with .


Mirage OS v2.0: The new features

The first release of Mirage OS back in December 2013 introduced the prototype of the unikernel concept, which realised the promise of a safe, flexible mechanism to build highly optimized software stacks purpose-built for deployment in the public cloud (see the overview of Mirage OS for some background). Since then, we’ve been hard at work using and extending Mirage for real projects and the community has been steadily growing.

Today, we’re thrilled to announce the release of Mirage OS v2.0! Over the past few weeks the team has been hard at work writing about all the new features in this latest release, which I’ve been busy co-ordinating. Below are summaries of those features and links to in-depth blog posts where you can learn more:

Thomas Leonard's Cubieboard2

Thomas Leonard’s Cubieboard2

ARM device support: While the first version of Mirage was specialised towards conventional x86 clouds, the code generation and boot libraries have now been made portable enough to operate on low-power embedded ARM devices such as the Cubieboard 2. This is a key part of our efforts to build a safe, unified multiscale programming model for both cloud and mobile workloads as part of the Nymote project. We also upstreamed the changes required to the Xen Project so that other unikernel efforts like HalVM or ClickOS can benefit.

Irmin – distributed, branchable storage: Unikernels usually execute in a distributed, disconnection-prone environment (particularly with the new mobile ARM support). We therefore built the Irmin library to explicitly make synchronization easier via a Git-like persistence model that can be used to build and easily trace the operation of distributed applications across all of these diverse environments.

OCaml TLS: The philosophy of Mirage is to construct the entire operating system in a safe programming style, from the device drivers up. This continues in this release with a comprehensive OCaml implementation of Transport Layer Security, the most widely deployed end-to-end encryption protocol on the Internet (and one that is very prone to bad security holes). The series of posts is written by Hannes Mehnert and David Kaloper.

Modularity and communication: Mirage is built on the concept of a library operating system, and this release provides many new libraries to flexibly extend applications with new functionality.

  • Fitting the modular Mirage TCP/IP stack together” by Mindy Preston explains the rather unique modular architecture of our TCP/IP stack that lets you swap between the conventional Unix sockets API, or a complete implementation of TCP/IP in pure OCaml.
  • Vchan: low-latency inter-VM communication channels” by Jon Ludlam shows how unikernels can communicate efficiently with each other to form distributed clusters on a multicore Xen host, by establishing shared memory rings with each other.
  • Modular foreign function bindings” by Jeremy Yallop continues the march towards abstraction by expaining how to interface safely with code written in C, without having to write any unsafe C bindings! This forms the basis for allowing Xen unikernels to communicate with existing libraries that they may want to keep at arm’s length for security reasons.

All the libraries required for these new features are regularly released into the OPAM package manager, so just follow the installation instructions to give them a spin. A release this size probably introduces minor hiccups that may cause build failures, so we very much encourage bug reports on our issue tracker or questions to our mailing lists. Don’t be shy: no question is too basic, and we’d love to hear of any weird and wacky uses you put this new release to! And finally, the lifeblood of Mirage is about sharing and publishing libraries that add new functionality to the framework, so do get involved and open-source your own efforts.

Posted in Announcements, Incubated Project.

Tagged with , , , , , , .


XCP Wiki Pages Have Been Migrated to Wiki.XenServer.org

XCP is Dead; Long Live XenServer!

Last year was a momentous year for all things related to the Xen Project.  2013 saw Xen Project become a Linux Foundation Collaborative Project.  It also saw the separate XenServer project become Open Source under the auspices of Citrix Systems.  And, as a result of that, the need for XCP — the free (as in “free beer”) version of XenServer — dissolved.  Since XenServer is now Open Source and available to all, XCP is superfluous.

Now, historically, the Xen Project Wiki has hosted the XCP pages.  But given the major shifts of the past year, coupled with the fact that XCP is not part of Xen Project but XenServer, the hosting situation has now changed.  From now on, the archive of XCP pages will reside on the XenServer Wiki.

Where Can I Find the XCP Pages?

You can find the XCP archive on the XenServer Wiki under the XCP category:

https://wiki.xenserver.org/index.php?title=Category:XCP

Why Did We Need to Migrate the Pages at All?  Why Not Just Leave Them As Is?

As part of the migration of Xen Project to the Linux Foundation, it is necessary to clearly delineate which things belong to the Xen Project as opposed to things which belong to other parties.  XCP is not part of Xen Project and it does not belong to Linux Foundation.  Therefore, it should not reside on Xen Project websites.  It makes perfect sense for XCP to be hosted by XenServer.org, since it always has been part of the XenServer effort.

Why Migrate Pages at All?  If XCP is Obsolete, Why Not Just Delete the Pages?

Just because XCP is obsolete, it doesn’t mean it is not used.  There are still many, many XCP users in the world, and they deserve access to documentation.  Eventually, they may become XenServer users and when that happens, the XenServer project can determine what to do with their Wiki pages.  But that is not the case now, so a transition is necessary.

So What Happens to All the External Sites Which Link to XCP Pages on Wiki.XenProject.org?

The original XCP pages are in process of being replaced with links to the new pages on Wiki.XenServer.org.  Most links are already in place, and the few stragglers should be handled in the next few weeks.  There is a temporary archive of the XCP pages on Wiki.XenProject.org which will be removed at the end of the process once all the pages have been linked and verified.

Why Do I Still See XCP Downloads on www.XenProject.org?

We are coordinating with the XenServer.org folks to make sure that the XCP downloads remain available during the time of transition.  Once the XCP downloads are available from XenServer.org, the downloads will disappear from www.XenProject.org.

What Happens If I Discover an XCP Wiki Page Which Does Not Exist on XenServer.org?

Contact me at russell (dot) pavlicek (at) XenProject.org and send me the link.  I will coordinate with the XenServer.org team to see that the page is migrated.

Posted in Announcements, XCP.

Tagged with , .


Mark Your Calendars! Great Xen Project Events Coming in August and September 2014

If you use — or are just interested in learning about — the Xen Project Hypervisor, you will want to mark your calendar now for two great events coming later this summer.

September 15: Xen Project User Summit, New York City, NY

The Xen Project User Summit is great for:

  • Users who employ Xen Project software in their datacenter or cloud
  • Integrators who leverage the Xen Project Hypervisor in their solution or service
  • Architects who need to know how to best utilize the software in their solution architecture
  • Managers who want to understand what current and future features will enable their goals

At just US$79 for one day of sessions, the User Summit is an unmatched training opportunity for users of Xen Project software. Located at the Lighthouse Executive Conference Center in the heart of New York City, this event is a fantastic way to increase your knowledge of all things relating to Xen Project.  The lineup of talks will be unlike any other event this year, so it makes sense to take a long weekend in Manhattan in September.

Plus, this will be the best event this year for Xen Project users to meet and greet one another.  Excellent things happen when the talents of the community are allowed to cross-pollinate at User Summit!

Expected topics will include:

  • What’s Coming in Xen Project 4.5
  • Understanding and Using Xen4CentOS
  • Xen Project on OpenStack in the SUSE Cloud
  • High Availability with Xen Project
  • Using Cloud Operating Systems
  • plus many more!

The full schedule will be announced shortly.

To register for User Summit, click here.

August 18-19: Xen Project Developer Summit, Chicago, IL

For those who delve into the innermost parts of Xen Project, there is the Xen Project Developer Summit in Chicago, IL USA in August.  Co-located with LinuxCon North America and CloudOpen North America, this two-day event is the place to be for those who write code for or around Xen Project.

Some of the topics at this year’s Developer Summit include:

  • Xen Project 4.5 Roadmap
  • libvirt support for libxenlight
  • Scaling Xen’s Aggregate Storage Performance
  • Xen on ARM: Status and Performance
  • An Overview of the Verizon Cloud Architecture
  • osstest, Xen’s Automatic Testing Facility
  • and much, much more!

Click here for the full schedule.

At US$99 for two days of highly technical talks, this is an extradinary value for Xen Project software engineers.

Click here to register for Developer Summit.

And Sponsorship Opportunities Are Still Available!

If your organization has a project, product, or service which would be of interest to either a group of Xen Project users or developers, there are still sponsorship opportunies available.  For as little as US$1000, you can get your name in front of a group of Xen Project users, integrators, service providers, and developers.

Send email to russell(dot)pavlicek(at)XenProject.org for more information.

And we’ll be looking for you at one of the Xen Project Summits this year!

Posted in Announcements, Community, Events.

Tagged with .


The Docker exploit and the security of containers

We normally only cover news and information directly related to Xen in this channel, but we thought it might be useful to briefly expand our scope a bit to mention the recent discussion about the Docker security exploit.

What’s the news?

Well to begin with, a few weeks ago Docker 1.0 was released, just in time for DockerCon.

Then last week, with timing that seems rather spiteful, someone released an exploit that allows a process running as root within a Docker container to break out of a Docker container.

I say it was a bit spiteful, because as the post-mortem on Docker’s site demonstrates, it exploits a vulnerability which was only present until Docker 0.11; it was fixed in Docker 0.12, which eventually became Docker 1.0.

Nonetheless, this kicked off a bit of a discussion about Docker, Linux containers, and security in several places, including Hacker News, the Register, seclists.org, and the OSv blog.

There is a lot of interesting discussion there. I recommend skimming through the Hacker News discussion in particular, if you have the time. But I think the comment on the Hacker News discussion by Solomon Hykes from Docker, puts it best:

As others already indicated this doesn’t work on 1.0. But it could have. Please remember that at this time, we don’t claim Docker out-of-the-box is suitable for containing untrusted programs with root privileges. So if you’re thinking “pfew, good thing we upgraded to 1.0 or we were toast”, you need to change your underlying configuration now. Add apparmor or selinux containment, map trust groups to separate machines, or ideally don’t grant root access to the application.

I’ve played around with Docker a little bit, and it seems like an excellent tool for packaging and deploying applications. Using containers to separate an application from the rest of the user-space of your distribution, exposing it only to the very forward-compatible Linux API is a really clever idea.

However, using containers for security isolation is not a good idea. In a blog last August, one of Docker’s engineers expressed optimism that containers would eventually catch up to virtual machines from a security standpoint. But in a presentation given in January, the same engineer said that the only way to have real isolation with Docker was to either run one Docker per host, or one Docker per VM. (Or, as Solomon Hykes says here, to use Dockers that trust each other in the same host or the same VM.)

We would concur with that assessment.

Posted in Community.

Tagged with , , .


Release management: Risk, intuition, and freeze exceptions

I’ve been release coordinator for Xen’s 4.3 and 4.4 releases. For the 4.5 release, I’ve handed this role off to Konrad Wilk, from Oracle. In this blog, I try to capture some of my thoughts and experience about one aspect of release management: deciding what patches to accept during a freeze.

I have three goals when doing release management:

  1. A bug-free release
  2. An awesome release
  3. An on-time release

One of the most time-consuming seasons of being a release manager is during any kind of freeze. You can read in detail about our release process elsewhere; I’ll just summarize it here. During normal development, any patch which has the approval of the relevant maintainer can be accepted. As the release approaches, however, we want to start being more and more conservative in what patches we accept.

Obviously, no patch would ever be considered for acceptance which didn’t improve Xen in some way, making it more awesome. However, it’s a fact of software that any change, no matter how simple or obvious, may introduce a bug. If this bug is discovered before the release, it may delay the release, making it not on-time; or it may not be found until after the release, making the release not bug-free. The job of helping decide whether to take a patch or not falls to the release coordinator.

But how do you actually make decisions? There are two general things to say about this.

Risk

The only simple rule to follow that will make sure that there are no new bugs introduced is to do no development at all. Since we must do development, we have to learn to deal with risk.

Making decisions about accepting or rejecting patches as release coordinator is about making calculated risks: look at the benefits, look at the potential costs, look at the probabilities, and try to balance them the best you can.

Part of making calculated risks is accepting that sometimes your gamble won’t pay off. You may approve a patch to go in, and it will then turn out to have a bug in it which delays the release. This is not necessarily a failure: if you can look back at your decision and say with honesty, “That was the best decision I could have made given what I knew at the time”, then your choice was the right one.

The extreme example of this kind of thinking that of a poker player: a poker player may make a bet that she knows she only has a 1 in 4 chance of winning, if the pay-off is more than 4 to 1; say, 5 to 1. Even though she loses 75% of the time, the 25% that she does win will pay for the losses. And when she makes the bet and loses (as she will 75% of the time), she knows she didn’t make a mistake; taking risks is just a part of the game.

Obviously as release coordinator, the costs of a bug are generally higher than the benefits of a feature. But the general principle — of taking calculated risks, and accepting occasional failure as the inevitable consequence of doing so — applies.

Intuition

But how do we actually calculate the risks? A poker player frequently deals with known quantities: she can calculate that there is exactly a 25% chance of winning, exactly a 5x monetary pay-off, and do the math; the release coordinator’s decisions are not so quantifiable.

shutterstock_58404295This is where intuition comes in. While there are a handful of metrics that can be applied to patches (e.g., the number of lines in the patch), for the most part the risk and benefit are not very quantifiable at all: expert judgement / intuition is the only thing we have.

Now, research has shown that the intuition of experts can, under the right circumstances, be very good. Intuition can quickly analyze hundreds of independent factors, and compare against thousands of instances, and give you a result which is often very close to the mark.

However, research has also shown that in other circumstances, expert intuition is worse than random guessing, and far worse than a simple algorithm. (For a reference, see the books listed at the bottom.)

One of the biggest ways intuition goes wrong is by only looking at part of the picture. It is very natural for programmers, when looking at a patch, to consider only the benefits. The programmer’s intuition then accurately gives them a good sense of the advantage of taking the patch; but doesn’t warn them about the risk because they haven’t thought about it. Since they have a positive feeling, then they may end up taking a patch even when it’s actually too risky.

The key then is to make sure that your intuition considers the risks properly, as well as the benefits. To help myself with this, during the 4.4 code freeze I developed a sort of checklist of things to think about and consider. They are as follows:

  1. What is the benefit of this patch?
  2. What is the probability this patch has a bug?
  3. If this patch had a bug, what kind of bug might it be?
  4. If this patch had a bug, what is the probability we would find it
    before the release?

When considering the probability of a bug, I look at two things:

  • The complexity of the patch
  • My confidence in my / the other reviewers’ judgement.

Sometimes you’re looking at code you’re very familiar with or is straightforward; sometimes you’re looking at code that is very complicated or you’re not that familiar with. If the patch looks good and it’s code you’re familiar with, it’s probably fine. If the patch looks good but it’s code you’re not familiar with, there’s a risk that your judgement may be off.

When trying to think of what kind of bug it might be, I look at the code that it’s modifying, and consider things on a spectrum:

  1. Error / tool crash on unexpected trusted input; or normal input to library-only commands
  2. Error / tool crash on normal input, secondary commands / new functionality
  3. Error / tool crash on normal input, core commands
  4. Performance
  5. Guest crash / hang
  6. Host crash / hang
  7. Security vulnerability
  8. Data loss

Usually you can tell right away where in the list a bug might be. Modifying xenpm or xenctx? 3 max. Modifying the scheduler? Probably #6. Modifying hypercalls available to guests? #7. And so on.

When asking whether we’d find the bug before the release, consider the kind of testing the codepath is likely to get. Is it tested in osstest? In XenRT? Or is it in a corner case that few people really use?

After thinking through those four questions, and going over the criteria in detail, then my intuition is probably about as well-formed as it’s going to get. Now I ask the fifth question: given the risks, is it worth it to accept this patch?

After giving it it some thought, I went with my best guess. Sometimes I’m just not sure; in which case go away and do something else for a couple of hours, then come back to it (going over again the four questions to make sure they’re fresh in my mind). The first few dozen times this took a very long time; as I gained experience, judgements came faster (although many were still painfully slow).

In some cases, I just didn’t have enough knowledge of the code to make the judgement myself; this happened once or twice with the ARM code in the 4.4 release. In that case, my goal was to try to make sure that those who did have the relevant knowledge were making sound decisions: thinking about both the benefits and the risks and weighing them appropriately.

For those who want to look further into risk and intuition, several books have had a pretty big influence on my thinking in this area. Probably the best one, but also the hardest (most dense) one, is Thinking, Fast and Slow, by Daniel Kahneman. It’s very-well written and accessible, but just contains a huge amount of information that is different to the way you normally think. Not a light read. Another one I would recommend is The Black Swan, by Nassim Nicholas Taleb. And finally, Blink, by Malcolm Gladwell.

Posted in Community, Xen Development.

Tagged with .


Developer Summit Line-up Announced

I am pleased to announce the schedule of the Xen Project Developer Summit. The event will take place in Chicago on August 18-19, 2014.

The Project’s second annual developer event highlights best practices, user testimonials and advancements with the industry-leading open source hypervisor. Powering many of the world’s largest clouds in production today, Xen Project developers are also leading the way in server density, million-node data centers, graphic-intensive workloads, cloud operating systems and sophisticated enterprise security.

This year’s summit will present the most relevant topics to Xen Project developers and users who are pushing the limits on virtualization, ranging from typical server virtualization and cloud computing on x86 servers to new developments with ARM servers, networking, automotive, cloud operating systems, enterprise security and mobility.

Following is a sampling of confirmed speakers and presentations to be discussed in Chicago:

  • James Bielman, Research and Engineering at Galois, XenStore Mandatory Access Control — proposes additional security access features for Xen Project software;
  • Mihai Donțu, Technical Project Manager at Bitdefender, Zero-Footprint Guest Memory Introspection from Xen — discusses how the introspection API in the Xen Project hypervisor can be used to detect, prevent and take action on several categories of malware attacks;
  • James Fehlig, Software Engineer at SUSE Linux, libvirt support for libxenlight – covers the status of Xen Project libvirt integration and outlines planned improvements;
  • Lars Kurth, Xen Project Advisory Board Chairman, State of Xen Project Software – gives an overview of the Xen Project development community and community at large;
  • Jun Nakajima, Principal Engineer at Intel Open Source Technology Center, Xen as a High-Performance Network Functions Virtualization (NFV) Platform – introduces Xen as a NFV platform and outlines solutions to remove challenges for deploying the Xen Project hypervisor for NFV applications as well as shares best practices;
  • Nathan Studer, Technical Lead at DornerWorks, Xen and The Art of Certification – gives an overview of certification requirements in emerging use-cases such as automotive, medical, and avionics and lays out a path toward certifying Xen Project technology in these industries;
  • Don Slutz, Software Architect at Verizon Terremark, Overview of Verizon Cloud Architecture – presents Verizon Cloud’s architecture, design goals and planned contributions to the Xen Project community; and
  • Stefano Stabellini, Senior Principal Software Engineer at Citrix and Xen Project Contributor, Xen on ARM Status Update and Performance Benchmarks — gives the latest developments with the Xen Project hypervisor on ARM architecture.

Birds of a Feather session and Discussions

Besides presentations, the developer summit will also provide an opportunity for in-depth interactive discussions (Birds of a Feather sessions), which allow deep interaction and collaboration between Xen Project developers and community members. These will happen in a second track alongside the main event. To submit a BoF, please go to the BoF submission page.

For more information about Xen Project Developer Summit 2014, including how to register and to view the complete schedule, visit: events.linuxfoundation.org/events/xen-project-developer-summit.

Posted in Uncategorized.