Comments on: NUMA and Xen: Part II, Scheduling and Placement http://blog.xen.org/index.php/2012/05/16/numa-and-xen-part-ii-scheduling-and-placement/ Community Blog Mon, 13 May 2013 10:17:52 +0000 hourly 1 By: Marcus Granado http://blog.xen.org/index.php/2012/05/16/numa-and-xen-part-ii-scheduling-and-placement/comment-page-1/#comment-2044 Marcus Granado Tue, 29 May 2012 10:09:22 +0000 http://blog.xen.org/?p=4650#comment-2044 As part of the node affinity concept you are developing, I wonder if there isn't a need for a couple of policies regarding how sets of VMs interact: affine: - optimized for data flowing across VMs in the same host, especially if data copying is involved (eg. a webapp and db VMs working together) - tries to keep VMs sharing data in the same node to optimize memory throughput for each VM - tries to keep VMs sharing pcpu caches to optimize caching for each VM - memory throughput limited by number of parallel access channels in local memory node anti-affine: - optimized for data flowing between VM and outside the host (eg. when pci-passthrough is used in NICs) - tries to keep VMs using different memory nodes to optimize memory throughput for each VM - tries to keep VMs using different pcpu caches to optimize caching for each VM - memory throughput not limited by number of parallel access channels in local memory node (since they are located in different nodes with their own independent memory controllers and paths) The idea is that sometimes you have a bunch of VMs that would like to cooperate (affine), and sometimes you have a bunch of VMs that would like to be kept as apart as possible from other VMs (anti-affine). Maybe xen can figure out the cases by watching the memory mapping activity for each VM. I would expect n sets of affine and anti-affine VMs, and even soft conflicts that the scheduler would need to solve when deciding on which pcpu to run a VM vcpu next to conform to all the affine and anti-affine sets. I have a feeling that it would be important to have policies that take into account all of the VM interactions. The policies you talked about (greedy, packed, spread) do not seem to take this into consideration, but let me know if I missed something. cheers, As part of the node affinity concept you are developing, I wonder if there isn’t a need for a couple of policies regarding how sets of VMs interact:

affine:
- optimized for data flowing across VMs in the same host, especially if data copying is involved (eg. a webapp and db VMs working together)
- tries to keep VMs sharing data in the same node to optimize memory throughput for each VM
- tries to keep VMs sharing pcpu caches to optimize caching for each VM
- memory throughput limited by number of parallel access channels in local memory node

anti-affine:
- optimized for data flowing between VM and outside the host (eg. when pci-passthrough is used in NICs)
- tries to keep VMs using different memory nodes to optimize memory throughput for each VM
- tries to keep VMs using different pcpu caches to optimize caching for each VM
- memory throughput not limited by number of parallel access channels in local memory node (since they are located in different nodes with their own independent memory controllers and paths)

The idea is that sometimes you have a bunch of VMs that would like to cooperate (affine), and sometimes you have a bunch of VMs that would like to be kept as apart as possible from other VMs (anti-affine). Maybe xen can figure out the cases by watching the memory mapping activity for each VM. I would expect n sets of affine and anti-affine VMs, and even soft conflicts that the scheduler would need to solve when deciding on which pcpu to run a VM vcpu next to conform to all the affine and anti-affine sets.

I have a feeling that it would be important to have policies that take into account all of the VM interactions. The policies you talked about (greedy, packed, spread) do not seem to take this into consideration, but let me know if I missed something.

cheers,

]]>