Why can't I start my VM when there is plenty of free memory?

Issue #1: A number of customers has reported than VM's fail to start when there is plently of memory.  The error reported from the Hyper-V manager is displayed below.  Here I tried to start a 4GB VM when there was 7+GB free.   What's going on?

The most common cause has to do with a bug in Windows Server 2008 NUMA memory allocations.  This means you'll only hit this bug on NUMA machines (all multi-proc AMD machines are NUMA + many high end Intel based servers).  You can verify you have this bug by looking at the Task Manager reported "Cached" and "Free" memory.
Here free memory is 9MB but really much of the "Cached" memory can be made free also.  Standby Pages, Modified Pages, and System File Cache make up the "Cached" memory.  Most (all but a couple 100MB) of the cached memory can be converted to "free",
Fix #1: To resolve the issue will need to contact Microsoft Product Support http://support.microsoft.com/contactus/cu_sc_more_master#tab1 and request the following hotfix - KB953585
Alternate Fix #1: The second way to avoid this bug is to not do any work in the root parition directly.  Create a "management" VM and do your work there.
Background #1:  The most common way of hitting this issue is by running large file copies in the root (aka host) parition.  Doing lots of file copies causes the System File Cache to bloat in size.  When Hyper-V goes to start it will flush the System File Cache if it needs memory.  The System File Cache pages (aka file pages from the copies) move to the Standy list and from there will get zeroed and freed.  The bug is WS08 prevents NUMA allocations from the Standyby list.
Issue #2: The second most common cause to the can't start the VM issue when it appears there is plenty of memory is due to configuring a NUMA machine to use fewer processors than the system actually has.  Typically this is done using "bcdedit /set numproc X". 
Fix #2: If you set NUMA machine in this mode for testing memory attached to the "hidden" processors will be inaccessible to Hyper-V.  Some machines allow you to reassign the memory (via the BIOS or front end controller) to the "active" processors.
  - Tony Voellm

Posted in: All