How to get a memory dump of a virtual machine from its hypervisor

[Igor Akhmetov]

Advice and Solutions (Forum Knowledgebase) Disclaimer. Read before using materials.

Description and cautions

This article explains how to save a virtual machine memory dump in different hypervisor environments. You may find this information useful if you need to save a memory dump of an unresponsive or non-booting virtual machine.

Instructions for each hypervisor environment: 

• VMware ESXi/vSphere
• Microsoft Hyper-V
• Proxmox VE
• KVM
• Citrix Hypervisor

VMware vSphere

You can do this either via vCenter Client or ESXi host client.

1. Create a snapshot of the VM

   1.1. Right click on the VM in the list of all virtual machines → Snapshots → Take snapshot
   
   
   1.2. Check the Snapshot the virtual machine's memory checkbox

2. Download snapshot file.

   2.1. For ESXi:
   2.1.1. Right-click on Storage → Browse datastores
   2.1.2. Select VM's datastore
   2.1.3. Open VM's folder
   
   2.1.4. Right-click on the newest .vmsn file → Download
   2.1.5. Right-click on the newest .vmem file → Download
   2.2. For vCenter:
   2.2.1. Open Datastores tab of the VM view and click on the datastore listed
   
   2.2.2. Open Files tab of the datastore view and find the folder of the virtual machine
   2.2.3. Select newest .vmem and .vmsn files and click Download

 

3. Download vmss2core utility.

vmss2core is included with VMWare Workstation, and is available in C:\Program Files(x86)\VMware\VMware Workstation\ on 64-bit versions of Windows
vmss2core: https://flings.vmware.com/vmss2core

4. Extract the memory dump from downloaded snapshot.

For a snapshot of a VM running:

1. Windows 8/Server 2012 and newer 

<path to vmss2core's folder>\vmss2core -W8 <.vmsn file path> <.vmem file path>

2. Older versions of Windows 

<path to vmss2core's folder>\vmss2core -W <.vmsn file path> <.vmem file path>

3. Linux 

<path to vmss2core's folder>\vmss2core -N <.vmsn file path> <.vmem file path>

It should save the dump file to the working directory.

Microsoft Hyper-V

This method is only applicable to Windows VMs.

To get a memory dump of a Hyper-V virtual machine, you need the kernel debugger included in the Windows SDK.

1. Download Windows SDK Installer and LiveKD to the Hyper-V host.

   Windows SDK Installer: https://go.microsoft.com/fwlink/?linkid=2237387
   LiveKD: https://download.sysinternals.com/files/LiveKD.zip

2. Run Windows SDK installer in Powershell:

   .\winsdksetup.exe /features OptionId.WindowsDesktopDebuggers /q /norestart

   To check if the installation has completed, check the Task Manager while installing the Windows SDK. It should look like the one shown in this screenshot:
   
   Once the SDK installation is complete, the winsdksetup.exe process should disappear.

3. Unpack LiveKD.zip by running the following in Powershell:

   Expand-Archive LiveKD.zip

4. Set _NT_SYMBOL_PATH environment variable.

   xset _NT_SYMBOL_PATH "srv*c:\symbols*http://msdl.microsoft.com/download/symbols"

5. Relogin to make the variable available to LiveKD.

6. Run following to save a memory dump to a specified path on Hyper-V server's storage:

   .\LiveKD\livekd64.exe -hv <VM Name> -k <Path to Windows SDK install>\Debuggers\x64 -p -o <Path to save memory dump>

    Default Windows SDK path is C:\Program Files(x86)\Windows Kits\10.

7. One way to copy the dump is to mount a network drive in Powershell and copy the file to it.

   $cred = Get-Credential <Domain\username>

   Get-Credential asks for the password of the specified user and stores the credential used by New-PSDrive in a variable. New-PSDrive mounts an SMB/CIFS share at specified network path as a network drive.

   New-PSDrive -Name <Drive Letter> -Persist -PSProvider FileSystem -Root "<network path>" -Credential $cred

Proxmox VE

1. Open Monitor tab of the VM.

2. To create a dump in ELF format, execute the following:

   dump-guest-memory -d <path to save the file>

   

   -d detaches the process from the shell, that is needed, which is necessary because Proxmox has a time limit on monitor operations.

   To create a dump in Windows crashdump format, VM has to be started with a vmcoreinfo device and have latest virtio-win drivers installed. The VM can be started with vmcoreinfo device by running the following in the node's shell:

   echo $(sudo qm showcmd <VMID>) -device vmcoreinfo | sudo bash -s --

   If the VM has a TPM configured:

   export VMID=<VMID> && swtpm socket --tpmstate backend-uri=file://<path to tpm state file>,mode=0600 --ctrl type=unixio,path=/var/run/qemu-server/$VMID.swtpm,mode=0600 --pid file=/var/run/qemu-server/$VMID.swtpm.pid --terminate --daemon --log file=/run/qemu-server/$VMID-swtpm.log,level=1,prefix=[id=$(date +%s)] --tpm2 && echo $(sudo qm showcmd $VMID) -device vmcoreinfo | sudo bash -s --

   By default Proxmox creates a Thin provisioned LVM storage, called local-lvm, which path is /dev/pve
   After that a dump can be created by running the following:

   dump-guest-memory -d -w <path to save the file>

   2.1. Wait until dump file size reaches the amount of ram allocated to the VM, if it is stuck at 0 bytes, it means that the VM couldn't load the vmcoreinfo driver and the only way is to create an ELF dump
   To check it run following in Proxmox node's shell, which can be accessed via Shell tab in node's view:

   watch -n 1 ls -al --block-size=M <dump file path>

   
   This command will run ls every second showing file's size, it may take some time to show anything, because of the way Proxmox is saving the dump.
   

Copy the dump from the node, one way it can be done is by using scp:

scp <user>@<KVM host ip>:<dump file path> <local path>

KVM

This part is applicable for generic KVM servers with libvirt, Alt Server-V, OpenStack, OpenNebula and any other virtualization environments based on them.

To save a memory dump in ELF format to the KVM host, run as root:

sudo virsh dump --memory-only <name of the vm> <path to dump>

All virsh commands can be run without sudo, if the user is in libvirt group

To save a dump in Windows crashdump format, VM has to have latest virtio-win drivers installed and vmcoreinfo feature has to be enabled in VM's configuration file:

export VMID=<vm name>; export xml_path="/etc/libvirt/qemu/$VMID.xml"; sudo grep vmcoreinfo $xml_path; if [ $? -ne 0 ]; virsh shutdown $VMID; sudo systemctl stop libvirtd; then sudo sed -i "s/<features>/&\n    <vmcoreinfo state=\"on\"\/>/" $xml_path; sudo systemctl start libvirtd; sudo virsh start $VMID; fi;

After VM boots up (or crashes), run the following to create the dump:

sudo virsh qemu-monitor-command dump-guest-memory -w <path to save the file>

Check dump file size, if it is 0 bytes, it means that the VM couldn't load the vmcoreinfo driver and the only way is to create an ELF dump

ls -al <dump file path>

Copy the dump from the node, one way it can be done is by using scp:

scp <user>@<KVM host ip>:<dump file path> <local path>

Citrix Hypervisor

The only way to capture a memory dump in a virtual machine running on Citrix Hypervisor is to use memory dump mechanisms built into the guest OS, but a crash of the guest can be triggered from the hypervisor by running:

xen-hvmcrash <id>

How to collect a full memory dump on Windows: https://support.kaspersky.com/common/diagnostics/10659