Veeam Error: [Warning] XXX all stuck VM snapshot consolidation attempts have failed

I recently ran into an interesting error with Veeam, one which I haven’t seen before despite using it for years. The error in question: [Warning] XXXX all stuck VM snapshot consolidation attempts have failed (where XXXX is the VM name).

Veeam has been pretty good at removing the snapshots for awhile now, especially since they introduced their Snapshot Hunter feature in v8, so I was a little surprised to see an error such as this. The first thing I did was try and consolidate the snapshots via the vSphere client but I ended up getting the ever so helpful error “An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED.” Great, at least now I could confirm the disk was being locked by something.

The message that I had originally received from Veeam (via email) indicated that an external process may be locking the file, which seemed to the case given the above message. I couldn’t think of anything that would be locking the VMDK files as we haven’t introduced anything new into the environment lately, nor do we use any other backup products that might interfere. Just for the heck of it I decided to power off the machine and try the consolidation but I received a similar error: “An error occurred while consolidating disks: msg.fileio.lock.”

Next up on the to do list: manually create a snapshot and then try consolidating that one. I ended up trying this twice, once with “Snapshot the virtual machine’s memory” checked off, and once without. In both cases I was able to create the snapshot without issue, but trying to consolidate the snapshots failed. I ended up deleting the snapshots and trying a trying a consolidation yet again, but I was faced with the same error message as above.

Lastly I recalled that when Veeam runs as a virtual appliance (specifically when the Veeam Proxy is running on a VM), it will mount the VM’s drives to itself when backing up.  Sure enough when I pulled up the VM Properties for the Veeam appliance, I saw the disks in question were attached. I selected the disks and clicked on Remove (careful to not delete the files, just remove them), applied the changes and sure enough the consolidation process kicked off.

In my case, the VM is about 60 GB and the consolidation process took just under 20 minutes to run. I had a ping running the whole time to see if we would be hit with VM stun, but only two packets dropped: once when the consolidation was initiated, and once when it wrapped up.

This is apparently an known issue as KB 1775 goes over the process. The actual cause is that Veeam uses a VMware API call to remove the VMDK, so for whatever reason that call either wasn’t made or wasn’t completed successfully. Hopefully this is just a one off – if I see that the consolidation issue crops up again I might try adding the VM to the job directly via the ESXi host, and not through vCenter.