Veeam Error: [Warning] XXX all stuck VM snapshot consolidation attempts have failed

I recently ran into an interesting error with Veeam, one which I haven’t seen before despite using it for years. The error in question: [Warning] XXXX all stuck VM snapshot consolidation attempts have failed (where XXXX is the VM name).

2015-09-veeam-Error1
Veeam has been pretty good at removing the snapshots for awhile now, especially since they introduced their Snapshot Hunter feature in v8, so I was a little surprised to see an error such as this. The first thing I did was try and consolidate the snapshots via the vSphere client but I ended up getting the ever so helpful error “An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED.” Great, at least now I could confirm the disk was being locked by something.

2015-09-veeam-Error2The message that I had originally received from Veeam (via email) indicated that an external process may be locking the file, which seemed to the case given the above message. I couldn’t think of anything that would be locking the VMDK files as we haven’t introduced anything new into the environment lately, nor do we use any other backup products that might interfere. Just for the heck of it I decided to power off the machine and try the consolidation but I received a similar error: “An error occurred while consolidating disks: msg.fileio.lock.”

Next up on the to do list: manually create a snapshot and then try consolidating that one. I ended up trying this twice, once with “Snapshot the virtual machine’s memory” checked off, and once without. In both cases I was able to create the snapshot without issue, but trying to consolidate the snapshots failed. I ended up deleting the snapshots and trying a trying a consolidation yet again, but I was faced with the same error message as above.

2015-09-veeam-Error3Lastly I recalled that when Veeam runs as a virtual appliance (specifically when the Veeam Proxy is running on a VM), it will mount the VM’s drives to itself when backing up.  Sure enough when I pulled up the VM Properties for the Veeam appliance, I saw the disks in question were attached. I selected the disks and clicked on Remove (careful to not delete the files, just remove them), applied the changes and sure enough the consolidation process kicked off.

In my case, the VM is about 60 GB and the consolidation process took just under 20 minutes to run. I had a ping running the whole time to see if we would be hit with VM stun, but only two packets dropped: once when the consolidation was initiated, and once when it wrapped up.

This is apparently an known issue as KB 1775 goes over the process. The actual cause is that Veeam uses a VMware API call to remove the VMDK, so for whatever reason that call either wasn’t made or wasn’t completed successfully. Hopefully this is just a one off – if I see that the consolidation issue crops up again I might try adding the VM to the job directly via the ESXi host, and not through vCenter.

4 thoughts on “Veeam Error: [Warning] XXX all stuck VM snapshot consolidation attempts have failed

  • Pingback: Veeam: msg.snapshot.erro-DISKLOCK

  • January 25, 2016 at 3:11 pm
    Permalink

    Man you’re a genius.
    Thank you so much as you not only resolved my problem but also explained the very puzzling issue we had of our print server which was acting as a defacto proxy getting all these additional disks.
    This is one for our work knowledge base.

    Reply
    • January 25, 2016 at 3:17 pm
      Permalink

      Glad to help out.

      I chuckled to myself thinking about having a print server that was spontaneously growing disks.

      Reply
      • January 25, 2016 at 3:31 pm
        Permalink

        Yeah trust me, nobody had a clue what was going on 🙂
        We had a feeling it had something to do with the other server that would’nt consolidate, but we just could’nt figure it out.
        I’m surprised your solution is’nt on all the veeam or vmware forums as all they recommend is restarting all management agents, storage migration, restarting hosts etc.
        Fair play Matt

        Reply

Leave a Reply

%d bloggers like this: