Matt That IT Guy

Racking And Stacking In A Cloud-Based World

HomelabVeeamVMware

Manually Removing I/O Filters From vSphere

A little while back I was playing around with the CDP feature coming in Veeam Backup & Recovery V11. One of the requirements to use it is installing an I/O Filter on your ESXi hosts. The filter is how the magic happens, and there is nothing unusual about this step. Many vendors have I/O Filters available for ESXi . But given that this is a homelab, I tend to do a lot silly things in there. In my case, I found that I was unable to vMotion a VM, which led me down a rabbit hole …

Heed this warning before ye wander further: If you find yourself needing to do any of the below, I cannot stress enough that:

  1. you should have support request open with VMware (if possible)
  2. make sure you have backups for any data that may be affected
  3. make sure you test those backups 🙂

HOW DID I GET HERE?

To start off, I found myself wanting to rebuild my home lab. While mucking about with my current vCenter, I found that I kept receiving some odd errors whilst poking around in vCenter. When I say odd errors, I’m talking about page timeouts, controller issues (e.g. the same NUCs showing different values), and just general weirdness. I opted to just blow away vCenter as I recently applied an update that seemed to not complete fully successfully – that is to say it took multiple attempts, and required some manual intervention. Between that and the weirdness, I figured I would blow away vCenter and spin up a new one. Not a big deal considering I am looking at re-architecting my lab anyways.

So I spin up a new vCenter, attach the hosts, make sure the shared iSCSI datastore is available across all the hosts, setup networking, and start vMotioning VMs off my NUCs. All is fine and dandy, until I hit a VM that I was using Veeam CDP with. When attempting to vMotion, I would receive the error pictured below (I/O filters are not installed on the destination host).

IOFilter Error

MANUALLY REMOVING THE I/O FILTER

Since I didn’t want to add the I/O Filter to another, non-clustered host, I figured the next best thing would be to remove the filter from the existing host. This process was actually much more straightforward than I would have thought. In short:

  • put your host into maintenance mode
  • Run “esxcli software vib list” and confirm the name of the filter(s)
  • Run “esxcli software vib remove -n veecdp” to remove the filters
  • Take your host out of maintenance mode – no reboot required 🙂

Typically, you can do something like the above via vCenter. However, me being me, I spun up a new vCenter, this was not an option. KB 76633 does cover what the steps would likely be at a high level. If you find yourself in the situation that I did, doing this via vCenter would be likely the preferred option … of course, the primary option should be using the vendor-supplied tools to uninstall the filter. In my case, because I removed the original vCenter, I couldn’t 🙂

VMOTIONING THE VM

With the VIBVMDK Descriptors uninstalled, I should be good to go, right? Nope. When attempting to vMotion the VM again, I was still greeted with the error “a specified parameter was not correct: vibid“. Some digging around shows that the VMDK file itself was still looking for the filter, which now did not exist on the host, or in vCenter. So where is the filter being registered? My first thought was it must be somewhere specific to the VM, likely something to do with the VMDK since this is an I/O Filter.

Some poking around led me to look at the VMDK file for the VM. Although the term VMDK is used quite often to mean the VM’s hard drive, there is also a VMDK file associated with VMs which is actually just a descriptor file. These files contain configuration details, such adapter types, thin provision status, etc. Whilst looking in there, I noted two entries:

  • ddb.iofilters
  • ddb.sidecars

Both had values related to the I/O Filter that I had ripped out of the environment. As mentioned, because these are just descriptor files, I simply copied the file first to have a backup. After deleting those two lines and saving the file, I powered the VM back on (successfully – first step out of the way), and was glad to see I could then vMotion the VM in question.

WHAT DID I LEARN?

Well for one, I seem to enjoy creating more work for myself than necessary. Seriously though, in a perfect world I wouldn’t run into issues like this. But, because it is a home lab, I cut a lot of corners that I would never recommend in any other environment. I probably could have gotten away with just editing the VMDK file to start with, but I still wouldn’t want those I/O Filters on the hosts.

In short, I decided to do a quick writeup on this. Chances are I’ll do the same thing again in 6 months and I won’t remember what I did to fix it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.