Proxmox VE provide really good support for ZFS. Right from the installation, we may install the OS on ZFS. Using the web interface, we may create filesystem or volume to be used by virtual machine. Virtual Machine snapshot make use of ZFS snapshot. It also support some feature that we have yet to test like replication between physical host.
Over the time, you will face a disk failure. Hard disk are not immortals !
In that scenario, you need to know how to replace a the failed disk by a new one without compromising all the system. Various guides that explain how to replace a disk in a ZFS pool are not applicable for Proxmox VE. They make assumption about how the disk should be formatted that is not compatible with Proxmox VE. The Proxmox wiki do provide a good procedure to replace the disk, but it's error prone and required you to run successively some commands line only to properly format the disk.
To avoid error and to make the procedure seamless, we have create a little script. zfs-format-disk help you to properly partition the disk and to install grub on it.
curl http://git.patrikdufresne.com/pdsl/zfs-format-disk/raw/master/zfs-format-disk > /usr/local/sbin/zfs-format-disk
chmod +x /usr/local/sbin/zfs-format-disk
Here's the "short" explanation. Continue to read to get more details.
sudo zpool status
sudo ledctl locate=/dev/sdf
sudo zfs-format-disk /dev/sdf
sudo zpool replace rpool sdf2
First, you should check the state of the pool:
sudo zpool status
It should be DEGRADED and one of the disk should be OFFLINE or UNAVAIL depending of the failure. In case the disk is OFFLINE, you should double check the reason. dmesg should provide more details. It's usually related to some SATA error. If the problem doesn't occur repetitively, it's safe enough to reuse the same disk and to bring it ONLINE again: zpool online rpool sdf2
Once you have identify the device, you also need to identify the physical disk to remove the tray and put the new disk in. If you have a server with a backplane, you may use ledctl to light on a led to locate the device.
sudo ledctl locate=/dev/sdf
Once the new disk is in place, you are ready to format the disk using zfs-format-disk to create the partition and install grub on it.
sudo zfs-format-disk /dev/sdf
sudo zpool replace rpool sdf2
You may need to force the replacement of the disk with -f if it was used in a different pool before.