DRBD: Troubleshooting and Error Recovery of Hard Drive Failure or Replacement

DRBD refers to block devices designed as a building block to form high availability cluster. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based RAID-1. (DRBD)

Sometimes we have a disk failure on hard drive which contains DRBD backing device. The following steps can be used to replace or recover the failed drive.

detach drbd resource from broken backing storage
```
# drbdadm detach <resource>
```

check state of drbd disk

# drbdadm dstate <resource>
Diskless/UpToDate

if using internal meta data, bind DRBD device to new hard disk
```
# drbdadm create-md <resource>
# drbdadm attach <resource>
```
if using external meta data DRBD unable to recognize hard drive was swapped, need additional step
```
# drbdadm create-md <resource>
# drbdadm attach <resource>
# drbdadm invalidate <resource>
```

Reference: http://www.drbd.org/users-guide-8.4/ch-troubleshooting.html