Multipath Madness

2018‑05-18 Wander Boessenkool multipath , storage , bork bork bork , data corruption

This is one guaranteed to cause some great storage shenanigans on systems from the RHEL6 era. Add the following line inside the defaults section and all device sections in /etc/multipath.conf:

  getuid_callout  "echo one-size-fits-none"

This will make the defaults section resemble something like the following. Line 6 is the line we’ve just added.

defaults {
  udev_dir    /dev
  polling_interval  10
  selector    "round-robin 0"
  path_grouping_policy  multibus
  getuid_callout  "echo one-size-fits-none"
  prio      alua
  path_checker    readsector0
  rr_min_io   100
  max_fds     8192
  rr_weight   priorities
  failback    immediate
  no_path_retry   fail
  user_friendly_names yes
}

What it does

If you’re not running multipathing: Nothing.

If you are running multipathing: Having this line in your `/etc/multipath.conf’ will generate tons of fun issues entropy. The multipath daemon will start treating every disk it queries as another path to the same disk. This will cause such fun such as missing volumes, incorrect data being read, and most fun of all; Data Corruption.

Why it works

The multipath daemon reads the WWIDs of all disks it scans, it does this using the program (and options) specified in the getuid_callout setting. The default for this is /lib/udev/scsi_id --whitelisted --device=/dev/%n, which boringly reads the actual WWID from your disk. This causes devices with the same WWID to be treated as different paths to the same underlying disk.

Our replacement on the other hand does something much more sensible, it returns the same string for each and every device. This will make the multipathing daemon treat every single block device it scans on your system as a path to a single disk. Actually using the multipath device will cause irreversible data loss fun.

TL;DR

• Data Integrity
• Fun