Rebalance / recovery speed

Form https://www.suse.com/support/kb/doc/?id=000019693:

• osd max backfills This is the maximum number of backfill operations allowed to/from OSD. The higher the number, the quicker the recovery, which might impact overall cluster performance until recovery finishes.
• osd recovery max active This is the maximum number of active recover requests. Higher the number, quicker the recovery, which might impact the overall cluster performance until recovery finishes.
• osd recovery op priority This is the priority set for recovery operation. Lower the number, higher the recovery priority. Higher recovery priority might cause performance degradation until recovery completes.

Defaults in Proxmox:

# ceph config get osd osd_max_backfills
1
# ceph config get osd osd_recovery_max_active 
0
# ceph config get osd osd_recovery_op_priority
3

Speed up CEPH recovery:

ceph tell 'osd.*' injectargs --osd-max-backfills=2 --osd-recovery-max-active=6 --osd_recovery_op_priority=2

or

ceph tell 'osd.*' injectargs --osd-max-backfills=3 --osd-recovery-max-active=9 --osd_recovery_op_priority=2

And return to defaults:

ceph tell 'osd.*' injectargs --osd-max-backfills=1 --osd-recovery-max-active=0 --osd_recovery_op_priority=3