meta data for this page
Book Creator
 Add this page to your book
Book Creator
 Remove this page from your book  

 Manage book(0 page(s))
 Help

CEPH performance

Performance tips

  • adapt PG to number of OSDs to spread traffic evenly
  • use krbd
  • enable writeback on VMs (possible data loss on consumer SSDs)

Issues

auth: unable to find a keyring

It is not possible to create ceph OSD neither from WebUI nor cmdline: 

pveceph osd create /dev/sdc
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap
 stderr: 2021-01-28T10:21:24.996+0100 7fd1a848f700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2021-01-28T10:21:24.996+0100 7fd1a848f700 -1 AuthRegistry(0x7fd1a0059030) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
/etc/pve/ceph.conf
[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring
 
[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

ceph.conf Variables

  • $cluster - cluster name. For proxmox it is ceph
  • $type - daemon process mds osd mon
  • $id - daemon or client indentifier. For osd.0 it is 0
  • $host - hostname where the process is running
  • $name - Expands to $type.$id. I.e: osd.2 or client.bootstrap
  • $pid - Expands to daemon pid

SOLUTION: 

cp /var/lib/ceph/bootstrap-osd/ceph.keyring /etc/pve/priv/ceph.client.bootstrap-osd.keyring

alternative to try: change ceph.conf

Unit -.mount is masked.

Running command: /usr/bin/systemctl start ceph-osd@2
 stderr: Failed to start ceph-osd@2.service: Unit -.mount is masked.
-->  RuntimeError: command returned non-zero exit status: 1

It was caused by gparted which wasn't correctly shutdown.

Solution: 

systemctl --runtime unmask -- -.mount

To list runtime masked units: 

ls -l /var/run/systemd/system | grep mount | grep '/dev/null' | cut -d ' ' -f 11

To unescape systemd unit names: 

 systemd-escape -u 'rpool-data-basevol\x2d800\x2ddisk\x2d0.mount'