meta data for this page
This is an old revision of the document!
CEPH performance
block.db and block.wal
The DB stores BlueStore’s internal metadata and the WAL is BlueStore’s internal journal or write-ahead log. It is recommended to use a fast SSD or NVRAM for better performance.
Important Since Ceph has to write all data to the journal (or WAL+DB) before it can ACK writes, having this metadata and OSD performance in balance is really important!
For hosts with multiple HDDs (multiple OSDs), it is possible to use one SSD for all OSDS DB/WAL (one partition per OSD).
NOTE: The recommended scenario for mixed setup for one host is to use
- multiple HDDS (one OSD per HDD)
- one fast SSD/NVMe drive for DB/WAL (i.e. 30GB per 2TB HDD only needed).
Proxmox UI and CLI expects only whole device as DB device, not partition!. It will not destroy existing drive. It expect LVM volume with free space and it will create new LVM volume for DB/WAK.
Ceph native CLI can work with partition specified as DB (it also works with whole drive or LVM).
MORE INFO:
DB/WAL sizes
- If there is <1GB of fast storage, the best is to use it as WAL only (without DB).
- if a DB device is specified but an explicit WAL device is not, the WAL will be implicitly colocated with the DB on the faster device.
DB size:
- (still true for Octopus 15.2.6 ) DB should be 30GB. And this doesn't depend on the size of the data partition.
- all block.db sizes except 4, 30, 286 GB are pointless,
- should have as large as possible logical volumes
- for RGW (Rados Gateway) workloads: min 4% of block device size
- for RBD (Rados Block Device) workloads: 1-2% is enough (2% from 2TB is 40GB)
- according to
ceph daemon osd.0 perf dump | jq .bluefs
80GB was reserved on HDD for DB, where 1.6-2.4GB is used.
Adding journal DB/WAL partition
If an OSD needs to be shutdown for maintenance (i.e. adding new disc), please set ceph osd set noout
to prevent unnecessary data balance.
Create parition on NVM drive
Reorganize existing NVM/SSD disc to make some free space. Create empty partition on free space.
Cut some space from zpool cache NVM partition
# remove cache partition from zpool zpool list -v zpool remove rpool /dev/nvme0n1p3 ... reorganize partition ... blkid zpool add rpool cache 277455ae-1bfa-41f6-8b89-fd362d35515e
Cut some space from zpool
Example how to cut some space from nvmpool
zpool with spare temporary drive:
- we have 1 spare HDD which will be new Ceph OSD in future
- zpool doesn't support online shrinking.
- move
nvmpool
to spare HDD, to release NVMnvmpool
partition.
zpool replace nvmpool nvme0n1p4 sdc
# zpool status nvmpool pool: nvmpool state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Thu May 6 14:13:32 2021 70.2G scanned at 249M/s, 21.0G issued at 74.4M/s, 70.2G total 21.1G resilvered, 29.91% done, 00:11:17 to go config: NAME STATE READ WRITE CKSUM nvmpool ONLINE 0 0 0 replacing-0 ONLINE 0 0 0 nvme0n1p4 ONLINE 0 0 0 sdc ONLINE 0 0 0 (resilvering)
- wait for resilver
- reorganize partitions
- replace disks again
zpool replace nvmpool sdc nvme0n1p4
Replace OSD
ceph osd tree ceph device ls-by-host pve5 DEVICE DEV DAEMONS EXPECTED FAILURE TOSHIBA_HDWD120_30HN40HAS sdc osd.2 ### Switch OFF OSD. Ceph should rebalance data from replicas when OSD is switched off directly ceph osd out X ## or better use lines below: # this is optional for safety for small clusters instead of using ceph out osd.2 ceph osd reweight osd.X 0 # wait for data migration away from osd.X watch 'ceph -s; ceph osd df tree' # Remove OSD ceph osd out X ceph osd safe-to-destroy osd.X ceph osd down X systemctl stop ceph-osd@X.service ceph osd destroy X #pveceph osd destroy X # to remove partition table, boot sector and any OSD leftover: ceph-volume lvm zap /dev/sdX --destroy ## it is not possible to specify DB partition with pveceph command (read begining of page): # pveceph osd create /dev/sdc --db_dev /dev/nvme0n1p3 ## it requires whole device as db dev with LVM and will create new LVM on free space, i.e. # pveceph osd create /dev/sdc --db_dev /dev/nvme0n1 --db_size 32G ## so direct ceph command will be used: # Prevent backfilling when new osd will be added ceph osd set nobackfill ### Create OSD: ceph-volume lvm create --osd-id X --bluestore --data /dev/sdc --block.db /dev/nvme0n1p3 # or split above into two step: ceph-volume lvm prepare --bluestore --data /dev/sdX --block.db /dev/nvme0n1pX ceph-volume lvm activate --bluestore X e56ecc53-826d-40b0-a647-xxxxxxxxxxxx # also possible: ceph-volume lvm activate --all ## DRAFTS: #ceph-volume lvm create --cluster-fsid 321bdc94-39a5-460a-834f-6e617fdd6c66 --data /dev/sdc --block.db /dev/nvme0n1p3 #ceph-volume lvm activate --bluestore <osd id> <osd fsid>
Verify:
ls -l /var/lib/ceph/osd/ceph-X/ lrwxrwxrwx 1 ceph ceph 93 sty 28 17:59 block -> /dev/ceph-16a69325-6fb3-4d09-84ee-c053c01f410f/osd-block-e56ecc53-826d-40b0-a647-5f1a1fc8800e lrwxrwxrwx 1 ceph ceph 14 sty 28 17:59 block.db -> /dev/nvme0n1p3 ceph daemon osd.X perf dump | jq '.bluefs' { "gift_bytes": 0, "reclaim_bytes": 0, "db_total_bytes": 42949664768, --> 39,99GB (40GB partition created) "db_used_bytes": 1452269568, --> 1,35GB "wal_total_bytes": 0, "wal_used_bytes": 0, ... # OR "db_total_bytes": 4294959104, --> 3,9GB (4GB partition) "db_used_bytes": 66052096, --> ceph device ls
And restore backfilling:
ceph osd unset nobackfill
Check benefits:
- Observe better latency on OSD with NVM/SSD:
watch ceph osd perf
- check
iotop
output. Now[bstore_kv_sync]
should take less time.
Issues
auth: unable to find a keyring
It is not possible to create ceph OSD neither from WebUI nor cmdline:
pveceph osd create /dev/sdc
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap stderr: 2021-01-28T10:21:24.996+0100 7fd1a848f700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory 2021-01-28T10:21:24.996+0100 7fd1a848f700 -1 AuthRegistry(0x7fd1a0059030) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
- /etc/pve/ceph.conf
[client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring
ceph.conf Variables
- $cluster - cluster name. For proxmox it is
ceph
- $type - daemon process
mds
osd
mon
- $id - daemon or client indentifier. For
osd.0
it is0
- $host - hostname where the process is running
- $name - Expands to $type.$id. I.e:
osd.2
orclient.bootstrap
- $pid - Expands to daemon pid
SOLUTION:
cp /var/lib/ceph/bootstrap-osd/ceph.keyring /etc/pve/priv/ceph.client.bootstrap-osd.keyring
alternative to try: change ceph.conf
Unit -.mount is masked.
Running command: /usr/bin/systemctl start ceph-osd@2 stderr: Failed to start ceph-osd@2.service: Unit -.mount is masked. --> RuntimeError: command returned non-zero exit status: 1
It was caused by gparted
which wasn't correctly shutdown.
Solution:
systemctl --runtime unmask -- -.mount
To list runtime masked units:
ls -l /var/run/systemd/system | grep mount | grep '/dev/null' | cut -d ' ' -f 11
To unescape systemd unit names:
systemd-escape -u 'rpool-data-basevol\x2d800\x2ddisk\x2d0.mount'