meta data for this page
This is an old revision of the document!
BTRFS on multiple devices
Prepare 2nd disc
Lets assume that we have 2nd disc for other than system stuff, but we want to use it as backup solution for system disc. System should be able to boot if one disc fails.
By default, metadata will be mirrored across two devices and data will be striped across all of the devices present.
If only one device is present, metadata will be duplicated on that one device.
Copy partition layouts from sdb to sda drive:
sudo sfdisk -d /dev/sdb > parts sudo sfdisk /dev/sda < parts
Recreate LVM layout if needed.
btrfs 2nd disk
Add 2nd disc to btrfs:
btrfs device add /dev/disc1/btrfs1 / btrfs device usage / btrfs balance start -v -dconvert=raid1 -mconvert=raid1 / btrfs device usage /
Make sure that data on both disc are the same. Especially system data are on both disc in RAID1 mode.
# btrfs device usage / /dev/sda3, ID: 1 Device size: 27.72GiB Device slack: 0.00B Data,RAID1: 18.93GiB Metadata,RAID1: 2.00GiB System,RAID1: 32.00MiB Unallocated: 6.75GiB /dev/sdc5, ID: 2 Device size: 49.40GiB Device slack: 0.00B Data,RAID1: 18.93GiB Metadata,RAID1: 2.00GiB System,RAID1: 32.00MiB Unallocated: 28.44GiB
Balance command
btrfs balance start -d <filter> -m <filter> /
Starts balance action on data (-d) and metadata (-m). For details please look into man btrfs balance Full command:
btrfs balance start -dconvert=<profile> -mconvert=<profile> /
Possible profiles are: raid0, raid1, raid10, raid5, raid6, dup, single
- single - is to put single copy of data (not stripping) and duplicated metadata
- dup - duplicate metadata - but it is not guaranteed that metadata is duplicated on separate devices
For now, RAID-1 means 'one copy of what's important exists on two of the drives
in the array no matter how many drives there may be in it'.“
It is possible to create raid1 profile in degraded state, to add real 2nd hard disc later. As 2nd disc use small loop device and after creation remove loop device and remount in in degraded mode.
Error: unable to start balance
ERROR in dmesg:
[ 2687.757913] BTRFS error (device sda1): unable to start balance with target data profile 32
Update to kernel 4.0.2 and btrfs-tools v4 doesn't help. Probably dup (profile 32) cannot be used for data.
It works:
btrfs balance start -dconvert=raid1 -mconvert=raid1 /
bootloader
To boot automatically kernel with degraded btrfs, please add “rootflags=degraded” to kernel commandline.
- /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet rootflags=degraded"
Add to grub and fstab. Then update grub:
update-grub grub-install /dev/sda grub-install /dev/sdb reboot
Boot issues
No degraded option
Boot fail BTRFS: failed to read the system array on sda1 BTRFS: open_cree failed
And initramfs console appear.
mkdir /mnt mount /dev/sda1 /mnt
the same error
mount /dev/sda1 /mnt -o degraded
WORKS!
Grub
PROBLEM
grub-pc
2.02~beta2-22+deb8u1 0 incorrectly handle multiple roots. Internal variable ${GRUB_DEVICE} which is used to generate root=
parameter include newline character, so grub menuconfig entry is broken:
linux<->/@/@root/boot/vmlinuz-4.8.0-0.bpo.2-amd64 root=/dev/mapper/disc2-btrfs2 /dev/mapper/disc1-btrfs1 ro rootflags=subvol=@/@root rootflags=degraded
More here: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1238347 and https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1582811 SOLUTION Fixed in Debian Stretch (testing) version grub-pc_2.02~beta3-3 Now UUID is used multiple roots. UUID of btrfs filesystem placed on 2 disc is the same:
/dev/mapper/disc2-btrfs2: UUID="52944cdd-f8d0-4798-bd1c-80539c45253d" UUID_SUB="54703ddb-5789-4cac-bfd0-691acadfa33c" TYPE="btrfs" /dev/mapper/disc1-btrfs1: UUID="52944cdd-f8d0-4798-bd1c-80539c45253d" UUID_SUB="9bc49b38-c014-40e3-876e-08f6873293b8" TYPE="btrfs"
So, kernel cmdline parameter root=UUID=…
is now correct.
no symbol table
PROBLEM Grub says:
grub error: no symbol table
SOLUTION Grub was not reinstalled after update
grub-install /dev/sda grub-install /dev/sdb update-grub
ALERT! /dev/disk/by-uuid/xxxxxxxxx does not exist. Dropping to a shell
PROBLEM
Grub loads kernel and initramfs correctly, but:
Begin: Waiting for root file system …
and Grub still boot into initramfs.
After some while, initramfs shell is available.
ALERT! Dropping to shell!
Unfortunatelly initramfs cannot activate LVM volumes when kernel cmdline root=UUID=…
is used.
lvm lvscan
To boot manually:
lvm vgchange -ay mount /dev/disc2/btrfs2 /root -o device=/dev/disc1/btrfs1 exit
System should start - but once.
Problem is located in /usr/share/initramfs-tools/scripts/local-top/lvm2
.
Script check if specified root device needs to be activated by LVM.
When UUID is used it is executing code:
/dev/*/*) # Could be /dev/VG/LV; use lvs to check if lvm lvs -- "$dev" >/dev/null 2>&1; then lvchange_activate "$dev" fi ;;
In result, command
lvm lvs -- disc/by-uuid/52944cdd-f8d0-4798-bd1c-80539c45253d<code> is executed, resulting following output: <code>"disc/by-uuid/52944cdd-f8d0-4798-bd1c-80539c45253d": Invalid path for Logical Volume.
Debian bugs about similar problem:
solution
Install lvm2 package version 2.02.168-1 (Debian Stretch). Previously was 2.02.111-2.2+deb8u1 (Debian Jessie).
update-initramfs -k all -u btrfs scrub /
or force LVM activation: Can't find LVM root dropped back to initramfs
Also GRUB_DISABLE_OS_PROBER=true
can be added.
- /etc/mkinitcpio.conf
BINARIES="/bin/btrfs"
update-init -u -k all
Disc fail (removed)
Make sure, all system, metadata and data is balanced to RAID1 mode. Without this it cannot be possible to mount degraded btrfs in RW mode. During RW operation in degraded mode, single system, metadata and data structures will be created on BTRFS so it will need rebalance after missing disc will be connected again.
error: disk not found
error: disk 'lvmid/8vc3Xl-....' not found. Entering rescue mode... grub rescue>
GRUB 2 is unable to find the grub folder or its contents are missing/corrupted. The grub folder contains the GRUB 2 menu, modules and stored environmental data.
grub.cfg sets prefix variable to missing LVM (lvimd/<VG UUID/<LV UUID>), and then call to load module which fails:
prefix='(lvmid/8vc3Xl-a40o-ILx2-AoRf-9Z37-4h51-jq0OR5/Hr8CPf-dVqs-uIm1-pkfH-fJdI-14io-rwtx3l)/@/@root/boot/grub' insmod gettext
Rescue shell is very limited. Try to start normal
shell.
set prefix='(lvm/disc1-btrfs1)/@/@root/boot/grub' insmod normal normal
Keeping /boot/grub in multidevice BTRFS is stupid idea. Another problem comes when BTRFS is not clean, and cannot be used by GRUB to load own modules.
Replace bad disc
btrfs device add /dev/disc3/btrfs3 / btrfs device delete missing /
Note: Word 'missing' is special device name for this command
ERROR: error removing the device 'missing' - No space left on device
btrfs filesystem show btrfs replace start 6 /dev/disc3/btrfs3 / # change 6 to your setup
Don't forget to restore correct data state:
btrfs scrub start / btrfs balance start -v -dconvert=raid1 -mconvert=raid1 /
Fine tuning
- Run dropbear on alternative SSH port
apt-get install dropbear vi /etc/default/dropbear
- Add your ssh public key to
/etc/initramfs-tools/root/.ssh/authorized_keys
apt-get install busybox-static
- Change MAC address of eth0 (for initramfs remote access)
- /etc/udev/rules.d/75-mac-spoof.rule
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="08:00:27:f9:3d:3e", RUN+="/sbin/ip link set dev %k address 08:00:27:f9:12:34"
Another option is to add local-top
script like described here for IPv6 https://www.danrl.com/2015/10/21/debian-jessi-ssh-fde-unlock.html
Don't forget to propgate changes:
update-grub update-initramfs -k all -u
TODO
- backup GPT header
- backup LVM header
- backup LUKS header
??????????? ??????????? ??????????? ??????????? ???????????
Grub installed to /dev/sda /dev/sdb Test - remove sda,
In grub menu - additional option to boot debian 8 from sdb1 shows. Booting from sdb1 stops an initramfs, because option to mount degraded is missing.
Booting from original options, stuck on
"A start job is running for dev-disk-by\x2duuid-fb8d9... 55s / 1min 30s"
After timeout 1:30, system boots into emergency mode.
"Give root password for maintenance" "(or type Control+D to continue)
journalctl -a
Jul 16 22:22:17 debian kernel: work still pending Jul 16 22:23:43 debian systemd[1]: Job dev-disk-by\x2duuid-fb8d97df\x2d625b\x2d440d\x2db63b\x2dcb529586df24.device/start timed out. Jul 16 22:23:43 debian systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-fb8d97df\x2d625b\x2d440d\x2db63b\x2dcb529586df24.device. Jul 16 22:23:43 debian systemd[1]: Dependency failed for /home. Jul 16 22:23:43 debian systemd[1]: Dependency failed for Local File Systems. Jul 16 22:23:43 debian systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/fb8d97df-625b-440d-b63b-cb529586df24. Jul 16 22:23:43 debian systemd-journal[177]: Runtime journal is using 4.0M (max allowed 9.8M, trying to leave 14.8M free of 94.4M available → current limit 9.
/lib/systemd/system/local-fs.target
This is because /home directory is missing (it was located on /dev/sda2) but whole /dev/sda removed. This bad behaviour, because I want system to be bootable even without additional disc.