BTRFS as root (LVM and raid)

To setup BTRFS on 2 disc, first setup complete system for one disc, then connect 2nd disc and:

1. clone partition layout to 2nd disc
2. add 2nd disc to btrfs
3. rebalance btrfs
4. install grub on 2nd disc
5. configure system to start with degraded disc, include btrfs tools in initramfs

1. /dev/sdb
   1. 1MB free space + alignment
   2. /dev/sdb1 LVM
      1. /dev/disc2/swap2 (4GB)
      2. /dev/disc2/btrfs2 (80GB)
      3. /dev/disc2/pool2 (rest)

NOTE: Compare exact bytes size of 1st and 2nd disc, and adapt partition sizes to smaller disc! http://serverfault.com/questions/279571/lvm-dangers-and-caveats

Grub on GPT Grub on GPT requires some reserved space to embed itself (core.img) on disc. Otherwise error will be shown:

grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
grub-install: error: embedding is not possible, but this is required for RAID and LVM install.

Small 2MiB partition with flag bios_grub is enough. Partition can be located everywhere on disc within 2TiB.

apt-get install parted lvm2
 
parted /dev/sdb
mklabel gpt
mkpart primary ext2 0 2MiB
set 1 bios_grub on
mkpart primary ext2 3MiB -1MiB
set 2 lvm on

pvcreate /dev/sdb2
lvmdiscscan
 
vgcreate disc2 /dev/sdb2
lvcreate -L 4GB -n swap2 disc2
lvcreate -L 80GB -n btrfs2 disc2
lvcreate -l 100%FREE -n pool2 disc2

NOTE: Create subvolumes for things which shouldn't be included in snapshot of parent volume. Like some /var/../cache, etc

mkfs.btrfs /dev/disc2/btrfs2
mount -o noatime,compress=lzo,defaults /dev/disc2/btrfs2 /mnt
cd /mnt
btrfs subvolume create @
btrfs subvolume create @/@root
btrfs subvolume create @/@home
btrfs subvolume create @/@var
btrfs subvolume create @/@var/log
btrfs subvolume create @snapshots
 
btrfs subvolume list /mnt
btrfs subvolume set-default 259 /mnt # set to @/@root id
cd ..
umount /mnt
 
mkdir /mnt/{home,var,root}
mount -o noatime,compress=lzo,defaults,subvol=@/@root /dev/disc2/btrfs2 /mnt/root
mount -o noatime,compress=lzo,defaults,subvol=@/@home /dev/disc2/btrfs2 /mnt/home
mount -o noatime,compress=lzo,defaults,subvol=@/@var /dev/disc2/btrfs2 /mnt/var
 
# mount old disc in /mnt/sda1, etc
 
cp -ax /mnt/sda1/. /mnt/root
# or
# rsync -avxHAXP --numeric-ids /mnt/sda1/ /mnt/root/
 
cat /proc/mounts >> /mnt/root/etc/fstab
# edit /mnt/root/etc/fstab

Install grub loader

grub-install --recheck --root-directory=/mnt/root /dev/sdb

or

mount -t proc none /mnt/root/proc
mount -o bind /dev /mnt/root/dev
mount -t sysfs sys /mnt/root/sys
mount -o bind /mnt/var /mnt/root/var
 
chroot /mnt/root /bin/bash
grub-install --recheck /dev/sdb
update-grub
exit

umount /mnt/root/{proc,sys,dev,var} umount /mnt/{var,home,root} sync reboot </code>

apt-get install cryptsetup

Fill with random data https://niziak.spox.org/wiki/linux:fs:luks#fill_with_random_data

cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 256 --hash sha256 --verify-passphrase --use-random /dev/disc2/pool2
cryptsetup luksOpen /dev/disc2/pool2 cryptpool
mkfs.btrfs /dev/mapper/cryptpool
mount -o noatime,compress=no /dev/mapper/cryptpool /mnt

Create subvolumes:

cd /mnt
btrfs subvolume create @pool
...

Copy data to /mnt Add fstab entry

/etc/fstab

/dev/mapper/cryptpool	/POOL		btrfs	noauto,nofail,x-systemd.device-timeout=60,noatime,defaults,compress=no,subvol=@pool	0	0

Copy partition layouts from sdb to sda drive:

sudo sfdisk -d /dev/sdb > parts
sudo sfdisk  /dev/sda < parts

Recreate LVM layout.

Add 2nd disc to btrfs:

btrfs device add /dev/disc1/btrfs1 / 
btrfs device usage /
btrfs balance start -v -dconvert=raid1 -mconvert=raid1 /
btrfs device usage /

Make sure that data on both disc are the same. Especially system data are on boths disc in RAID1 mode.

Let system boot without one disc:

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="rootflags=degraded"

update-grub

Install grub

sudo grub-install /dev/sdb
sudo grub-install /dev/sdc
reboot

PROBLEM grub-pc 2.02~beta2-22+deb8u1 0 incorrectly handle multiple roots. Internal variable ${GRUB_DEVICE} which is used to generate root= parameter include newline character, so grub menuconfig entry is broken:

       linux<->/@/@root/boot/vmlinuz-4.8.0-0.bpo.2-amd64 root=/dev/mapper/disc2-btrfs2
/dev/mapper/disc1-btrfs1 ro rootflags=subvol=@/@root  rootflags=degraded

More here: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1238347 and https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1582811 SOLUTION Fixed in Debian Stretch (testing) version grub-pc_2.02~beta3-3 Now UUID is used multiple roots. UUID of btrfs filesystem placed on 2 disc is the same:

/dev/mapper/disc2-btrfs2: UUID="52944cdd-f8d0-4798-bd1c-80539c45253d" UUID_SUB="54703ddb-5789-4cac-bfd0-691acadfa33c" TYPE="btrfs"
/dev/mapper/disc1-btrfs1: UUID="52944cdd-f8d0-4798-bd1c-80539c45253d" UUID_SUB="9bc49b38-c014-40e3-876e-08f6873293b8" TYPE="btrfs"

So, kernel cmdline parameter root=UUID=… is now correct.

PROBLEM Grub says:

grub error: no symbol table

SOLUTION Grub was not reinstalled after update

grub-install /dev/sda
grub-install /dev/sdb
update-grub

PROBLEM Grub loads kernel and initramfs correctly, but: Begin: Waiting for root file system … and Grub still boot into initramfs. After some while, initramfs shell is available.

ALERT! 
Dropping to shell!

Unfortunatelly initramfs cannot activate LVM volumes when kernel cmdline root=UUID=… is used.

lvm lvscan

To boot manually:

lvm vgchange -ay
mount /dev/disc2/btrfs2 /root -o device=/dev/disc1/btrfs1
exit

System should start - but once.

Problem is located in /usr/share/initramfs-tools/scripts/local-top/lvm2. Script check if specified root device needs to be activated by LVM. When UUID is used it is executing code:

/dev/*/*)
        # Could be /dev/VG/LV; use lvs to check
        if lvm lvs -- "$dev" >/dev/null 2>&1; then
                lvchange_activate "$dev"
        fi
        ;;

In result, command

lvm lvs -- disc/by-uuid/52944cdd-f8d0-4798-bd1c-80539c45253d<code>
is executed, resulting following output:
<code>"disc/by-uuid/52944cdd-f8d0-4798-bd1c-80539c45253d": Invalid path for Logical Volume.

Debian bugs about similar problem:

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612402
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741342

solution

Install lvm2 package version 2.02.168-1 (Debian Stretch). Previously was 2.02.111-2.2+deb8u1 (Debian Jessie).

update-initramfs -k all -u
btrfs scrub /

or force LVM activation: Can't find LVM root dropped back to initramfs

Also GRUB_DISABLE_OS_PROBER=true can be added.

/etc/mkinitcpio.conf

BINARIES="/bin/btrfs"

update-init -u -k all

Make sure, all system, metadata and data is balanced to RAID1 mode. Without this it cannot be possible to mount degraded btrfs in rw mode.

During RW operation in degraded mode, single system, metadata and data structures will be created on BTRFS so it will need rebalance after missing disc will be connected again.

error: disk 'lvmid/8vc3Xl-....' not found.
Entering rescue mode...
grub rescue>

GRUB 2 is unable to find the grub folder or its contents are missing/corrupted. The grub folder contains the GRUB 2 menu, modules and stored environmental data.

grub.cfg sets prefix variable to missing LVM (lvimd/<VG UUID/<LV UUID>), and then call to load module which fails:

prefix='(lvmid/8vc3Xl-a40o-ILx2-AoRf-9Z37-4h51-jq0OR5/Hr8CPf-dVqs-uIm1-pkfH-fJdI-14io-rwtx3l)/@/@root/boot/grub'
insmod gettext

Rescue shell is very limited. Try to start normal shell.

set prefix='(lvm/disc1-btrfs1)/@/@root/boot/grub'
insmod normal
normal

Keeping /boot/grub in multidevice BTRFS is stupid idea. Another problem comes when BTRFS is not clean, and cannot be used by GRUB to load own modules.

btrfs device add /dev/disc3/btrfs3 /
btrfs device delete missing /

btrfs filesystem show
btrfs replace start 6 /dev/disc3/btrfs3 /  # change 6 to your setup

Don't forget to restore correct data state:

btrfs scrub start /
btrfs balance start -v -dconvert=raid1 -mconvert=raid1 /

• Add your ssh public key to /etc/initramfs-tools/root/.ssh/authorized_keys
• Change MAC address of eth0 (for initramfs remote access)

/etc/udev/rules.d/75-mac-spoof.rule

ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="08:00:27:f9:3d:3e", RUN+="/sbin/ip link set dev %k address 08:00:27:f9:12:34"

Another option is to add local-top script like described here for IPv6 https://www.danrl.com/2015/10/21/debian-jessi-ssh-fde-unlock.html

• backup GPT header
• backup LVM header
• backup LUKS header