meta data for this page
This is an old revision of the document!
BTRFS as root (LVM and raid)
To setup BTRFS on 2 disc, first setup complete system for one disc, then connect 2nd disc and:
- clone partition layout to 2nd disc
- add 2nd disc to btrfs
- rebalance btrfs
- install grub on 2nd disc
- configure system to start with degraded disc, include btrfs tools in initramfs
/dev/sdb partitioning
- /dev/sdb
- 1MB free space + alignment
- /dev/sdb1 LVM
- /dev/disc2/swap2 (4GB)
- /dev/disc2/btrfs2 (80GB)
- /dev/disc2/pool2 (rest)
NOTE: Compare exact bytes size of 1st and 2nd disc, and adapt partition sizes to smaller disc! http://serverfault.com/questions/279571/lvm-dangers-and-caveats
Grub on GPT Grub on GPT requires some reserved space to embed itself (core.img) on disc. Otherwise error will be shown:
grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible. grub-install: error: embedding is not possible, but this is required for RAID and LVM install.
Small 2MiB partition with flag bios_grub
is enough. Partition can be located everywhere on disc within 2TiB.
apt-get install parted lvm2 parted /dev/sdb mklabel gpt mkpart primary ext2 0 2MiB set 1 bios_grub on mkpart primary ext2 3MiB -1MiB set 2 lvm on
pvcreate /dev/sdb2 lvmdiscscan vgcreate disc2 /dev/sdb2 lvcreate -L 4GB -n swap2 disc2 lvcreate -L 80GB -n btrfs2 disc2 lvcreate -l 100%FREE -n pool2 disc2
Btrfs setup
NOTE: Create subvolumes for things which shouldn't be included in snapshot of parent volume. Like some /var/../cache, etc
mkfs.btrfs /dev/disc2/btrfs2 mount -o noatime,compress=lzo,defaults /dev/disc2/btrfs2 /mnt cd /mnt btrfs subvolume create @ btrfs subvolume create @/@root btrfs subvolume create @/@home btrfs subvolume create @/@var btrfs subvolume create @/@var/log btrfs subvolume create @snapshots btrfs subvolume list /mnt btrfs subvolume set-default 259 /mnt # set to @/@root id cd .. umount /mnt mkdir /mnt/{home,var,root} mount -o noatime,compress=lzo,defaults,subvol=@/@root /dev/disc2/btrfs2 /mnt/root mount -o noatime,compress=lzo,defaults,subvol=@/@home /dev/disc2/btrfs2 /mnt/home mount -o noatime,compress=lzo,defaults,subvol=@/@var /dev/disc2/btrfs2 /mnt/var # mount old disc in /mnt/sda1, etc cp -ax /mnt/sda1/. /mnt/root # or # rsync -avxHAXP --numeric-ids /mnt/sda1/ /mnt/root/ cat /proc/mounts >> /mnt/root/etc/fstab # edit /mnt/root/etc/fstab
Install grub loader
grub-install --recheck --root-directory=/mnt/root /dev/sdb
or
mount -t proc none /mnt/root/proc mount -o bind /dev /mnt/root/dev mount -t sysfs sys /mnt/root/sys mount -o bind /mnt/var /mnt/root/var chroot /mnt/root /bin/bash grub-install --recheck /dev/sdb update-grub exit
umount /mnt/root/{proc,sys,dev,var} umount /mnt/{var,home,root} sync reboot </code>
Encrypted /dev/disc2/pool2 (btrfs)
apt-get install cryptsetup
Fill with random data https://niziak.spox.org/wiki/linux:fs:luks#fill_with_random_data
cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 256 --hash sha256 --verify-passphrase --use-random /dev/disc2/pool2 cryptsetup luksOpen /dev/disc2/pool2 cryptpool mkfs.btrfs /dev/mapper/cryptpool mount -o noatime,compress=no /dev/mapper/cryptpool /mnt
Create subvolumes:
cd /mnt btrfs subvolume create @pool ...
Copy data to /mnt
Add fstab entry
- /etc/fstab
/dev/mapper/cryptpool /POOL btrfs noauto,nofail,x-systemd.device-timeout=60,noatime,defaults,compress=no,subvol=@pool 0 0
Add 2nd disc
Copy partition layouts from sdb to sda drive:
sudo sfdisk -d /dev/sdb > parts sudo sfdisk /dev/sda < parts
Recreate LVM layout.
Add 2nd disc to btrfs:
btrfs device add /dev/disc1/btrfs1 / btrfs device usage / btrfs balance start -v -dconvert=raid1 -mconvert=raid1 / btrfs device usage /
Let system boot without one disc:
- /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="rootflags=degraded"
update-grub
Install grub
sudo grub-install /dev/sdb sudo grub-install /dev/sdc reboot
Issues
Grub
PROBLEM
grub-pc
2.02~beta2-22+deb8u1 0 incorrectly handle multiple roots. Internal variable ${GRUB_DEVICE} which is used to generate root=
parameter include newline character, so grub menuconfig entry is broken:
linux<->/@/@root/boot/vmlinuz-4.8.0-0.bpo.2-amd64 root=/dev/mapper/disc2-btrfs2 /dev/mapper/disc1-btrfs1 ro rootflags=subvol=@/@root rootflags=degraded
More here: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1238347 and https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1582811 SOLUTION Fixed in Debian Stretch (testing) version grub-pc_2.02~beta3-3 Now UUID is used multiple roots. UUID of btrfs filesystem placed on 2 disc is the same:
/dev/mapper/disc2-btrfs2: UUID="52944cdd-f8d0-4798-bd1c-80539c45253d" UUID_SUB="54703ddb-5789-4cac-bfd0-691acadfa33c" TYPE="btrfs" /dev/mapper/disc1-btrfs1: UUID="52944cdd-f8d0-4798-bd1c-80539c45253d" UUID_SUB="9bc49b38-c014-40e3-876e-08f6873293b8" TYPE="btrfs"
So, kernel cmdline parameter root=UUID=…
is now correct.
no symbol table
PROBLEM Grub says:
grub error: no symbol table
SOLUTION Grub was not reinstalled after update
grub-install /dev/sda grub-install /dev/sdb update-grub
ALERT! Dropping to shell!
PROBLEM
Grub loads kernel and initramfs correctly, but:
Begin: Waiting for root file system …
and Grub still boot into initramfs.
After some while, initramfs shell is available.
ALERT! Dropping to shell!
Unfortunatelly initramfs cannot activate LVM volumes when kernel cmdline root=UUID=…
is used.
lvm lvscan
To boot manually:
lvm vgchange -ay mount /dev/disc2/btrfs2 /root -o device=/dev/disc1/btrfs1 exit
System should start - but once.
Problem is located in /usr/share/initramfs-tools/scripts/local-top/lvm2
.
Script check if specified root device needs to be activated by LVM.
When UUID is used it is executing code:
/dev/*/*) # Could be /dev/VG/LV; use lvs to check if lvm lvs -- "$dev" >/dev/null 2>&1; then lvchange_activate "$dev" fi ;;
In result, command
lvm lvs -- disc/by-uuid/52944cdd-f8d0-4798-bd1c-80539c45253d<code> is executed, resulting following output: <code>"disc/by-uuid/52944cdd-f8d0-4798-bd1c-80539c45253d": Invalid path for Logical Volume.
Debian bugs about similar problem: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612402
solution Install lvm2 package version 2.02.168-1 (Debian Stretch). Previously was 2.02.111-2.2+deb8u1 (Debian Jessie).
update-initramfs -k all -u btrfs scrub /
Also GRUB_DISABLE_OS_PROBER=true
can be added.
- /etc/mkinitcpio.conf
BINARIES="/bin/btrfs"
update-init -u -k all
Replace bad disc
btrfs device add /dev/disc3/btrfs3 / btrfs device delete missing / btrfs fi balance /
btrfs filesystem show btrfs replace start 6 /dev/disc3/btrfs3 / # change 6 to your setup
TODO
- backup GPT header
- backup LVM header
- backup LUKS header