BTRFS on multiple devices

Lets assume that we have 2nd disc for other than system stuff, but we want to use it as backup solution for system disc. System should be able to boot if one disc fails.

“By default, metadata will be mirrored across two devices and data will be striped across all of the devices present. If only one device is present, metadata will be duplicated on that one device. ”

Make second disc bootable: grub-install /dev/sdb

Now we have running and bootable system from BTRFS at /dev/sda1 Add second device to / volume. Device is unformatted, only partitioned.

btrfs device add /dev/sdb1 /

# btrfs filesystem show Label: none uuid: 2a3f41d6-571c-4ed5-91f1-ca5f5c1d3b5e

  Total devices 2 FS bytes used 1.37GiB
  devid    1 size 4.08GiB used 4.08GiB path /dev/sda1
  devid    2 size 5.00GiB used 0.00B path /dev/sdb1

Btrfs v3.17

Replicate data to second volume. General command:

btrfs balance start -d <filter> -m <filter> /

Starts balance action on data (-d) and metadata (-m). For details please look into man btrfs balance Full command:

btrfs balance start -dconvert=<profile> -mconvert=<profile> /

Possible profiles are: raid0, raid1, raid10, raid5, raid6, dup, single

single - is to put single copy of data (not stripping) and duplicated metadata

dup - duplicate metadata - but it is not guaranteed that metadata is duplicated on separate devices

“For now, RAID-1 means 'one copy of what's important exists on two of the drives in the array no matter how many drives there may be in it'.”

It is possible to crate raid1 profile in degraded state, to add real 2nd hard disc later. As 2nd disc use small loop device and after creation remove loop device and remount in degraded mode.

# btrfs fi df / Data, single: total=2.69GiB, used=948.35MiB System, single: total=32.00MiB, used=4.00KiB Metadata, single: total=1.36GiB, used=456.89MiB GlobalReserve, single: total=32.00MiB, used=0.00B

btrfs balance start -dconvert=dup -mconvert=dup /

ERROR in dmesg: [ 2687.757913] BTRFS error (device sda1): unable to start balance with target data profile 32

Update to kernel 4.0.2 and btrfs-tools v4 doesn't help. Probably dup (profile 32) cannot be used for data.

It works: btrfs balance start -dconvert=raid1 -mconvert=raid1 /

Now shutdown system and disconnect 2nd drive

Boot fail BTRFS: failed to read the system array on sda1 BTRFS: open_cree failed

And initramfs console appear.

mkdir /mnt mount /dev/sda1 /mnt

the same error

mount /dev/sda1 /mnt -odegraded

WORKS!

To boot automatically kernel with degraded btrfs, please add “rootflags=degraded” to kernel commandline. /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=“quiet rootflags=degraded” Add to grub and fstab. Then update grub: update-grub grub-install /dev/sda grub-install /dev/sdb

Remove missing device : btrfs device delete missing /

Add new disc: btrfs device add /dev/sdb /

Remove missing device - word 'missing' is special device name for this command btrfs device delete missing /

ERROR: error removing the device 'missing' - No space left on device

There is command btrfs replace

btrfs replace start -r 2 /dev/sdb1 /

Because device is removed from system followin parameters are used: -r - use raid redundancy mechanism 2 - dev id /dev/sdb1 - new free device (not assigned to any btrfs) / - mount point

Device needs to be same or greater size, but I made ideal clone of /dev/sda disc using sfdisk and above command complains that: BTRFS error (device sda1): target device is smaller than source device!

So I created bigger device and replace process starts. With some errors in dmesg:

[ 3185.709933] BTRFS: checksum error at logical 6796906496 on dev /dev/sda1, sector 5898312, root 5, inode 163698, offset 244748288, length 4096, links 1 (path: zero) [ 3185.711283] BTRFS: bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 [ 3185.712447] BTRFS: unable to fixup (regular) error at logical 6796906496 on dev /dev/sda1 [ 3190.557027] scrub_handle_errored_block: 4678 callbacks suppressed

and finally

[ 3305.403916] BTRFS: dev_replace from <missing disk> (devid 2) to /dev/sdb1 finished

btrfs replace status / Started on 15.Jul 22:38:55, finished on 15.Jul 22:40:55, 0 write errs, 0 uncorr. read errs

Grub installed to /dev/sda /dev/sdb Test - remove sda,

In grub menu - additional option to boot debian 8 from sdb1 shows. Booting from sdb1 stops an initramfs, because option to mount degraded is missing.

Booting from original options, stuck on “A start job is running for dev-disk-by\x2duuid-fb8d9… 55s / 1min 30s” After timeout 1:30, system boots into emergency mode. “Give root password for maintenance” “(or type Control+D to continue)

journalctl -a Jul 16 22:22:17 debian kernel: work still pending Jul 16 22:23:43 debian systemd[1]: Job dev-disk-by\x2duuid-fb8d97df\x2d625b\x2d440d\x2db63b\x2dcb529586df24.device/start timed out. Jul 16 22:23:43 debian systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-fb8d97df\x2d625b\x2d440d\x2db63b\x2dcb529586df24.device. Jul 16 22:23:43 debian systemd[1]: Dependency failed for /home. Jul 16 22:23:43 debian systemd[1]: Dependency failed for Local File Systems. Jul 16 22:23:43 debian systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/fb8d97df-625b-440d-b63b-cb529586df24. Jul 16 22:23:43 debian systemd-journal[177]: Runtime journal is using 4.0M (max allowed 9.8M, trying to leave 14.8M free of 94.4M available → current limit 9.

/lib/systemd/system/local-fs.target

This is because /home directory is missing (it was located on /dev/sda2) but whole /dev/sda removed. This bad behaviour, because I want system to be bootable even without additional disc.

Network in initramfs with SSH to repair remotely Network with SSH in maintenance mode

Run dropbear on alternative SSH port apt-get install dropbear /etc/default/dropbear