====== Docker ======

===== does not support file handles =====
<code bash>
overlayfs: fs on '/var/lib/docker/overlay2/l/EDSZIPXO5E6HIZ5EIOYLLFCEW6' does not support file handles, falling back to xino=off.
</code>

[[https://forum.proxmox.com/threads/docker-lxc-unprivileged-container-on-proxmox-7-with-zfs.99796/page-2#post-451845]]

===== docker inside unprivileged LXC =====

Some hints:
  * Docker is recommended to be used inside VM.
  * ''overlay''/''overlay2'' is not possible on ZFS (as documented by docker).
  * In priviledged container it works with ''ZFS'' using ''AUFS''. See below.
      * From [[https://linuxcontainers.org/lxc/news/|LXC-NEWS]]: The aufs storage driver has been deprecated since LXC 2.1 and is now officially removed.
  * ZFS over ZFS is not possible (ZFS nesting)

From documentation: [[https://pve.proxmox.com/wiki/Linux_Container#pct_configuration]]

  * edit LXC container config <file | /etc/pve/local/lxc/contained_id.conf>
features:  keyctl=1,nesting=1
</file>
  * stop/start LXC container <code bash>docker run hello-world</code>

==== VFS FS is used by docker. ====


  The vfs backend is a very simple fallback that has no copy-on-write support. Each layer is just a separate directory. Creating a new layer based on another layer is done by making a deep copy of the base layer into a new directory.
  Since this backend doesn’t share diskspace use between layers, and since creating a new layer is a slow operation this is not a very practical backend. However, it still has its uses, for instance to verify other backends against, or if you need a super robust (if slow) backend that works everywhere.
  
<code bash>
docker info
...
 Server Version: 19.03.8
 Storage Driver: vfs
...
</code>

When restarted in privileged container:
NOTE: restarting in privileged container do mess with user permission. Make backup/clone before.

<code bash>
docker info
...
 Storage Driver: aufs
  Root Dir: /var/lib/docker/aufs
  Backing Filesystem: zfs
  Dirs: 0
  Dirperm1 Supported: true
...
</code>


==== Use RAW image for Docker ====

Edit existing ''local'' storage and add following content types:
  * ''Disk image''
  * ''Container''

Add mount point to LXC:
  * Resources --> Add --> Mount Point
    * Storage: ''local''
    * Backup: NO
    * Path: ''/var/lib/docker''
    * Mount options: ''noatime''

And new RAW disc will be created, with EXT4 FS. 
Docker will use ''overlay2'' driver with this FS.

==== Bind mount host btrfs subvolume ====

  * Only directory hierarchy under ''/mnt/bindmounts'' are allowed to be bind-mounterd inside LXC containers.
  * Host permission and ACL will be used. To play with permission to shared folder please read: [[https://pve.proxmox.com/wiki/Unprivileged_LXC_containers|Unprivileged LXC containers]]
  * [[https://pve.proxmox.com/wiki/Linux_Container#pct_settings|Container Settings]])
  * CONS: no disk size control inside guest. Possible to use btrfs subvolume quotas

Login to host using SSH node

<code bash>
btrfs subvol create /mnt/bindmounts
btrfs quota enable /mnt/bindmounts
btrfs subvol create /mnt/bindmounts/<cid>-docker
btrfs qgroup limit 50G /mnt/bindmounts/<cid>-docker

# Disable COW for performance:
chattr +C -f -R /mnt/bindmounts

# Give unprivilged container right to write
chown 100000.100000 /mnt/bindmounts/<cid>-docker
# setfacl -Rm user:100000:rwx,default:user:100000:rwx

pct set <cid> -mp0 /mnt/bindmounts/<cid>-docker,mp=/var/lib/docker
</code>

<code bash>
btrfs quota rescan /
btrfs qgroup show -pcre 
</code>


==== Enabling ZFS tools and access - no success ====

<code bash>
dockerd -D --storage-driver zfs
...
zfs command is not available: exec: "zfs": executable file not found in $PATH  storage-driver=zfs
...
apt-get install zfsutils-linux --no-install-recommends

dockerd -D --storage-driver zfs
...
DEBU[2020-05-06T16:37:51.502473451Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs
...
</code>

<file | /etc/pve/lxc/${container_id}.conf>
lxc.mount.entry: /dev/zfs dev/zfs none bind,create=file
</file>
Exposing ''/dev/zfs'' works. NOTE: it gives too wide permisions, like ''zfs list'' shows ALL info about host ZFS.
After this trick, Docker starts and detects ZFS without any additional configuration.
But usage is not possible. Docker cannot create additional subvolumes

<code bash>
~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Extracting [==================================================>]  2.529kB/2.529kB
docker: failed to register layer: exit status 1: "/sbin/zfs fs create -o mountpoint=legacy rpool/data/subvol-803-disk-0/007d22d205263c9f89d2e53ab75787430a20a1b1b3b0270acf4eb67459de09ed" => cannot create 'rpool/data/subvol-803-disk-0/007d22d205263c9f89d2e53ab75787430a20a1b1b3b0270acf4eb67459de09ed': permission denied
</code>

Try to add permission on host (note, not possible to use UID which not exists in /etc/passwd)
<code bash>
zfs allow -u 100000 create,destroy,mount rpool/data/subvol-803-disk-0
</code>

Doesn't help. Give up and switch to RAW EXT4 image.



==== Trying to run on ZFS - without success ====

Solution:
<code bash>
cp /etc/apparmor.d/lxc/lxc-default-with-nesting /etc/apparmor.d/lxc/lxc-default-with-nesting-docker
</code>

Edit new file and update ''profile'' name and add some mount permissions:

<file|/etc/apparmor.d/lxc/lxc-default-with-nesting-docker>
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-nesting-docker flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>
  #include <abstractions/lxc/start-container>

  deny /dev/.lxc/proc/** rw,
  deny /dev/.lxc/sys/** rw,
  mount fstype=proc -> /var/cache/lxc/**,
  mount fstype=sysfs -> /var/cache/lxc/**,
  mount options=(rw,bind),
  mount fstype=cgroup -> /sys/fs/cgroup/**,
  mount fstype=cgroup2 -> /sys/fs/cgroup/**,
  mount fstype=aufs,
  mount fstype=overlay,
}
</file>

<code bash>systemctl reload apparmor</code>

Edit ''/etc/pve/lxc/${container_id}.conf'' and append this line:
<file | /etc/pve/lxc/${container_id}.conf>
lxc.apparmor.profile: lxc-container-default-with-nesting-docker
</file>

=== Disabling apparmor ===

<file | /etc/pve/lxc/${container_id}.conf>
lxc.apparmor.profile = unconfined
</file>

<code bash>systemctl reload apparmor</code>



=== Forcing ''aufs'' ===

Error: AUFS cannot be used in non-init user namespace

=== Forcing ''overlay2'' ===

On host log:
<code>
kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support244358035/upper' not supported as upperdir
kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support445538983/upper' not supported as upperdir
</code>
There are some requirements to use overlayfs. It is not possible to use it over ''ZFS''.