Google Coral USB in LXC

to Frigate container

After power-up Google Coral is in boot mode:

Bus 002 Device 005: ID 1a6e:089a Global Unichip Corp. 

It needs software to run. So Proxmox host or LXC must upload firmware. After successful init Google Coral changes its USB id to:

Bus 002 Device 006: ID 18d1:9302 Google Inc. 

Frigate contains Coral firmware and can init it.

Coral:

• Frigate needs access to e.g.: /dev/bus/usb/002/005
• Most solution on Internet provides workarounds:
  • ugly and unsafe solution like:chmod 666 by udev and unprivileged LXC
  • still unsafe using lxc.idmap to give access from LXC to devices in video and render group (I don't want to give so wide access).
  • one time solution - like chown /dev/bus/usb/002/* before container start by LXC hook.
    • this solution cannot survive lots of scenarios (udev reload, usb reset, usb cable reconnect)

iGPU acceleration:

• Frigate needs access to /dev/dri/renderD128 - givin access to LXC simply works

iGPU performance monitors:

docker-compose.yml

services:
  frigate:
    privileged: true  # still in unpriv LXC so don't care
    devices:
      - /dev/bus/usb:/dev/bus/usb  # Passes the USB bus
      - /dev/dri/renderD128:/dev/dri/renderD128  # For intel hwaccel

/etc/pve/lxc/307.conf

features: fuse=1, nesting=1
unprivileged: 1
lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU
lxc.cgroup2.devices.allow: c 189:* rwm # USB Coral TPU
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render)

In Frigate's LXC shell:

groupadd -g 11000 lxc_gpu_shares
groupadd -g 11002 lxc_usb2_shares
usermod -aG lxc_gpu_shares,lxc_usb2_shares root

On Proxmox host:

/etc/udev/rules.d/71-coral.rules

SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", OWNER="100000", GROUP="111002"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", OWNER="100000", GROUP="111002"

/etc/udev/rules.d/99-igpu.rules

KERNEL=="renderD128", MODE="0664", OWNER="100000", GROUP="111000"

udevadm control --reload-rules && udevadm trigger

This error is “informational”. Rendering on /dev/dri/renderD128 works but container lacks the necessary permissions to access the performance monitoring unit. Frigate wants this to show GPU usage statistics.

Reproduce in LXC:

# apt install intel-gpu-tools
# intel_gpu_top
Failed to initialize PMU! (Permission denied)

Host Kernel prevents access to performance events subsystem for unprivileged users. Security level is controller by sysctl:

# sysctl kernel.perf_event_paranoid
kernel.perf_event_paranoid = 4

where values:

• -1 Allow use of (almost) all events by all users. Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
• >=0 Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN. Disallow raw tracepoint access by users without CAP_SYS_ADMIN
• >=1 Disallow CPU event access by users without CAP_SYS_ADMIN
• >=2 Disallow kernel profiling by users without CAP_SYS_ADMIN

sysctl --write kernel.perf_event_paranoid=NEWVALUE

I suppose this is not possible to use unpriv LXC and enable CAP_SYS_ADMIN (multiple trials with lxc.cap.drop and lxc.cap.keep). It works when

sysctl --write kernel.perf_event_paranoid=0

which is still safer (there were kernel vulnerabilities) than giving container CAP_SYS_ADMIN.

When Frigate is running and performance events are accessible in LXC, command

intel_gpu_top

show nicely formatted text statistics:

• Coral-in-LXC
• Frigate inside LXC on Proxmox [Google Coral USB, iGPU passthrough]
• Coral USB changing ID and Vendor #536