====== Google Coral USB in LXC ======

to Frigate container

===== Google Coral =====

After power-up Google Coral is in boot mode:

<code bash>
Bus 002 Device 005: ID 1a6e:089a Global Unichip Corp. 
</code>

It needs software to run. So Proxmox host or LXC must upload firmware.
After successful init Google Coral changes its USB id to:
<code bash>
Bus 002 Device 006: ID 18d1:9302 Google Inc. 
</code>

Frigate contains Coral firmware and can init it.

===== Issues to solve =====

Coral:
  * Frigate needs access to e.g.: ''/dev/bus/usb/002/005''
  * Most solution on Internet provides workarounds:
    * ugly and unsafe solution like:''chmod 666'' by udev and unprivileged LXC
    * still unsafe using ''lxc.idmap'' to give access from LXC to devices in ''video'' and ''render'' group (I don't want to give so wide access).
    * one time solution - like ''chown /dev/bus/usb/002/*'' before container start by LXC hook.
      * this solution cannot survive lots of scenarios (udev reload, usb reset, usb cable reconnect)

iGPU acceleration:
  * Frigate needs access to ''/dev/dri/renderD128'' - givin access to LXC simply works

iGPU performance monitors:


===== Setup =====

==== docker-compose ====

<file yaml docker-compose.yml>
services:
  frigate:
    privileged: true  # still in unpriv LXC so don't care
    devices:
      - /dev/bus/usb:/dev/bus/usb  # Passes the USB bus
      - /dev/dri/renderD128:/dev/dri/renderD128  # For intel hwaccel
</file>

==== LXC ====

<file ini /etc/pve/lxc/307.conf>
features: fuse=1, nesting=1
unprivileged: 1
lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU
lxc.cgroup2.devices.allow: c 189:* rwm # USB Coral TPU
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render)
</file>

In Frigate's LXC shell:
<code bash>
groupadd -g 11000 lxc_gpu_shares
groupadd -g 11002 lxc_usb2_shares
usermod -aG lxc_gpu_shares,lxc_usb2_shares root
</code>

==== Host ====

On Proxmox host:

<file ini /etc/udev/rules.d/71-coral.rules>
SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", OWNER="100000", GROUP="111002"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", OWNER="100000", GROUP="111002"
</file>

<file ini /etc/udev/rules.d/99-igpu.rules>
KERNEL=="renderD128", MODE="0664", OWNER="100000", GROUP="111000"
</file>

<code bash>
udevadm control --reload-rules && udevadm trigger
</code>


===== ERROR: Unable to poll intel GPU stats: Failed to initialize PMU! (Permission denied) =====

This error is "informational".
Rendering on ''/dev/dri/renderD128'' **works** but container lacks the necessary permissions to access the performance monitoring unit. 
Frigate wants this to show GPU usage statistics.

Reproduce in LXC:
<code bash>
# apt install intel-gpu-tools
# intel_gpu_top
Failed to initialize PMU! (Permission denied)
</code>

Host Kernel prevents access to performance events subsystem for unprivileged users. Security level is controller by sysctl:
<code bash>
# sysctl kernel.perf_event_paranoid
kernel.perf_event_paranoid = 4
</code>

where values:
  * ''-1'' Allow use of (almost) all events by all users. Ignore mlock limit after ''perf_event_mlock_kb'' without ''CAP_IPC_LOCK''
  * ''>=0'' Disallow ftrace function tracepoint by users without ''CAP_SYS_ADMIN''. Disallow raw tracepoint access by users without ''CAP_SYS_ADMIN''
  * ''>=1'' Disallow CPU event access by users without ''CAP_SYS_ADMIN''
  * ''>=2'' Disallow kernel profiling by users without ''CAP_SYS_ADMIN''

<code bash>
sysctl --write kernel.perf_event_paranoid=NEWVALUE
</code>

I suppose this is not possible to use unpriv LXC and enable ''CAP_SYS_ADMIN'' (multiple trials with ''lxc.cap.drop'' and ''lxc.cap.keep'').
It works when <code bash>sysctl --write kernel.perf_event_paranoid=0</code> which is still safer (there were kernel vulnerabilities) than giving container ''CAP_SYS_ADMIN''.

When Frigate is running and performance events are accessible in LXC, command <code bash>intel_gpu_top</code> show nicely formatted text statistics:

{{:vm:proxmox:lxc:pasted:20241208-144931.png}}

===== references =====

  * [[https://github.com/Bytelake/Coral-in-LXC|Coral-in-LXC]]
  * [[https://www.reddit.com/r/frigate_nvr/comments/1cr9akm/frigate_inside_lxc_on_proxmox_google_coral_usb/|Frigate inside LXC on Proxmox [Google Coral USB, iGPU passthrough] ]]
  * [[https://github.com/google-coral/edgetpu/issues/536|Coral USB changing ID and Vendor
#536]]