====== Proxmox ======

  * [[https://pve.proxmox.com/pve-docs/pve-admin-guide.html|Proxmox VE Administration Guide]]
  * [[https://pve.proxmox.com/wiki/Main_Page|Wiki]]
  * [[https://www.proxmox.com/en/training/video-tutorials|Video Tutorials]]

===== Upgrading =====

Always upgrade with ''dist-upgrade'' !
<code bash>apt update && apt dist-upgrade</code>


===== Installation =====

  * BIOS:
      * Enable VT
      * Enable IOMMU
  * Choose ZFS
    * Leave some free space on HDD for swap partition
      * Click "Advanced Options" and manually set "hdsize"
      * Installer 6.1-2 BUG: On system with 2x 2TB HDDs, Proxmox installer on ZFS RAID1 propose only ''hdsize=953MB''. This issue gone with Proxmox 6.2.
    * Do not create swap on ZFS zvol!
  * reboot
  * manually configure  VLAN network from console [[https://pve.proxmox.com/wiki/Network_Configuration#_vlan_802_1q|VLAN 802.1Q]]
    * check if it is still necessary. Perhaps enabling ''VLAN aware birdge'' from GUI is enough
  * https://ip_address:8006

==== issues ====

=== initramfs ===
Proxmox installed on ZFS RADI1. After reboot, Proxmox boots into initramfs.
From initramfs console:
<code bash>
zfs list
Try running '/sbin/modprobe zfs' as root to load them.
</code>
REASON: no root cmdline, because UEFI boot was used instead of grub loader.
SOLUTION: force legacy boot in BIOS
OR TRY: 
<code bash>
echo -n " rootdelay=5" >> /etc/kernel/cmdline
pve-efiboot-tool refresh
</code>

=== initramfs ===

REASON: ZFS invoked too soon: [[https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox]]
<code bash>
modprobe zfs
zpool import -N -f rpool
exit
</code>

  
==== post-installation tips ====
  * set MAC address prefix: `Datacenter` -> `Options`. Locally Administered Address Ranges are: <code>x2-xx-xx-xx-xx-xx
x6-xx-xx-xx-xx-xx
xA-xx-xx-xx-xx-xx
xE-xx-xx-xx-xx-xx
</code>
  * Speedup migrations to set unencrypted network traffic (in secured networks!): <file /etc/pve/datacenter.cfg>
migration: network=first_cluster_host_ip/22,type=insecure</file>
  * disable subscription repo:\\ <code bash>sed -i.bak 's|deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise|\#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise|' /etc/apt/sources.list.d/pve-enterprise.list</code>
  * add no subscription repo: <code bash>echo 'deb http://download.proxmox.com/debian/pve buster pve-no-subscription' > /etc/apt/sources.list.d/pve-no-sub.list</code>
  * <code bash>apt-get update</code>
  * <code bash>apt-get upgrade</code>
  * <code bash>apt-get install dropbear busybox-static --no-install-recommends</code>
  * Update grub config: <file | /etc/default/grub>
GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=15 mitigations=off panic=30 nomodeset textonly video=vesafb:off video=efifb:off"
</file><code bash>update-grub</code>
  * SMART disk monitoring\\ <file | /etc/smartd.conf>
#                             Short   1AM  Long  Sat    2AM
DEVICESCAN -a -o on -S on -s (S/../.././01|L/../../6/02) -m root
</file><code bash>systemctl restart smartmontools.service</code>
    * More here: [[https://pve.proxmox.com/wiki/Disk_Health_Email_Alerts|Disk Health Email Alerts]]
  * Activate ZFS e-mail notification [[https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs]]
    * <code bash>systemctl enable --now zfs-zed</code>
  * Use own internal CA signed cert for Web UI: [[https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_certificate_management|Certificate Management]]
    * For clustering Proxmox is using own self signed certs! Do not replace them!
  * Postfix relay:<code bash>apt-get install postfix bsd-mailx
wget https://raw.githubusercontent.com/Tontonjo/proxmox/master/ez_proxmox_mail_configurator.sh
bash ez_proxmox_mail_configurator.sh
</code>
  * Assign admin permissions: [[https://pve.proxmox.com/wiki/User_Management#_real_world_examples]]
  * For HDDs storage, limit Proxmox operation speed to 50000KiB: <code bash>pvesm set local --bwlimit default=50000</code>
    * to remove limit: <code bash>pvesm set local --delete bwlimit</code>
  * Enable IOMMU 
    * in `GRUB_CMDLINE_LINUX_DEFAULT` and add kernel parameter `intel_iommu=on` or `amd_iommu=on`
    * <file /etc/modules>
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
</file>