Update 6.2 to 6.3

Node is reachable from SSH but PVE services are not working correctly. Journal shows:

/etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.

Directory `/etc/pve` is empty. No cluster FS propagated.

systemctl start pve-cluster

Propagte Cluster FS but still errors in journal:

Nov 26 19:20:13 pve5 pmxcfs[31368]: [main] notice: unable to acquire pmxcfs lock - trying again
Nov 26 19:20:13 pve5 pmxcfs[31368]: [main] notice: unable to acquire pmxcfs lock - trying again

reboot solves problem :(

But another problem with CEPH appear (see below):

lis 26 20:07:01 pve3 kernel: libceph: mon4 (1)192.168.28.235:6789 socket closed (con state CONNECTING)
lis 26 20:07:02 pve3 kernel: libceph: mon4 (1)192.168.28.235:6789 socket closed (con state CONNECTING)
lis 26 20:07:03 pve3 kernel: libceph: mon4 (1)192.168.28.235:6789 socket closed (con state CONNECTING)
lis 26 20:07:08 pve3 kernel: libceph: mon0 (1)192.168.28.230:6789 socket closed (con state OPEN)
lis 26 20:07:13 pve3 kernel: libceph: mon2 (1)192.168.28.233:6789 socket error on write
lis 26 20:07:14 pve3 kernel: libceph: mon2 (1)192.168.28.233:6789 socket error on write
lis 26 20:07:15 pve3 kernel: libceph: mon2 (1)192.168.28.233:6789 socket error on write
lis 26 20:07:16 pve3 kernel: libceph: mon4 (1)192.168.28.235:6789 socket closed (con state CONNECTING)
lis 26 20:07:17 pve3 kernel: libceph: mon4 (1)192.168.28.235:6789 socket closed (con state CONNECTING)
lis 26 20:07:18 pve3 kernel: libceph: mon4 (1)192.168.28.235:6789 socket closed (con state CONNECTING)
lis 26 20:07:18 pve3 mount[7891]: mount error 110 = Connection timed out
lis 26 20:07:18 pve3 systemd[1]: mnt-pve-cephfs.mount: Mount process exited, code=exited, status=32/n/a
lis 26 20:07:18 pve3 systemd[1]: mnt-pve-cephfs.mount: Failed with result 'exit-code'.
lis 26 20:07:18 pve3 systemd[1]: Failed to mount /mnt/pve/cephfs.
lis 26 20:07:18 pve3 pvestatd[2923]: mount error: See "systemctl status mnt-pve-cephfs.mount" and "journalctl -xe" for details.
lis 26 20:07:18 pve3 kernel: ceph: No mds server is up or the cluster is laggy
lis 26 20:07:24 pve3 pvestatd[2923]: got timeout
lis 26 20:07:24 pve3 pvestatd[2923]: status update time (99.487 seconds)

systemctl status ceph-{mon,mgr}@pve5
● ceph-mon@pve5.service - Ceph cluster monitor daemon
   Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/ceph-mon@.service.d
           └─ceph-after-pve-cluster.conf
   Active: inactive (dead)
 
● ceph-mgr@pve5.service - Ceph cluster manager daemon
   Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/ceph-mgr@.service.d
           └─ceph-after-pve-cluster.conf
   Active: inactive (dead)

systemctl restart ceph-mon.target
systemctl restart ceph-mgr.target
systemctl restart ceph-mds.target

• No GUI nor SSH after upgrade 6.1 -> 6.3 - needs manual restart of services
• Upgrade to ProxMox 6.3 failure

Yesterday evening (26.11.2020) Proxmox releases some fixes to CEPH packages: