I have the same issue with Ubuntu 20.04 on arm64 (RaspberryPi 4 B). I have even tried to uninstall the apparmor* packages to ensure that its not an apparmor issue.

Primary DHCP configuration:
root@dexter:~# grep -E -v "^#|^$" /etc/dhcp/dhcpd-common.conf
ddns-update-style none;
option all-subnets-local true;
option domain-name "se.lemche.net";
option domain-name-servers 192.168.1.3, 192.168.1.4;
default-lease-time 600;
max-lease-time 7200;
authoritative;
subnet 192.168.1.0 netmask 255.255.255.0 {
  option subnet-mask 255.255.255.0;
  option routers 192.168.1.254;
  option domain-name "pc.se.lemche.net";
  option domain-search "pc.se.lemche.net","se.lemche.net","lemche.net";
  pool {
    failover peer "dhcp-failover";
    range 192.168.1.94 192.168.1.126;
  }
}
root@dexter:~# grep -E -v "^#|^$" /etc/dhcp/dhcpd.conf
server-name "dexter";
failover peer "dhcp-failover" {
  primary;
  address 192.168.1.3;
  port 520;
  peer address 192.168.1.4;
  peer port 520;
  max-response-delay 60;
  max-unacked-updates 10;
  load balance max seconds 3;
  mclt 3600;
  split 128;
}
include "/etc/dhcp/dhcpd-common.conf";
root@dexter:~# grep -E -v "^#|^$" /etc/default/isc-dhcp-server
OPTIONS="-4"
INTERFACESv4="inside"
INTERFACESv6=""
root@dexter:~# dpkg -l isc-dhcp-server
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version          Architecture Description
+++-===============-================-============-===================================================
ii  isc-dhcp-server 4.4.1-2.1ubuntu5 arm64        ISC DHCP server for automatic IP address assignment
root@dexter:~#

Secondary DHCP configuration:
root@didi:~# grep -E -v "^#|^$" /etc/dhcp/dhcpd-common.conf
ddns-update-style none;
option all-subnets-local true;
option domain-name "se.lemche.net";
option domain-name-servers 192.168.1.3, 192.168.1.4;
default-lease-time 600;
max-lease-time 7200;
authoritative;
subnet 192.168.1.0 netmask 255.255.255.0 {
  option subnet-mask 255.255.255.0;
  option routers 192.168.1.254;
  option domain-name "pc.se.lemche.net";
  option domain-search "pc.se.lemche.net","se.lemche.net","lemche.net";
  pool {
    failover peer "dhcp-failover";
    range 192.168.1.94 192.168.1.126;
  }
}
root@didi:~# grep -E -v "^#|^$" /etc/dhcp/dhcpd.conf
server-name "didi";
failover peer "dhcp-failover" {
  secondary;
  address 192.168.1.4;
  port 520;
  peer address 192.168.1.3;
  peer port 520;
  max-response-delay 60;
  max-unacked-updates 10;
  load balance max seconds 3;
}
include "/etc/dhcp/dhcpd-common.conf";
root@didi:~# grep -E -v "^#|^$" /etc/default/isc-dhcp-server
OPTIONS="-4"
INTERFACESv4="inside"
INTERFACESv6=""
root@didi:~# dpkg -l isc-dhcp-server
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version          Architecture Description
+++-===============-================-============-===================================================
ii  isc-dhcp-server 4.4.1-2.1ubuntu5 arm64        ISC DHCP server for automatic IP address assignment

Messages on my loghost when I first start the primary, and then the secondary:
May  3 12:23:59 dexter systemd[1]: Started ISC DHCP IPv4 server.
May  3 12:23:59 dexter dhcpd[1797]: Internet Systems Consortium DHCP Server 4.4.1
May  3 12:23:59 dexter sh[1797]: Internet Systems Consortium DHCP Server 4.4.1
May  3 12:23:59 dexter sh[1797]: Copyright 2004-2018 Internet Systems Consortium.
May  3 12:23:59 dexter sh[1797]: All rights reserved.
May  3 12:23:59 dexter sh[1797]: For info, please visit https://www.isc.org/software/dhcp/
May  3 12:23:59 dexter dhcpd[1797]: Copyright 2004-2018 Internet Systems Consortium.
May  3 12:23:59 dexter dhcpd[1797]: All rights reserved.
May  3 12:23:59 dexter dhcpd[1797]: For info, please visit https://www.isc.org/software/dhcp/
May  3 12:23:59 dexter dhcpd[1797]: Config file: /etc/dhcp/dhcpd.conf
May  3 12:23:59 dexter sh[1797]: Config file: /etc/dhcp/dhcpd.conf
May  3 12:23:59 dexter sh[1797]: Database file: /var/lib/dhcp/dhcpd.leases
May  3 12:23:59 dexter sh[1797]: PID file: /run/dhcp-server/dhcpd.pid
May  3 12:23:59 dexter dhcpd[1797]: Database file: /var/lib/dhcp/dhcpd.leases
May  3 12:23:59 dexter sh[1797]: Wrote 0 leases to leases file.
May  3 12:23:59 dexter dhcpd[1797]: PID file: /run/dhcp-server/dhcpd.pid
May  3 12:23:59 dexter dhcpd[1797]: Wrote 0 leases to leases file.
May  3 12:23:59 dexter dhcpd[1797]: Listening on LPF/inside/dc:a6:32:86:5f:84/192.168.1.0/24
May  3 12:23:59 dexter sh[1797]: Listening on LPF/inside/dc:a6:32:86:5f:84/192.168.1.0/24
May  3 12:23:59 dexter sh[1797]: Sending on   LPF/inside/dc:a6:32:86:5f:84/192.168.1.0/24
May  3 12:23:59 dexter sh[1797]: No subnet declaration for management (192.168.0.3).
May  3 12:23:59 dexter sh[1797]: ** Ignoring requests on management.  If this is not what
May  3 12:23:59 dexter sh[1797]:    you want, please write a subnet declaration
May  3 12:23:59 dexter sh[1797]:    in your dhcpd.conf file for the network segment
May  3 12:23:59 dexter sh[1797]:    to which interface management is attached. **
May  3 12:23:59 dexter sh[1797]: No subnet declaration for wlan0 (no IPv4 addresses).
May  3 12:23:59 dexter sh[1797]: ** Ignoring requests on wlan0.  If this is not what
May  3 12:23:59 dexter dhcpd[1797]: Sending on   LPF/inside/dc:a6:32:86:5f:84/192.168.1.0/24
May  3 12:23:59 dexter sh[1797]:    you want, please write a subnet declaration
May  3 12:23:59 dexter sh[1797]:    in your dhcpd.conf file for the network segment
May  3 12:23:59 dexter sh[1797]:    to which interface wlan0 is attached. **
May  3 12:23:59 dexter sh[1797]: No subnet declaration for eth0 (no IPv4 addresses).
May  3 12:23:59 dexter sh[1797]: ** Ignoring requests on eth0.  If this is not what
May  3 12:23:59 dexter sh[1797]:    you want, please write a subnet declaration
May  3 12:23:59 dexter sh[1797]:    in your dhcpd.conf file for the network segment
May  3 12:23:59 dexter sh[1797]:    to which interface eth0 is attached. **
May  3 12:23:59 dexter sh[1797]: Sending on   Socket/fallback/fallback-net
May  3 12:23:59 dexter dhcpd[1797]:
May  3 12:23:59 dexter dhcpd[1797]: No subnet declaration for management (192.168.0.3).
May  3 12:23:59 dexter dhcpd[1797]: ** Ignoring requests on management.  If this is not what
May  3 12:23:59 dexter dhcpd[1797]:    you want, please write a subnet declaration
May  3 12:23:59 dexter dhcpd[1797]:    in your dhcpd.conf file for the network segment
May  3 12:23:59 dexter dhcpd[1797]:    to which interface management is attached. **
May  3 12:23:59 dexter dhcpd[1797]:
May  3 12:23:59 dexter dhcpd[1797]:
May  3 12:23:59 dexter dhcpd[1797]: No subnet declaration for wlan0 (no IPv4 addresses).
May  3 12:23:59 dexter dhcpd[1797]: ** Ignoring requests on wlan0.  If this is not what
May  3 12:23:59 dexter dhcpd[1797]:    you want, please write a subnet declaration
May  3 12:23:59 dexter dhcpd[1797]:    in your dhcpd.conf file for the network segment
May  3 12:23:59 dexter dhcpd[1797]:    to which interface wlan0 is attached. **
May  3 12:23:59 dexter dhcpd[1797]:
May  3 12:23:59 dexter dhcpd[1797]:
May  3 12:23:59 dexter sh[1797]: failover peer dhcp-failover: I move from recover to startup
May  3 12:23:59 dexter dhcpd[1797]: No subnet declaration for eth0 (no IPv4 addresses).
May  3 12:23:59 dexter dhcpd[1797]: ** Ignoring requests on eth0.  If this is not what
May  3 12:23:59 dexter dhcpd[1797]:    you want, please write a subnet declaration
May  3 12:23:59 dexter dhcpd[1797]:    in your dhcpd.conf file for the network segment
May  3 12:23:59 dexter dhcpd[1797]:    to which interface eth0 is attached. **
May  3 12:23:59 dexter dhcpd[1797]:
May  3 12:23:59 dexter dhcpd[1797]: Sending on   Socket/fallback/fallback-net
May  3 12:23:59 dexter dhcpd[1797]: failover peer dhcp-failover: I move from recover to startup
May  3 12:23:59 dexter dhcpd[1797]: Server starting service.
May  3 12:24:01 didi systemd[1]: Started ISC DHCP IPv4 server.
May  3 12:24:01 didi dhcpd[2352]: Internet Systems Consortium DHCP Server 4.4.1
May  3 12:24:01 didi sh[2352]: Internet Systems Consortium DHCP Server 4.4.1
May  3 12:24:01 didi sh[2352]: Copyright 2004-2018 Internet Systems Consortium.
May  3 12:24:01 didi sh[2352]: All rights reserved.
May  3 12:24:01 didi sh[2352]: For info, please visit https://www.isc.org/software/dhcp/
May  3 12:24:01 didi dhcpd[2352]: Copyright 2004-2018 Internet Systems Consortium.
May  3 12:24:01 didi dhcpd[2352]: All rights reserved.
May  3 12:24:01 didi dhcpd[2352]: For info, please visit https://www.isc.org/software/dhcp/
May  3 12:24:01 didi dhcpd[2352]: Config file: /etc/dhcp/dhcpd.conf
May  3 12:24:01 didi dhcpd[2352]: Database file: /var/lib/dhcp/dhcpd.leases
May  3 12:24:01 didi sh[2352]: Config file: /etc/dhcp/dhcpd.conf
May  3 12:24:01 didi sh[2352]: Database file: /var/lib/dhcp/dhcpd.leases
May  3 12:24:01 didi sh[2352]: PID file: /run/dhcp-server/dhcpd.pid
May  3 12:24:01 didi dhcpd[2352]: PID file: /run/dhcp-server/dhcpd.pid
May  3 12:24:01 didi sh[2352]: Wrote 0 leases to leases file.
May  3 12:24:01 didi dhcpd[2352]: Wrote 0 leases to leases file.
May  3 12:24:01 didi dhcpd[2352]: Listening on LPF/inside/dc:a6:32:86:2e:dc/192.168.1.0/24
May  3 12:24:01 didi sh[2352]: Listening on LPF/inside/dc:a6:32:86:2e:dc/192.168.1.0/24
May  3 12:24:01 didi sh[2352]: Sending on   LPF/inside/dc:a6:32:86:2e:dc/192.168.1.0/24
May  3 12:24:01 didi sh[2352]: No subnet declaration for management (192.168.0.4).
May  3 12:24:01 didi sh[2352]: ** Ignoring requests on management.  If this is not what
May  3 12:24:01 didi sh[2352]:    you want, please write a subnet declaration
May  3 12:24:01 didi sh[2352]:    in your dhcpd.conf file for the network segment
May  3 12:24:01 didi sh[2352]:    to which interface management is attached. **
May  3 12:24:01 didi sh[2352]: No subnet declaration for wlan0 (no IPv4 addresses).
May  3 12:24:01 didi sh[2352]: ** Ignoring requests on wlan0.  If this is not what
May  3 12:24:01 didi sh[2352]:    you want, please write a subnet declaration
May  3 12:24:01 didi sh[2352]:    in your dhcpd.conf file for the network segment
May  3 12:24:01 didi sh[2352]:    to which interface wlan0 is attached. **
May  3 12:24:01 didi dhcpd[2352]: Sending on   LPF/inside/dc:a6:32:86:2e:dc/192.168.1.0/24
May  3 12:24:01 didi sh[2352]: No subnet declaration for eth0 (no IPv4 addresses).
May  3 12:24:01 didi sh[2352]: ** Ignoring requests on eth0.  If this is not what
May  3 12:24:01 didi sh[2352]:    you want, please write a subnet declaration
May  3 12:24:01 didi sh[2352]:    in your dhcpd.conf file for the network segment
May  3 12:24:01 didi sh[2352]:    to which interface eth0 is attached. **
May  3 12:24:01 didi sh[2352]: Sending on   Socket/fallback/fallback-net
May  3 12:24:01 didi dhcpd[2352]:
May  3 12:24:01 didi dhcpd[2352]: No subnet declaration for management (192.168.0.4).
May  3 12:24:01 didi dhcpd[2352]: ** Ignoring requests on management.  If this is not what
May  3 12:24:01 didi dhcpd[2352]:    you want, please write a subnet declaration
May  3 12:24:01 didi dhcpd[2352]:    in your dhcpd.conf file for the network segment
May  3 12:24:01 didi dhcpd[2352]:    to which interface management is attached. **
May  3 12:24:01 didi dhcpd[2352]:
May  3 12:24:01 didi dhcpd[2352]:
May  3 12:24:01 didi dhcpd[2352]: No subnet declaration for wlan0 (no IPv4 addresses).
May  3 12:24:01 didi dhcpd[2352]: ** Ignoring requests on wlan0.  If this is not what
May  3 12:24:01 didi dhcpd[2352]:    you want, please write a subnet declaration
May  3 12:24:01 didi dhcpd[2352]:    in your dhcpd.conf file for the network segment
May  3 12:24:01 didi dhcpd[2352]:    to which interface wlan0 is attached. **
May  3 12:24:01 didi dhcpd[2352]:
May  3 12:24:01 didi dhcpd[2352]:
May  3 12:24:01 didi dhcpd[2352]: No subnet declaration for eth0 (no IPv4 addresses).
May  3 12:24:01 didi dhcpd[2352]: ** Ignoring requests on eth0.  If this is not what
May  3 12:24:01 didi sh[2352]: failover peer dhcp-failover: I move from recover to startup
May  3 12:24:01 didi dhcpd[2352]:    you want, please write a subnet declaration
May  3 12:24:01 didi dhcpd[2352]:    in your dhcpd.conf file for the network segment
May  3 12:24:01 didi dhcpd[2352]:    to which interface eth0 is attached. **
May  3 12:24:01 didi dhcpd[2352]:
May  3 12:24:01 didi dhcpd[2352]: Sending on   Socket/fallback/fallback-net
May  3 12:24:01 didi dhcpd[2352]: failover peer dhcp-failover: I move from recover to startup
May  3 12:24:01 didi dhcpd[2352]: Server starting service.
May  3 12:24:01 didi dhcpd[2352]: failover peer dhcp-failover: peer moves from unknown-state to recover
May  3 12:24:01 dexter dhcpd[1797]: failover peer dhcp-failover: I move from startup to recover
May  3 12:24:01 dexter dhcpd[1797]: Sent update request all message to dhcp-failover
May  3 12:24:01 didi dhcpd[2352]: failover peer dhcp-failover: requesting full update from peer
May  3 12:24:01 didi dhcpd[2352]: failover peer dhcp-failover: I move from startup to recover
May  3 12:24:01 didi dhcpd[2352]: Sent update request all message to dhcp-failover
May  3 12:24:01 didi dhcpd[2352]: failover peer dhcp-failover: peer moves from recover to recover
May  3 12:24:01 didi dhcpd[2352]: failover peer dhcp-failover: requesting full update from peer
May  3 12:24:01 didi dhcpd[2352]: Sent update request all message to dhcp-failover
May  3 12:24:01 didi dhcpd[2352]: Update request all from dhcp-failover: sending update
May  3 12:24:01 dexter dhcpd[1797]: failover peer dhcp-failover: peer moves from recover to recover
May  3 12:24:01 dexter dhcpd[1797]: failover peer dhcp-failover: requesting full update from peer
May  3 12:24:01 dexter dhcpd[1797]: Sent update request all message to dhcp-failover
May  3 12:24:01 didi dhcpd[2352]: Received update request while old update still flying!  Silently discarding old request.
May  3 12:24:01 dexter dhcpd[1797]: Update request all from dhcp-failover: sending update
May  3 12:24:01 didi dhcpd[2352]: Update request all from dhcp-failover: sending update
May  3 12:24:01 dexter dhcpd[1797]: Received update request while old update still flying!  Silently discarding old request.
May  3 12:24:01 dexter dhcpd[1797]: Update request all from dhcp-failover: sending update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.94 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.95 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.96 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.97 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.98 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.99 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.100 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.101 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.102 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 didi dhcpd[2352]: bind update on 192.168.1.103 from dhcp-failover rejected: incoming update is less critical than outgoing update
May  3 12:24:01 dexter sh[1797]: ../../../../lib/isc/unix/socket.c:3361: INSIST(!sock->pending_send) failed, back trace
May  3 12:24:01 dexter sh[1797]: #0 0xffffa7aaaf64 in ??
May  3 12:24:01 dexter sh[1797]: #1 0xffffa7aaaea4 in ??
May  3 12:24:01 dexter sh[1797]: #2 0xffffa7ae3788 in ??
May  3 12:24:01 dexter sh[1797]: #3 0xffffa79014fc in ??
May  3 12:24:01 dexter sh[1797]: #4 0xffffa79fcf2c in ??
May  3 12:24:01 didi dhcpd[2352]: peer dhcp-failover: disconnected
May  3 12:24:01 dexter systemd[1]: isc-dhcp-server.service: Main process exited, code=killed, status=6/ABRT
May  3 12:24:01 dexter systemd[1]: isc-dhcp-server.service: Failed with result 'signal'.

I do see an apparmor message as this. But I see the exact same messages above with or without apparmor installed.

[  246.347047] Loading iSCSI transport class v2.0-870.
[ 1160.044720] audit: type=1400 audit(1588497999.807:30): apparmor="DENIED" operation="capable" profile="/usr/sbin/dhcpd" pid=2191 comm="dhcpd" capability=1  capname="dac_override"
[ 1171.065442] audit: type=1400 audit(1588498010.826:31): apparmor="DENIED" operation="capable" profile="/usr/sbin/dhcpd" pid=2195 comm="dhcpd" capability=1  capname="dac_override"
[ 1188.582870] audit: type=1400 audit(1588498028.346:32): apparmor="DENIED" operation="capable" profile="/usr/sbin/dhcpd" pid=2208 comm="dhcpd" capability=1  capname="dac_override"