R4.1 build 16: Netronome build 73: virtio-forwarder failed to come up

[Ankit Jain]

I've taken netronome build 73 , but see that virtio-forwarder is not able to come up as shown below

root@nodel9:/home/ubuntu# systemctl status virtio-forwarder
● virtio-forwarder.service - Virtio-forwarder
   Loaded: loaded (/lib/systemd/system/virtio-forwarder.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-10-22 18:54:05 UTC; 1h 31min ago
 Main PID: 4393 (code=exited, status=255)

Oct 22 18:52:59 nodel9 virtio-forwarder[4396]: DPDK version: DPDK 17.11.4
Oct 22 18:52:59 nodel9 vio4wd-start.sh[4393]: EAL: No free hugepages reported in hugepages-1048576kB
Oct 22 18:52:59 nodel9 vio4wd-start.sh[4393]: EAL: VFIO support initialized
Oct 22 18:52:59 nodel9 virtio-forwarder[4396]: EAL: VFIO support initialized
Oct 22 18:54:04 nodel9 vio4wd-start.sh[4393]: EAL: Failed to remap 2 MB pages
Oct 22 18:54:04 nodel9 vio4wd-start.sh[4393]: EAL: FATAL: Cannot init memory
Oct 22 18:54:04 nodel9 vio4wd-start.sh[4393]: EAL: Cannot init memory
Oct 22 18:54:04 nodel9 systemd[1]: virtio-forwarder.service: Main process exited, code=exited, status=255/n/a
Oct 22 18:54:05 nodel9 systemd[1]: virtio-forwarder.service: Unit entered failed state.
Oct 22 18:54:05 nodel9 systemd[1]: virtio-forwarder.service: Failed with result 'exit-code'.
root@nodel9:/home/ubuntu# exit
exit

[Frik Botha (fjbotha)]

This issue is a result of a memory setting which is configured too low by default. 'max_map_count' has to be increased to > 2x the number of 2M pages. For example,
echo 140000 > /proc/sys/vm/max_map_count
works for 65536 2M pages. virtio-forwarder starts fine with this change in place. Startup time is proportional to the number of pages, so expect a slow start with the above config.

[Ankit Jain (ankitja)]

Frik had helped me out to debug the problem, the reason is as he mentioned above.
This was due to the increase in the number of huge pages. The problem should also be solved by reducing the huge pages count by 1 and redeployment should also work.

[Sudheendra Rao (sudheendra-k)]

The problem was due to the limit to allow configuring number of huge pages. The number of huge pages should be configured less than 65535. If configured more than 65534, max_map_count needs to be modified.

The work around is:
1. Stop the virtio-forwarder using the command "systemctl stop virtio-forwarder"
2. Update the file using the command "echo 140000 > /proc/sys/vm/max_map_count"
3. start the virtio-forwarder "systemctl stop virtio-forwarder"