Multipath 0.8.8 reports High CPU usage when having Lun's using Multipath

[Ninoy Radhakrishnan]

When Multipath 0.8.8 to 0.9.3 is used with Ubuntu20.04 and especially with 22.04, showing High CPU usage for systemd and multipathd processes, making node unresponsive. Multipath 0.9.4 works just fine. But as there is no official multipath package for 0.9.4 with Ubuntu 20.04/22.04, need to compile.

The ask here is to have 0.9.4 available for Ubuntu 20.04 / 22.04. Currently its available for 23.x versions.

I was able to use the Mantic package repo and install with 22.04 and make it work.

Redhat fixed the same issue and below is the link,

Redhat Bug description: https://access.redhat.com/solutions/7059978

May be below is the code path having issue
Seems this impacting 0.8.8 till 0.9.3.
https://github.com/opensvc/multipath-tools/pull/31/commits/38851dcaafe31073655761228db7f09ffc5a9a58

[Mitchell Dzurick (mitchdz)]

Hi Ninoy,

Thank you for making this bug report and making Ubuntu better!

I would say getting multipath-tools version 0.9.4 back into Jammy/Focal is very unlikely due to the amount of changes and the risk of breaking other things in those stable releases.

However, if the commit you mentioned[0] is enough to fix the issue, that change that can make it into those releases (given proper regression testing). Although, I'm not sure right away if it applies cleanly to the code.

Would you like to go through the SRU process to get those changes in if you are familiar with it?

[0] - https://github.com/opensvc/multipath-tools/pull/31/commits/38851dcaafe31073655761228db7f09ffc5a9a58

[Mitchell Dzurick (mitchdz)]

Alternatively I can attempt to apply the upstream patch to our stable releases if you would be willing to help test out the updated package.

[Mitchell Dzurick (mitchdz)]

Also clarifying question - it is mentioned that focal (Ubuntu 20.04) needs this fix, however focal has version 0.8.3-1ubuntu2.3 in our archives (0.8.3-1ubuntu2.4 is in proposed currently).

This bug supposedly effects version 0.8.8 through 0.9.3, so therefore nothing is needed on Focal. Have you seen issues with high cpu usage on Focal too?

[Ninoy Radhakrishnan (nradhakrishnan)]

Hi Mitchell,

Responding to the last 3 emails you have send,

1. I am not sure about the SRU process
2. If you can apply the upstream patch to the stable release, yes, I can test it out. This will be great.
3. Ubuntu 20.04(focal) , yes agree the highest Multipath version is 0.8.3-1. The number of times, seen with that is less compared to Ubuntu 22.04(jammy with Multipath 0.8.8

Thanks,
Ninoy Radhakrishnan (he/him/his)
DevOps Solutions Architect
[phone]

> On May 21, 2024, at 1:20 PM, Mitchell Dzurick <email address hidden> wrote:
>
> Also clarifying question - it is mentioned that focal (Ubuntu 20.04)
> needs this fix, however focal has version 0.8.3-1ubuntu2.3 in our
> archives (0.8.3-1ubuntu2.4 is in proposed currently).
>
> This bug supposedly effects version 0.8.8 through 0.9.3, so therefore
> nothing is needed on Focal. Have you seen issues with high cpu usage on
> Focal too?
>
> ** Changed in: multipath-tools (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2066286
>
> Title:
> Multipath 0.8.8 reports High CPU usage when having Lun's using
> Multipath
>
> Status in multipath-tools package in Ubuntu:
> Incomplete
>
> Bug description:
> When Multipath 0.8.8 to 0.9.3 is used with Ubuntu20.04 and especially
> with 22.04, showing High CPU usage for systemd and multipathd
> processes, making node unresponsive. Multipath 0.9.4 works just fine.
> But as there is no official multipath package for 0.9.4 with Ubuntu
> 20.04/22.04, need to compile.
>
> The ask here is to have 0.9.4 available for Ubuntu 20.04 / 22.04.
> Currently its available for 23.x versions.
>
> I was able to use the Mantic package repo and install with 22.04 and
> make it work.
>
>
> Redhat fixed the same issue and below is the link,
>
> Redhat Bug description: https://access.redhat.com/solutions/7059978
>
> May be below is the code path having issue
> Seems this impacting 0.8.8 till 0.9.3.
> https://github.com/opensvc/multipath-tools/pull/31/commits/38851dcaafe31073655761228db7f09ffc5a9a58
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2066286/+subscriptions
>

[Mitchell Dzurick (mitchdz)]

Thank you for the quick reply!

I'll set Focal to Invalid due to the patch not applying cleanly and supposedly not being affected by this issue. Let me know if you think that is wrong.

I applied the patch to Jammy in my PPA for testing and it applied with fuzz. I didn't see any major concerns applying it. The package in my PPA is available at [0], so please test it once you have a chance.

[0] - https://launchpad.net/~mitchdz/+archive/ubuntu/lp2066286-multipath-tools-high-cpu-usage

[Ninoy Radhakrishnan (nradhakrishnan)]

I was able to install.

Is this the fix backported to 0.8.8 and the multipath version will be still
names 0.8.8-* or will it be like multipath 0.9.4 available with Ubuntu 22.04

root@ninoy-ubuntu-22-04-template:~# apt list multipath\*
Listing... Done
multipath-tools-boot/jammy 0.8.8-1ubuntu1.22.04.5~jammy1 all
multipath-tools/jammy 0.8.8-1ubuntu1.22.04.5~jammy1 amd64 [upgradable from:
0.8.8-1ubuntu1.22.04.4]
root@ninoy-ubuntu-22-04-template:~# apt --only-upgrade install
multipath-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  kpartx
Suggested packages:
  multipath-tools-boot
The following packages will be upgraded:
  kpartx multipath-tools
2 upgraded, 0 newly installed, 0 to remove and 108 not upgraded.
Need to get 414 kB of archives.
After this operation, 2,048 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1
https://ppa.launchpadcontent.net/mitchdz/lp2066286-multipath-tools-high-cpu-usage/ubuntu
jammy/main amd64 kpartx amd64 0.8.8-1ubuntu1.22.04.5~jammy1 [53.8 kB]
Get:2
https://ppa.launchpadcontent.net/mitchdz/lp2066286-multipath-tools-high-cpu-usage/ubuntu
jammy/main amd64 multipath-tools amd64 0.8.8-1ubuntu1.22.04.5~jammy1 [361
kB]
Fetched 414 kB in 1s (288 kB/s)
(Reading database ... 74437 files and directories currently installed.)
Preparing to unpack .../kpartx_0.8.8-1ubuntu1.22.04.5~jammy1_amd64.deb ...
Unpacking kpartx (0.8.8-1ubuntu1.22.04.5~jammy1) over
(0.8.8-1ubuntu1.22.04.4) ...
Preparing to unpack
.../multipath-tools_0.8.8-1ubuntu1.22.04.5~jammy1_amd64.deb ...
Unpacking multipath-tools (0.8.8-1ubuntu1.22.04.5~jammy1) over
(0.8.8-1ubuntu1.22.04.4) ...
Setting up kpartx (0.8.8-1ubuntu1.22.04.5~jammy1) ...
Setting up multipath-tools (0.8.8-1ubuntu1.22.04.5~jammy1) ...
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@ninoy-ubuntu-22-04-template:~# apt list multipath\*
Listing... Done
multipath-tools-boot/jammy 0.8.8-1ubuntu1.22.04.5~jammy1 all
multipath-tools/jammy,now 0.8.8-1ubuntu1.22.04.5~jammy1 amd64 [installed]
root@ninoy-ubuntu-22-04-template:~#

On Tue, May 21, 2024 at 2:35 PM Ninoy Radhakrishnan (he/him) <
<email address hidden>> wrote:

> Thank you for the quick help! I will test and let you know
>
> Regards,
> Ninoy
> ------------------------------
> *From:* <email address hidden> <email address hidden> on behalf of
> Mitchell Dzurick <email address hidden>
> *Sent:* Tuesday, May 21, 2024 2:17:53 PM
> *To:* <email address hidden> <email address hidden>
> *Subject:* [Bug 2066286] Re: Multipath 0.8.8 reports High CPU usage when
> having Lun's using Multipath
>
> Thank you for the quick reply!
>
> I'll set Focal to Invalid due to the patch not applyi...

[Mitchell Dzurick (mitchdz)]

The version in my PPA I shared is still 0.8.8 with the upstream fix applied and nothing else.

[Mitchell Dzurick (mitchdz)]

Are you able to see any improvements in CPU usage with the fix?

[Ninoy Radhakrishnan (nradhakrishnan)]

Will ask the QE team to do some regression tests. Please give me some time
for doing that.

On Tue, May 21, 2024 at 4:10 PM Mitchell Dzurick <email address hidden>
wrote:

> Are you able to see any improvements in CPU usage with the fix?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2066286
>
> Title:
> Multipath 0.8.8 reports High CPU usage when having Lun's using
> Multipath
>
> Status in multipath-tools package in Ubuntu:
> Incomplete
> Status in multipath-tools source package in Focal:
> Invalid
> Status in multipath-tools source package in Jammy:
> Triaged
>
> Bug description:
> When Multipath 0.8.8 to 0.9.3 is used with Ubuntu20.04 and especially
> with 22.04, showing High CPU usage for systemd and multipathd
> processes, making node unresponsive. Multipath 0.9.4 works just fine.
> But as there is no official multipath package for 0.9.4 with Ubuntu
> 20.04/22.04, need to compile.
>
> The ask here is to have 0.9.4 available for Ubuntu 20.04 / 22.04.
> Currently its available for 23.x versions.
>
> I was able to use the Mantic package repo and install with 22.04 and
> make it work.
>
>
> Redhat fixed the same issue and below is the link,
>
> Redhat Bug description: https://access.redhat.com/solutions/7059978
>
> May be below is the code path having issue
> Seems this impacting 0.8.8 till 0.9.3.
>
> https://github.com/opensvc/multipath-tools/pull/31/commits/38851dcaafe31073655761228db7f09ffc5a9a58
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2066286/+subscriptions
>
>

[Ninoy Radhakrishnan (nradhakrishnan)]

We are still testing and has not completed yet. Will update you as soon as I have one

> On May 23, 2024, at 7:10 PM, Mitchell Dzurick <email address hidden> wrote:
>
> ** Merge proposal linked:
> https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/466356
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2066286
>
> Title:
> Multipath 0.8.8 reports High CPU usage when having Lun's using
> Multipath
>
> Status in multipath-tools package in Ubuntu:
> Incomplete
> Status in multipath-tools source package in Focal:
> Invalid
> Status in multipath-tools source package in Jammy:
> Triaged
>
> Bug description:
> When Multipath 0.8.8 to 0.9.3 is used with Ubuntu20.04 and especially
> with 22.04, showing High CPU usage for systemd and multipathd
> processes, making node unresponsive. Multipath 0.9.4 works just fine.
> But as there is no official multipath package for 0.9.4 with Ubuntu
> 20.04/22.04, need to compile.
>
> The ask here is to have 0.9.4 available for Ubuntu 20.04 / 22.04.
> Currently its available for 23.x versions.
>
> I was able to use the Mantic package repo and install with 22.04 and
> make it work.
>
>
> Redhat fixed the same issue and below is the link,
>
> Redhat Bug description: https://access.redhat.com/solutions/7059978
>
> May be below is the code path having issue
> Seems this impacting 0.8.8 till 0.9.3.
> https://github.com/opensvc/multipath-tools/pull/31/commits/38851dcaafe31073655761228db7f09ffc5a9a58
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2066286/+subscriptions
>

[Ninoy Radhakrishnan (nradhakrishnan)]

 We are still having issue with the new build provided. But the 0.9.4 from mantic just works fine

Regards,
Ninoy
________________________________
From: <email address hidden> <email address hidden> on behalf of Mitchell Dzurick <email address hidden>
Sent: Thursday, May 23, 2024 6:10:23 PM
To: <email address hidden> <email address hidden>
Subject: [Bug 2066286] Re: Multipath 0.8.8 reports High CPU usage when having Lun's using Multipath

** Merge proposal linked:
   https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/466356

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/2066286

Title:
  Multipath 0.8.8 reports High CPU usage when having Lun's using
  Multipath

Status in multipath-tools package in Ubuntu:
  Incomplete
Status in multipath-tools source package in Focal:
  Invalid
Status in multipath-tools source package in Jammy:
  Triaged

Bug description:
  When Multipath 0.8.8 to 0.9.3 is used with Ubuntu20.04 and especially
  with 22.04, showing High CPU usage for systemd and multipathd
  processes, making node unresponsive. Multipath 0.9.4 works just fine.
  But as there is no official multipath package for 0.9.4 with Ubuntu
  20.04/22.04, need to compile.

  The ask here is to have 0.9.4 available for Ubuntu 20.04 / 22.04.
  Currently its available for 23.x versions.

  I was able to use the Mantic package repo and install with 22.04 and
  make it work.

  Redhat fixed the same issue and below is the link,

  Redhat Bug description: https://access.redhat.com/solutions/7059978

  May be below is the code path having issue
  Seems this impacting 0.8.8 till 0.9.3.
  https://github.com/opensvc/multipath-tools/pull/31/commits/38851dcaafe31073655761228db7f09ffc5a9a58

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2066286/+subscriptions

[Mitchell Dzurick (mitchdz)]

Interesting, is there no improvements at all? Did you ensure the correct version is installed?

I can take a dive into the git history to see if I can find other potential fixes, but I remember looking a little while back and not finding anything that looked like a smoking gun, especially since I'm not sure what the actual bug is other than the resulting effect of high CPU usage.

[Ninoy Radhakrishnan (nradhakrishnan)]

Hi Mitchell, will you be able to add the full multipath 0.9.4 packages to the fix.

[Mitchell Dzurick (mitchdz)]

No, Version 0.9.4 is not planned to be backported to Jammy. Big updates like these do not happen often to stable releases.

To get this fixed, the specific bug will be narrowed down and just the fix itself will need to be backported. I'd be glad to do some investigation on what other commits may look like the fix if you are able to test them out.

Alternatively, do you have steps to reproduce this issue? I can investigate it more in my testbed.

[andres moreno (andymoreno)]

Hi,

I just happened to experience something similar. However in my case the multipath generates a ton of connections, locking the machine completely - making it unresponsive.

This behaviour only happens in ubuntu 22.04.

The issue is hard to reproduce - it does not happen often.
but if I have to narrow it down, I'd say it happens when umounting/mounting a disk.
And performing a sudo multipath -F cleans up this connections (at least most of the time)

(I'm attaching 3 pictures into one, I don't see the option to upload multiple ones)

[Mitchell Dzurick (mitchdz)]

Andres,

Thanks a lot for chiming in. It's always good data to see multiple people hitting this issue to help narrow things down.

> but if I have to narrow it down, I'd say it happens when umounting/mounting a disk.
And performing a sudo multipath -F cleans up this connections (at least most of the time)

I have a question about this - does the CPU usage stay high until you run -F? Or will it over time diminish?

Also, would either of you (Andres, Ninoy) be willing and able to run a profiler to help narrow down exactly what function is using all the CPU time? If we find what function it is, I can narrow down my git archaeology efforts.

[andres moreno (andymoreno)]

>I have a question about this - does the CPU usage stay high until you run -F? Or will it over time diminish?

as soon as I trigger the command, the CPU usage goes down. And as I stated before also all the related "Connecting" status -(old attachment).

I would like to help out running a profiler, however excuse my ignorance -But how I can do this. (list of steps)

[Mitchell Dzurick (mitchdz)]

Andres,

Thanks for being willing to help out! Let me see if there is some easy way to profile multipath-tools, I'll get back to you on that.

[Mitchell Dzurick (mitchdz)]

Andres, Apologies for the delay.

It looks like the default solution is simply using perf, which I can help setup, but the problem is that perf logs a ton of info. The perf logs will balloon in size substantially. How often does this issue occur for you? If it does not happen frequently then I'd not suggest perf and try something else. I was looking into callgrind[0] which looks pretty good, but personally have not used it so I can't vouch for it fully.

[0] - https://valgrind.org/docs/manual/cl-manual.html

[andres moreno (andymoreno)]

After some testing lately, I'm trying to narrow down even more the issue.

However this will take some time.

I'll keep you updated.

feeling so far:
after a sudo /sbin/dmsetup remove --force --retry XXXXX
and a sudo /sbin/multipathd reconfigure.
followed by a mount.
it triggers the block state.

[Mitchell Dzurick (mitchdz)]

Thanks Andres, that is very helpful information. Let me know if there's anything I can do to help out.

[Ninoy Radhakrishnan (nradhakrishnan)]

When the issue happens, and if the node has not hung fully and allows me to do a ssh session, below steps help me to fix the issue

1. Kill multipathd
2. systemctl restart multipathd
3. multipath -F
4. systemctl status multipathd
5. multipath -ll

[Mitchell Dzurick (mitchdz)]

Any luck narrowing down the issue more Andres?

Thanks for the remedy steps Ninoy.