Continuously high CPU utilization by PBZIP2 utility in linux system

Bug #1680316 reported by Deepak Rathi on 2017-04-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
pbzip2
Undecided
Unassigned

Bug Description

Environment:

[root@sadmapp3 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)

PBZIP2 version installed:

[ntms@sadmapp3 ~]$ pbzip2 -V
Parallel BZIP2 v1.1.6 - by: Jeff Gilchrist [http://compression.ca]
[Oct. 30, 2011] (uses libbzip2 by Julian Seward)
Major contributions: Yavor Nikolov <email address hidden>

Issue Statement artefacts:

Snapshot attached for reference showing abnormally high usage by PBSIP2 PID's.

Hi Team,

Please help in addressing this issue as multiple service failure incidents observed due to high cpu usage on a particular node, i have cluster setup here bu usage is high on sadmapp3 only.

Thanks,
Deepak Rathi

Deepak Rathi (rathi26) wrote :
Yavor Nikolov (yavor-nikolov) wrote :

Why do you consider the CPU usage to be abnormal?

pbzip2 is a parallel bzip2 utility which allows to engage more of the CPU power in compression/decompression activities (so that it runs faster). High CPU usage is in general not something abnormal.

There is a -p flag which you may use to set the number of worker threads to use (if not explicitly set - by default that's equal to the number of your CPUs). As far as I see from the screenshot - there is some kind of backup running (the tar processes) in at least 2 processes (it means it's engaging 2 x #CPUs threads).

So you can tweak that backup - by specifying a different -p# for pbzip2, or to run in single thread - switch to plain bzip2, or you may change the priority of the process.

--
Maybe not related to the issue, but I also see you're using a bit old version of pbzip2. You may find the latest release at https://launchpad.net/pbzip2

Hi Nikolov,

Thanks for reviewing and your response.

Since pbzip2 is installed on all blade servers in the cluster but it is cpu
usage of only a specific blade i.e. sadmapp3 is going high during mentioned
backup execution duration.

Can we evenly distribute the load of subjected pbzip2 process at the time
of backup among CPU's of all cluster members instead of only using
resources of sadmapp3.

Pls guide with procedure for same if feasible wrt pbzip2 utility.

PS: CPU usage is a area of concern in my case as here the collective load
due to backup/archiving procedures + high CPU due to pbzip2, proving to be
a lethal combination and suspected to be a reason for application service
down on sadmapp3 during that time.

Thanks,
Deepak Rathi

On Apr 6, 2017 2:41 PM, "Yavor Nikolov" <email address hidden> wrote:

Why do you consider the CPU usage to be abnormal?

pbzip2 is a parallel bzip2 utility which allows to engage more of the
CPU power in compression/decompression activities (so that it runs
faster). High CPU usage is in general not something abnormal.

There is a -p flag which you may use to set the number of worker threads
to use (if not explicitly set - by default that's equal to the number of
your CPUs). As far as I see from the screenshot - there is some kind of
backup running (the tar processes) in at least 2 processes (it means
it's engaging 2 x #CPUs threads).

So you can tweak that backup - by specifying a different -p# for pbzip2,
or to run in single thread - switch to plain bzip2, or you may change
the priority of the process.

--
Maybe not related to the issue, but I also see you're using a bit old
version of pbzip2. You may find the latest release at
https://launchpad.net/pbzip2

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1680316

Title:
  Continuously high CPU utilization by PBZIP2 utility in linux system

Status in pbzip2:
  New

Bug description:
  Environment:

  [root@sadmapp3 ~]# cat /etc/redhat-release
  Red Hat Enterprise Linux Server release 6.4 (Santiago)

  PBZIP2 version installed:

  [ntms@sadmapp3 ~]$ pbzip2 -V
  Parallel BZIP2 v1.1.6 - by: Jeff Gilchrist [http://compression.ca]
  [Oct. 30, 2011] (uses libbzip2 by Julian Seward)
  Major contributions: Yavor Nikolov <email address hidden>

  Issue Statement artefacts:

  Snapshot attached for reference showing abnormally high usage by
  PBSIP2 PID's.

  Hi Team,

  Please help in addressing this issue as multiple service failure
  incidents observed due to high cpu usage on a particular node, i have
  cluster setup here bu usage is high on sadmapp3 only.

  Thanks,
  Deepak Rathi

To manage notifications about this bug go to:
https://bugs.launchpad.net/pbzip2/+bug/1680316/+subscriptions

Yavor Nikolov (yavor-nikolov) wrote :

Hi,

> Can we evenly distribute the load of subjected pbzip2 process at the time of backup among CPU's of all cluster members instead of only using resources of sadmapp3.

That's a question not about pbzip2 but about the job which is running it. It's mostly about details of these processes which are running the tar command. If you know more about them you might be able to split it somehow. E.g. - split the files/streams into smaller chunks, evenly distribute the pieces across all servers, then run that command on each of them. Note: if the files are really big - and if you run pbzip2 with same number of threads, you may still cause high CPU usage (on all servers) - it just may take shorter time to complete.

* One possible idea is to copy/redirect these backup streams to another server (not so sensitive in terms of CPU usage) and run compression there.
* Another thing I already suggested: change these commands to use either pbzip2 with -p option (e.g. -p1 or -p2); or use bzip2 instead of pbzip2.

Jeff Gilchrist (jeff-gilchrist) wrote :

Hi Deepak,

Remember the purpose of pbzip2 is to compress data more quickly because you
can use all the CPU resources available instead of be stuck with one
processor/core. If you are worried about pbzip2 using up so much CPU
resources that it is negatively effecting other processes, you could also
consider reducing the priority of pbzip2, such as using "nice". That way
pbzip2 may still look like it is using a lot of CPU resources but it will
only be using the resources that are otherwise not being used and let your
important processes have the CPU time they need to work.

Or as Yavor suggested, just us bzip2 which only uses one CPU. It will take
a lot longer but use much less CPU resources.

Jeff.

On Thu, Apr 6, 2017 at 6:59 PM, Yavor Nikolov <email address hidden>
wrote:

> Hi,
>
> > Can we evenly distribute the load of subjected pbzip2 process at the
> time of backup among CPU's of all cluster members instead of only using
> resources of sadmapp3.
>
> That's a question not about pbzip2 but about the job which is running
> it. It's mostly about details of these processes which are running the
> tar command. If you know more about them you might be able to split it
> somehow. E.g. - split the files/streams into smaller chunks, evenly
> distribute the pieces across all servers, then run that command on each
> of them. Note: if the files are really big - and if you run pbzip2 with
> same number of threads, you may still cause high CPU usage (on all
> servers) - it just may take shorter time to complete.
>
>
> * One possible idea is to copy/redirect these backup streams to another
> server (not so sensitive in terms of CPU usage) and run compression there.
> * Another thing I already suggested: change these commands to use either
> pbzip2 with -p option (e.g. -p1 or -p2); or use bzip2 instead of pbzip2.
>
> --
> You received this bug notification because you are subscribed to pbzip2.
> https://bugs.launchpad.net/bugs/1680316
>
> Title:
> Continuously high CPU utilization by PBZIP2 utility in linux system
>
> Status in pbzip2:
> New
>
> Bug description:
> Environment:
>
> [root@sadmapp3 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 6.4 (Santiago)
>
> PBZIP2 version installed:
>
> [ntms@sadmapp3 ~]$ pbzip2 -V
> Parallel BZIP2 v1.1.6 - by: Jeff Gilchrist [http://compression.ca]
> [Oct. 30, 2011] (uses libbzip2 by Julian Seward)
> Major contributions: Yavor Nikolov <email address hidden>
>
>
> Issue Statement artefacts:
>
> Snapshot attached for reference showing abnormally high usage by
> PBSIP2 PID's.
>
> Hi Team,
>
> Please help in addressing this issue as multiple service failure
> incidents observed due to high cpu usage on a particular node, i have
> cluster setup here bu usage is high on sadmapp3 only.
>
> Thanks,
> Deepak Rathi
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/pbzip2/+bug/1680316/+subscriptions
>

Deepak Rathi (rathi26) wrote :
Download full text (4.6 KiB)

Thanks guys for response, will be working on below suggestion to mitigate
the issue along with version upgrade:

Another thing I already suggested: change these commands to use either
pbzip2 with -p option (e.g. -p1 or -p2);

On Fri, Apr 7, 2017 at 5:53 AM, Jeff Gilchrist <email address hidden>
wrote:

> Hi Deepak,
>
> Remember the purpose of pbzip2 is to compress data more quickly because you
> can use all the CPU resources available instead of be stuck with one
> processor/core. If you are worried about pbzip2 using up so much CPU
> resources that it is negatively effecting other processes, you could also
> consider reducing the priority of pbzip2, such as using "nice". That way
> pbzip2 may still look like it is using a lot of CPU resources but it will
> only be using the resources that are otherwise not being used and let your
> important processes have the CPU time they need to work.
>
> Or as Yavor suggested, just us bzip2 which only uses one CPU. It will take
> a lot longer but use much less CPU resources.
>
> Jeff.
>
> On Thu, Apr 6, 2017 at 6:59 PM, Yavor Nikolov <email address hidden>
> wrote:
>
> > Hi,
> >
> > > Can we evenly distribute the load of subjected pbzip2 process at the
> > time of backup among CPU's of all cluster members instead of only using
> > resources of sadmapp3.
> >
> > That's a question not about pbzip2 but about the job which is running
> > it. It's mostly about details of these processes which are running the
> > tar command. If you know more about them you might be able to split it
> > somehow. E.g. - split the files/streams into smaller chunks, evenly
> > distribute the pieces across all servers, then run that command on each
> > of them. Note: if the files are really big - and if you run pbzip2 with
> > same number of threads, you may still cause high CPU usage (on all
> > servers) - it just may take shorter time to complete.
> >
> >
> > * One possible idea is to copy/redirect these backup streams to another
> > server (not so sensitive in terms of CPU usage) and run compression
> there.
> > * Another thing I already suggested: change these commands to use either
> > pbzip2 with -p option (e.g. -p1 or -p2); or use bzip2 instead of pbzip2.
> >
> > --
> > You received this bug notification because you are subscribed to pbzip2.
> > https://bugs.launchpad.net/bugs/1680316
> >
> > Title:
> > Continuously high CPU utilization by PBZIP2 utility in linux system
> >
> > Status in pbzip2:
> > New
> >
> > Bug description:
> > Environment:
> >
> > [root@sadmapp3 ~]# cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.4 (Santiago)
> >
> > PBZIP2 version installed:
> >
> > [ntms@sadmapp3 ~]$ pbzip2 -V
> > Parallel BZIP2 v1.1.6 - by: Jeff Gilchrist [http://compression.ca]
> > [Oct. 30, 2011] (uses libbzip2 by Julian Seward)
> > Major contributions: Yavor Nikolov <email address hidden>
> >
> >
> > Issue Statement artefacts:
> >
> > Snapshot attached for reference showing abnormally high usage by
> > PBSIP2 PID's.
> >
> > Hi Team,
> >
> > Please help in addressing this issue as multiple service failure
> > incidents observed due to high cp...

Read more...

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers