/etc/pm/sleep.d/ is no more processed

[Alex Muntada]

[Impact (unattended-upgrades)]

 * Unattended-upgrades does not gracefully stop installing updates when the system goes to hibernation potentially letting the system to hibernate in a state where the system can't resume from.

[Test Case (unattended-upgrades)]

 * Configure the system to have several 20+ packages upgradable by unattended-upgrades. One easy way of setting this up is starting with a system where packages from -security are installed but packages from -updates are not and enabling -updates in u-u:

  # echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too

* Pre-download upgrades
  # unattended-upgraded --download-only

* Trigger unattended-upgrades run:
  # service apt-daily-upgrade start

* Hibernate, then resume the system and watch unattended-upgrades being gracefully stopped, checking /var/log/unattended-upgrades/unattended-upgrades.log. There should be packages left to be upgraded.

[Regression Potential (unattended-upgrades)]

 * The added hook for unattended-upgrades may block the system from hibernation for 25 minutes if unattended-upgrades (or most likely a package upgrades) hangs. There is a better solution proposed based on systemd's inhibitor interface at https://github.com/mvo5/unattended-upgrades/issues/162 .

[Original Bug Text]

I added a new script in /etc/pm/sleep.d/ that worked fine before the upgrade to vivid. I was wondering why it doesn't work and I find out that /var/log/pm-suspend.log is empty, and pm-powersave.log too (last logs from pm-suspend were on Apr 28, when I upgraded from utopic to vivid.

The laptop suspends and wakes fine, it just seems that no pm scripts are run after suspend/powersave.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: pm-utils 1.4.1-15
ProcVersionSignature: Ubuntu 3.19.0-16.16-generic 3.19.3
Uname: Linux 3.19.0-16-generic x86_64
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Thu May 14 15:36:26 2015
DistributionChannelDescriptor:
 # This is a distribution channel descriptor
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-precise-amd64-20120703-2
EcryptfsInUse: Yes
InstallationDate: Installed on 2013-03-12 (792 days ago)
InstallationMedia: Ubuntu 12.04 "Precise" - Build amd64 LIVE Binary 20120703-15:08
PackageArchitecture: all
SourcePackage: pm-utils
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[gubbins (dave-gittins)]

Identical problems for me after upgrading Xubuntu to 14.04.

[gubbins (dave-gittins)]

sorry, I meant to say: identical problems for me after upgrading Xubuntu to 15.04 (not 14.04)

[Martin Thornton (thornton-ma)]

Was caused by switch to systemd from upstart. Instead of pm-suspend, systemd uses systemd-sleep. Instead of running scripts in /etc/pm/sleep.d/ and /usr/lib/pm-utils/sleep.d/ it runs scripts in /lib/systemd/system-sleep/. However, no script is installed here to call pm-powersave, and systemd doesn't seem to have an alternative to it, so I added my own.

For example, I converted the /usr/lib/pm-utils/sleep.d/00powersave as follows:

###UPSTART:
#!/bin/sh
. "${PM_FUNCTIONS}"
ARCH=`uname -m`
command_exists pm-powersave || exit $NA
case $1 in
    suspend|hibernate) [ "$ARCH" != "${ARCH#arm}" ] || pm-powersave false ;;
    resume|thaw) pm-powersave ;;
    *) exit $NA ;;
esac
exit 0

###SYSTEMD:
#!/bin/sh
case $1 in
    post) pm-powersave ;;
    *) [ "$ARCH" != "${ARCH#arm}" ] || pm-powersave false ;;
esac
exit 0

It was necessary to remove the lines:
> . "${PM_FUNCTIONS}"
> command_exists pm-powersave || exit $NA
and note that parameters to script differ, see man systemd-sleep.

[Martin Thornton (thornton-ma)]

https://bugs.launchpad.net/ubuntu/+source/pm-utils/+bug/1458390 is probably a duplicate bug.

https://bugs.launchpad.net/ubuntu/+source/pm-utils/+bug/1461386 is a related issue concerning calling pm-powersave on AC loss/connection.

[Martin Thornton (thornton-ma)]

Improved previous attachment. You'll also need to:
sudo cp [FILENAME] /lib/systemd/system-sleep/00powersave
sudo chown root.root /lib/systemd/system-sleep/00powersave
sudo chmod u+x /lib/systemd/system-sleep/00powersave

[Alex Muntada (alex.muntada)]

I moved my script to systemd, adapted it following Martin's instructions in comment #5 and it just works as expected.

Thanks a lot :)

[Alex Muntada (alex.muntada)]

On the other side I removed pm-utils from the system and there are a couple of remaining scripts in /etc/pm/sleep.d that are worth mentioning:

 * 10_grub-common
 * 10_unattended-upgrades-hibernate

Thus, those packages are also affected by this bug I guess.

[gubbins (dave-gittins)]

Thanks Martin.

I'm confused why this isn't causing widespread problems for lots of users. I've got about 10 scripts in those folders which were installed by Ubuntu or various packages. Surely they did something useful? If I understand you rightly, they've all stopped getting run, for any user who has upgraded to vivid.

I don't know if I should try to convert them all, or remove them, or ignore them!

[gubbins (dave-gittins)]

These look relevant too:
http://askubuntu.com/questions/620494/ubuntu-15-04-suspend-doesnt-run-pm-suspend
http://askubuntu.com/questions/613741/ubuntu-15-04-pm-utils-do-not-look-into-etc-pm-power-d-anymore-what-instead
http://askubuntu.com/questions/609695/does-systemd-read-etc-pm

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

14:02 <rbasak> pitti: could you please take a look at bug 1455097? Alleged potential issues since systemd no longer uses /etc/pm/sleep.d/ but packages put stuff in there. Do we have the functionality we need provided for the systemd mechanisms also?

14:10 <pitti> rbasak: yes, I checked precise's apt-sources manpage, seems to be there

14:11 <pitti> rbasak: there is, one can put stuff into /lib/systemd/system-sleep/, but we don't actually want all the old pm-suspend quirks any more

14:12 <pitti> rbasak: so we could check which of the bits are really still needed, and link them from there

[Robie Basak (racb)]

So I think this should probably be Invalid or Won't Fix for pm-utils if the new mechanism should be /lib/systemd/system-sleep/, but we should have bug tasks for each package that used /etc/pm/sleep.d/ and still expects it to run, so I won't make that change yet.

[Christian Ehrhardt  (paelzer)]

I did the task of identifying the remaining packages that are affected.

$ apt-file search /etc/pm/sleep.d/

aiccu: /etc/pm/sleep.d/60aiccu
ceph: /etc/pm/sleep.d/25-ceph
grub-common: /etc/pm/sleep.d/10_grub-common
ifplugd: /etc/pm/sleep.d/74ifplugd
lizardfs-chunkserver: /etc/pm/sleep.d/25-lizardfs-chunkserver
oss4-base: /etc/pm/sleep.d/30oss4-pm
toshset: /etc/pm/sleep.d/novatel_3g_suspend
unattended-upgrades: /etc/pm/sleep.d/10_unattended-upgrades-hibernate

I have renamed the bug and will assign all those packages here.
Those packages should then consider individually if they can drop their old pm quirk, migrate to /lib/systemd/system-sleep/ or chose a different way out.

[Jeroen Massar (massar)]

On 2016-04-06 17:46, ChristianEhrhardt wrote:
> I did the task of identifying the remaining packages that are affected.
>
> $ apt-file search /etc/pm/sleep.d/
>
> aiccu: /etc/pm/sleep.d/60aiccu

There has been a bug out for this for 4 years already that this should
never ever have existed:

 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=689584

Short version: please remove any kind of trace from /etc/pm/sleep.d for
aiccu.

That should take care of your bug report for aiccu at least.

Greets,
 Jeroen

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 10.2.2-0ubuntu2

---------------
ceph (10.2.2-0ubuntu2) yakkety; urgency=medium

  * d/ceph.install,d/etc/*: Drop pm based power management scripts as they
    are not used under systemd (LP: #1455097).
  * d/*: Split ceph-osd and ceph-mon into separate binary packages and add
    new ceph-base binary package inline with upstream packaging changes
    (LP: #1596063).
  * d/rules,*.{postinst,prerm}: Drop install and management of upstart
    configuration files; this package version onwards will only ever
    target >= Xenial, which is a systemd baseline.

 -- James Page <email address hidden> Thu, 14 Jul 2016 10:51:10 +0100

[Lars Düsing (lars.duesing)]

aiccu package should be taken out of distribution due to closing of sixxs.net - there is no running tunnel broker any more.

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.98ubuntu1

---------------
unattended-upgrades (0.98ubuntu1) artful; urgency=medium

  * Merge from Debian unstable (LP: #1722426)
    - Remaining changes:
      - unattended-upgrades: Do not automatically upgrade the development
        release of Ubuntu unless Unattended-Upgrade::DevRelease is true.
    - Dropped changes, included in Debian:
      - Use lsb_release instead of dpkg-vendor in postinst

unattended-upgrades (0.98) unstable; urgency=medium

  * Rename test to test_non_minimal_steps_upgrade to reflect content
  * Catch SystemError while keeping apt lock unlocked (LP: #1632361)
  * Add --stop-only option to unattended-upgrade-shutdown and use it on
    hibernation. This prevents starting unattended-upgrades right before
    hibernating when Unattended-Upgrade::InstallOnShutdown is true.
    (Closes: #610333)
  * Stop already running unattended-upgrades before hibernation even with
    systemd (LP: #1455097)
  * Use lsb_release instead of dpkg-vendor in postinst (LP: #1719630)
  * Update README.md with the fact that unattended=upgrades is enabled
    by default (Closes: #865519)

 -- Balint Reczey <email address hidden> Tue, 10 Oct 2017 03:47:58 +0200

[Brian Murray (brian-murray)]

Hello Alex, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alex Muntada (alex.muntada)]

Hi Brian,

I can't help testing this new package since I'm not running Ubuntu on my laptop anymore.

Cheers!
Alex

[Łukasz Zemczak (sil2100)]

Hello Alex, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Alex, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Balint Reczey (rbalint)]

Verified 1.1ubuntu1.18.04.7~16.04.2 on Xenial:

ubuntu@ip-172-31-9-2:~$ dpkg -l unattended-upgrades | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-==========================-============-===========================================
ii unattended-upgrades 1.1ubuntu1.18.04.7~16.04.2 all automatic installation of security upgrades

ubuntu@ip-172-31-9-2:~$ sudo service apt-daily-upgrade start & tail -f /var/log/unattended-upgrades/unattended-upgrades.log
[2] 13368
2019-04-09 16:11:45,975 INFO Initial blacklisted packages:
2019-04-09 16:11:45,975 INFO Initial whitelisted packages:
2019-04-09 16:11:45,976 INFO Starting unattended upgrades script
2019-04-09 16:11:45,976 INFO Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial, o=Ubuntu,a=xenial-updates
2019-04-09 16:13:06,974 INFO Initial blacklisted packages:
2019-04-09 16:13:06,974 INFO Initial whitelisted packages:
2019-04-09 16:13:06,974 INFO Starting unattended upgrades script
2019-04-09 16:13:06,974 INFO Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial, o=Ubuntu,a=xenial-updates
2019-04-09 16:13:11,782 INFO Packages that will be upgraded: apt apt-transport-https apt-utils busybox-initramfs busybox-static cloud-init distro-info-data file grub-common grub-legacy-ec2 grub-pc grub-pc-bin grub2-common libapt-inst2.0 libapt-pkg5.0 libmagic1 libpam-systemd libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libsystemd0 libudev1 linux-aws linux-headers-aws linux-image-aws ntfs-3g policykit-1 rsyslog snapd systemd systemd-sysv ubuntu-core-launcher udev wget
2019-04-09 16:13:11,782 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2019-04-09 16:13:17,835 WARNING SIGTERM received, will stop
2019-04-09 16:13:18,514 WARNING SIGNAL received, stopping

... connection lost.

After resume:
$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-aws-headers-4.4.0-1079 linux-headers-4.4.0-1079-aws linux-image-4.4.0-1079-aws linux-modules-4.4.0-1079-aws
The following packages will be upgraded:
  apt apt-transport-https apt-utils busybox-initramfs busybox-static cloud-init file grub-common grub-legacy-ec2 grub-pc grub-pc-bin grub2-common libapt-inst2.0
  libapt-pkg5.0 liblxc1 libmagic1 libpam-systemd libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libsystemd0 libudev1 linux-aws linux-headers-aws
  linux-image-aws login lxc-common ntfs-3g passwd policykit-1 python-apt-common python3-apt rsyslog snapd systemd systemd-sysv ubuntu-core-launcher udev uidmap wget
40 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.9 MB/56.1 MB of archives.
After this operation, 118 MB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
ubuntu@ip-172-31-9-2:~$...

[Balint Reczey (rbalint)]

I need to verify this bug again because acpi was pulled in to the image.

[Balint Reczey (rbalint)]

Verified 1.1ubuntu1.18.04.7~16.04.2 on Xenial, now without acpdid:

ubuntu@ip-172-31-3-29:~$ sudo service apt-daily-upgrade start & sleep 1; tail -f /var/log/unattended-upgrades/unattended-upgrades.log
[1] 1607
2019-04-09 20:58:27,975 INFO Initial blacklisted packages:
2019-04-09 20:58:27,975 INFO Initial whitelisted packages:
2019-04-09 20:58:27,975 INFO Starting unattended upgrades script
2019-04-09 20:58:27,975 INFO Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial
2019-04-09 20:58:31,986 INFO Packages that will be upgraded: busybox-initramfs busybox-static file libmagic1 libpam-systemd libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libsystemd0 libudev1 linux-aws linux-headers-aws linux-image-aws ntfs-3g policykit-1 snapd systemd systemd-sysv ubuntu-core-launcher udev wget
2019-04-09 20:58:31,987 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2019-04-09 20:58:49,942 WARNING SIGTERM received, will stop
2019-04-09 20:58:52,944 WARNING SIGTERM received, will stop
2019-04-09 20:58:53,159 WARNING SIGNAL received, stopping

ubuntu@ip-172-31-3-29:~$ cat /var/log/unattended-upgrades/unattended-upgrades-dpkg.log .
Log started: 2019-04-09 20:58:35
(Reading database ... 76460 files and directories currently installed.)
Preparing to unpack .../libpolkit-gobject-1-0_0.105-14.1ubuntu0.5_amd64.deb ...
Unpacking libpolkit-gobject-1-0:amd64 (0.105-14.1ubuntu0.5) over (0.105-14.1ubuntu0.4) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libpolkit-gobject-1-0:amd64 (0.105-14.1ubuntu0.5) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Log ended: 2019-04-09 20:58:37
...
ubuntu@ip-172-31-3-29:~$ dpkg -l unattended-upgrades | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-==========================-============-===========================================
ii unattended-upgrades 1.1ubuntu1.18.04.7~16.04.2 all automatic installation of security upgrades
ubuntu@ip-172-31-3-29:~$ yes no | sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-aws-headers-4.4.0-1079 linux-headers-4.4.0-1079-aws linux-image-4.4.0-1079-aws
  linux-modules-4.4.0-1079-aws
The following packages will be upgraded:
  apt apt-transport-https apt-utils cloud-init distro-info-data file grub-common grub-legacy-ec2 grub-pc
  grub-pc-bin grub2-common libapt-inst2.0 libapt-pkg5.0 liblxc1 libmagic1 libpam-systemd libsystemd0 libudev1
  linux-aws linux-headers-aws linux-image-aws login lxc-common ntfs-3g passwd policykit-1 python-apt-common
  python3-apt rsyslog snapd systemd systemd-sysv ubuntu-core-launcher udev uidmap
35 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/54.7 MB of archives.
After this operation, 118 MB of additional disk space will be used.
Do you wa...

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2

---------------
unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium

  * Don't check blacklist too early and report updates from not allowed origins
    as kept back. (LP: #1781176)
  * test/test_blacklisted_wrong_origin.py: Fix and enable test
  * Filter out progress indicator from dpkg log (LP: #1599646)
  * Clear cache when autoremoval fails (LP: #1779157)
  * Find autoremovable kernel packages using the patterns in APT's way
    (LP: #1815494)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium

  * Start service after systemd-logind.service to be able to take inhibition
    lock (LP: #1806487)
  * Handle gracefully when logind is down (LP: #1806487)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium

  * Backport to Xenial (LP: #1702793)
  * Revert to build-depending on debhelper (>= 9~) and dh-systemd
  * Revert configuration example changes to avoid triggering a debconf question
  * debian/postinst: Update recovery to be triggered on Xenial's package versions

unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium

  * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
    Performing upgrades in service's ExecStop did not work when the upgrades
    involved restarting services because systemd blocked other stop/start
    actions making maintainer scripts time out and be killed leaving a broken
    system behind.
    Running unattended-upgrades.service before shutdown.target as a oneshot
    service made it run after unmounting filesystems and scheduling services
    properly on shutdown is a complex problem and adding more services to the
    mix make it even more fragile.
    The solution of monitoring PrepareForShutdown() signal from DBus
    allows Unattended Upgrade to run _before_ the jobs related to shutdown are
    queued thus package upgrades can safely restart services without
    risking causing deadlocks or breaking part of the shutdown actions.
    Also ask running unattended-upgrades to stop when shutdown starts even in
    InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
    UnattendedUpgradesShutdown class. (LP: #1778219)
  * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219)
    This allows more time for unattended-upgrades to shut down gracefully
    or even install a few packages in InstallOnShutdown mode, but is still a
    big step back from the 30 minutes allowed for InstallOnShutdown previously.
    Users enabling InstallOnShutdown node are advised to increase
    InhibitDelayMaxSec even further possibly to 30 minutes.
    - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
      changes
  * Ignore "W503 line break before binary operator"
    because it will become the best practice and breaks the build
  * Stop using ActionGroups, they interfere with apt.Cache.clear()
    causing all autoremovable packages to be handled as newly autoremovable
    ones and be removed by default. Dropping ActionGroup usage does not slow
    down the most frequent case of not having anything to upgrade a...