[UBUNTU 24.04] IOMMU DMA mode changed in kernel config causes massive throughput degradation for PCI-related network workloads

[bugproxy]

SRU Justification:

[Impact]

 * With the introduction of c76c067e488c "s390/pci: Use dma-iommu layer"
   (upstream with since kernel v6.7-rc1) there was a move (on s390x only)
   to a different dma-iommu implementation.

 * And with 92bce97f0c34 "s390/pci: Fix reset of IOMMU software counters"
   (again upstream since 6.7(rc-1) the IOMMU_DEFAULT_DMA_LAZY kernel config
   option should now be set to 'yes' by default for s390x.

 * Since CONFIG_IOMMU_DEFAULT_DMA_STRICT and IOMMU_DEFAULT_DMA_LAZY
   are related to each other CONFIG_IOMMU_DEFAULT_DMA_STRICT needs to be
   set to "no" by default, which was upstream done by b2b97a62f055
   "Revert "s390: update defconfigs"".

 * These changes are all upstream, but were not picked up by the Ubuntu
   kernel config.

 * And not having these config options set properly is causing significant
   PCI-related network throughput degradation (up to -72%).

 * This shows for almost all workloads and numbers of connections,
   deteriorating with the number of connections increasing.

 * Especially drastic is the drop for a high number of parallel connections
   (50 and 250) and for small and medium-size transactional workloads.
   However, also for streaming-type workloads the degradation is clearly
   visible (up to 48% degradation).

[Fix]

 * The (upstream accepted) fix is to set
   IOMMU_DEFAULT_DMA_STRICT=no
   and
   IOMMU_DEFAULT_DMA_LAZY=y
   (which is needed for the changed DAM IOMMU implementation since v6.7).

[Test Case]

 * Setup two Ubuntu Server 24.04 systems (with kernel 6.8)
   (one acting as server and as client)
   that have (PCIe attached) RoCE Express devices attached
   and that are connected to each other.

 * Verify if the the iommu_group type of the used PCI device is DMA-FQ:
   cat /sys/bus/pci/devices/<device>\:00\:00.0/iommu_group/type
   DMA-FQ

 * Sample workload rr1c-200x1000-250 with rr1c-200x1000-250.xml:
   <?xml version="1.0"?>
   <profile name="TCP_RR">
           <group nprocs="250">
                   <transaction iterations="1">
                           <flowop type="connect" options="remotehost=<remote IP> protocol=tcp tcp_nodelay" />
                   </transaction>
                   <transaction duration="300">
                           <flowop type="write" options="size=200"/>
                           <flowop type="read" options="size=1000"/>
                   </transaction>
                   <transaction iterations="1">
                           <flowop type="disconnect" />
                   </transaction>
           </group>
   </profile>

 * Install uperf on both systems, client and server.

 * Start uperf at server: uperf -s

 * Start uperf at client: uperf -vai 5 -m uperf-profile.xml

 * Switch from strict to lazy mode
   either using the new kernel (or the test build below)
   or using kernel cmd-line parameter iommu.strict=0.

 * Restart uperf on server and client, like before.

 * Verification will be performed by IBM.

[Regression Potential]

 * The is a certain regression potential, since the behavior with
   the two modified kernel config options will change significantly.

 * This may solve the (network) throughput issue with PCI devices,
   but may also come with side-effects on other PCIe based devices
   (the old compression adapters or the new NVMe carrier cards).

[Other]

 * CCW devices are not affected.

 * This is s390x-specific only, hence will not affect any other architecture.

__________

Symptom:
Comparing Ubuntu 24.04 (kernelversion: 6.8.0-31-generic) against Ubuntu 22.04, all of our PCI-related network measurements on LPAR show massive throughput degradations (up to -72%). This shows for almost all workloads and numbers of connections, detereorating with the number of connections increasing. Especially drastic is the drop for a high number of parallel connections (50 and 250) and for small and medium-size transactional workloads. However, also for streaming-type workloads the degradation is clearly visible (up to 48% degradation).

Problem:
With kernel config setting CONFIG_IOMMU_DEFAULT_DMA_STRICT=y, IOMMU DMA mode changed from lazy to strict, causing these massive degradations.
Behavior can also be changed with a kernel commandline parameter (iommu.strict) for easy verification.

The issue is known and was quickly fixed upstream in December 2023, after being present for little less than two weeks.
Upstream fix: https://github.com/torvalds/linux/commit/b2b97a62f055dd638f7f02087331a8380d8f139a

Repro:
rr1c-200x1000-250 with rr1c-200x1000-250.xml:

<?xml version="1.0"?>
<profile name="TCP_RR">
        <group nprocs="250">
                <transaction iterations="1">
                        <flowop type="connect" options="remotehost=<remote IP> protocol=tcp tcp_nodelay" />
                </transaction>
                <transaction duration="300">
                        <flowop type="write" options="size=200"/>
                        <flowop type="read" options="size=1000"/>
                </transaction>
                <transaction iterations="1">
                        <flowop type="disconnect" />
                </transaction>
        </group>
</profile>

0) Install uperf on both systems, client and server.
1) Start uperf at server: uperf -s
2) Start uperf at client: uperf -vai 5 -m uperf-profile.xml

3) Switch from strict to lazy mode using kernel commandline parameter iommu.strict=0.
4) Repeat steps 1) and 2).

Example:
For the following example, we chose the workload named above (rr1c-200x1000-250):

iommu.strict=1 (strict): 233464.914 TPS
iommu.strict=0 (lazy): 835123.193 TPS

[Frank Heimes (fheimes)]

I just had a look at the Ubuntu kernel noble master-next tree and can find commit:
"Revert "s390: update defconfigs"" under the hash b2b97a62f055
and I can see that it got reverted with that:
$ git show b2b97a62f055 | grep CONFIG_IOMMU_DEFAULT_DMA_STRICT
    CONFIG_IOMMU_DEFAULT_DMA_STRICT option needs to be disabled.
-CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
-CONFIG_IOMMU_DEFAULT_DMA_STRICT=y

But git also tells me that it is in since kernel v6.8 and with that since the first ubuntu 6.8 kernel we had: Ubuntu-6.8.0-6 -- so should also be in Ubuntu-6.8.0-31.
But it does not seem to be reflected in the kernel options of the Ubuntu kernel:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
$ uname -a
Linux hwe0008 6.8.0-31-generic #31-Ubuntu SMP Sat Apr 20 00:14:26 UTC 2024 s390x s390x s390x GNU/Linux
$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-6.8.0-31-generic
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
also not in the current, updated kernel:
Linux hwe0008 6.8.0-36-generic #36-Ubuntu SMP Mon Jun 10 09:59:13 UTC 2024 s390x s390x s390x GNU/Linux
$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-6.8.0-31-generic
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
For some reason the change in the upstream commit was not taken over into the Ubuntu kernel configs ...

[Kleber Sacilotto de Souza (kleber-souza)]

Upstream commit b2b97a62f055dd638f7f02087331a8380d8f139a changes the s390x defconfig's, which don't impact the Ubuntu kernel configs.

Looking at the configs for the 22.04 generic kernel, 'CONFIG_IOMMU_DEFAULT_DMA_STRICT=y' seems to be set for s390x on all Ubuntu 5.15 generic kernel builds as well. Therefore it doesn't seem the config has regressed, and although changing the kernel parameter during the boot time improves the performance it is likely not the only aspect causing the degradation.

Can IBM please confirm which 22.04 kernel version doesn't show the performance degradation and what are the config values for 'CONFIG_IOMMU_DEFAULT_DMA_STRICT' and 'CONFIG_IOMMU_DEFAULT_DMA_LAZY' on the running kernel?

Before changing the kernel config I would like to gather some more information to assess the consequences of the change and whether we need to make other changes as well.

Thank you.

[Frank Heimes (fheimes)]

Btw. I just checked the jammy kernel config options and jammy/22.04 had and still has strict=yes:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
$ uname -a
Linux maasrrc1 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 08:10:47 UTC 2024 s390x s390x s390x GNU/Linux
ubuntu@maasrrc1:~$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-5.15.0-112-generic
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
$ grep CONFIG_IOMMU_DEFAULT_DMA_STRICT /boot/config-5.15.0-112-generic
I also checked older jammy kernels and strict was always yes.

So this has obviously never changed from jammy to noble!

Have you modified/tweaked the setting manually on your side before testing on jammy?!

[Frank Heimes (fheimes)]

<Looks like Kleber and me commented in parallel ...>

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-07-01 04:49 EDT-------
For reference the the CONFIG_DMA_DEFAULT_DMA setting is relevant to s390 only
since upstream commit c76c067e488c ("s390/pci: Use dma-iommu layer").

This commit also includes the following hunk to default to CONFIG_DEFAULT_DMA_LAZY on s390x. Though I guess this wouldn't overwrite a pre-existing config value:

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index cd6727898b11..3199fd54b462 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -91,7 +91,7 @@ config IOMMU_DEBUGFS
choice
prompt "IOMMU default domain type"
depends on IOMMU_API
- default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64
+ default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64 || S390
default IOMMU_DEFAULT_DMA_STRICT
help
Choose the type of IOMMU domain used to manage DMA API usage by

The line has been modified since to drop IA64 but does contain S390 on current
upstream as well.

[Frank Heimes (fheimes)]

Hi Niklas, thanks for the backgrounds.

So to sum it up:

The Ubuntu kernel config options for IOMMU_DEFAULT_DMA_LAZY and IOMMU_DEFAULT_DMA_STRICT haven't changed since jammy/22.04:

5.15.0-1.1 to 5.15.0-115.125
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
# CONFIG_IOMMU_DEFAULT_DMA_LAZY is not set

6.8.0-6.6 to 6.8.0-38.38
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
# CONFIG_IOMMU_DEFAULT_DMA_LAZY is not set

But with the introduction of
c76c067e488c s390/pci: Use dma-iommu layer
which moves to a different dma-iommu implementation and
92bce97f0c34 s390/pci: Fix reset of IOMMU software counters
(both available since 6.7(rc-1)
the IOMMU_DEFAULT_DMA_LAZY kernel config option should be set to 'yes' by default for s390x.

Does CONFIG_IOMMU_DEFAULT_DMA_STRICT need to be set to No on top? I don't think so - if lazy is 'yes'.

So I believe we can go with 'IOMMU_DEFAULT_DMA_LAZY=y' for s390x only and should be good again.

I assume that such situations happen not very often,
so it would be ideal if IBM could give us a quick heads-up in such cases (where a kernel config default value for s390x is changed upstream), so that we can double check and potentially take over changes into the Ubuntu config by hand.

[Frank Heimes (fheimes)]

Looks like it's not sufficient to have IOMMU_DEFAULT_DMA_LAZY=y only
in addition IOMMU_DEFAULT_DMA_STRICT=n seems to be needed on top (otherwise the build failed for me).
(Which I think is not super great upstream, one option should be fine -- with the two kernel options, there are now two unspecific cases: both n and both y - anyway ...)

I did a test build in PPA here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2071471
would be great if this can be tried.

[Frank Heimes (fheimes)]

Modification was sent to kernel teams mailing list:
https://lists.ubuntu.com/archives/kernel-team/2024-July/thread.html#151928

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-07-03 05:24 EDT-------
Interesting with "make defconfig" on upstream I get

?
# CONFIG_IOMMU_DEFAULT_DMA_STRICT is not set
CONFIG_IOMMU_DEFAULT_DMA_LAZY=y
?

And that works well.

[Frank Heimes (fheimes)]

Well, 'make defconfig' is what I cannot use, since we have the Ubuntu specific kernel options (and tools).
But since "not set" defaults to no, I guess it should be okay to explicitly set CONFIG_IOMMU_DEFAULT_DMA_STRICT=n.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-07-03 06:53 EDT-------
(In reply to comment #11)
> Looks like it's not sufficient to have IOMMU_DEFAULT_DMA_LAZY=y only
> in addition IOMMU_DEFAULT_DMA_STRICT=n seems to be needed on top (otherwise
> the build failed for me).
> (Which I think is not super great upstream, one option should be fine --
> with the two kernel options, there are now two unspecific cases: both n and
> both y - anyway ...)
>
> I did a test build in PPA here:
> https://launchpad.net/~fheimes/+archive/ubuntu/lp2071471
> would be great if this can be tried.

I gave this a quick test on a z/VM with a RoCE VF.

root@redacted:~# uname -a
Linux redacted 6.8.0-38-generic #38~lp2071471-Ubuntu SMP Wed Jul 3 07:09:08 UTC 2024 s390x s390x s390x GNU/Linux
root@redacted:~# cat /sys/bus/pci/devices/18d1\:00\:00.0/iommu_group/type
DMA-FQ

So this uses the DMA Flush Queue mechanism as it should. Thanks!

[Frank Heimes (fheimes)]

Many thanks for confirming, Niklas!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.8.0-40.40 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-07-19 02:38 EDT-------
I verified that with 6.8.0-40-generic from -proposed the degradation is indeed gone.
Thanks a lot!

[Frank Heimes (fheimes)]

Thanks Barbara for the verification!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-gt-tdx/6.8.0-1009.10+tdx1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-ibm-gt-tdx' to 'verification-done-noble-linux-ibm-gt-tdx'. If the problem still exists, change the tag 'verification-needed-noble-linux-ibm-gt-tdx' to 'verification-failed-noble-linux-ibm-gt-tdx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.8.0-40.40

---------------
linux (6.8.0-40.40) noble; urgency=medium

  * noble/linux: 6.8.0-40.40 -proposed tracker (LP: #2072201)

  * FPS of glxgear with fullscreen is too low on MTL platform (LP: #2069380)
    - drm/i915: Bypass LMEMBAR/GTTMMADR for MTL stolen memory access

  * a critical typo in the code managing the ASPM settings for PCI Express
    devices (LP: #2071889)
    - PCI/ASPM: Restore parent state to parent, child state to child

  * [UBUNTU 24.04] IOMMU DMA mode changed in kernel config causes massive
    throughput degradation for PCI-related network workloads (LP: #2071471)
    - [Config] Set IOMMU_DEFAULT_DMA_STRICT=n and IOMMU_DEFAULT_DMA_LAZY=yes for
      s390x

  * UBSAN: array-index-out-of-bounds in
    /build/linux-D15vQj/linux-6.5.0/drivers/md/bcache/bset.c:1098:3
    (LP: #2039368)
    - bcache: fix variable length array abuse in btree_iter

  * Mute/mic LEDs and speaker no function on EliteBook 645/665 G11
    (LP: #2071296)
    - ALSA: hda/realtek: fix mute/micmute LEDs don't work for EliteBook 645/665
      G11.

  * failed to enable IPU6 camera sensor on kernel >= 6.8: ivsc_ace
    intel_vsc-5db76cf6-0a68-4ed6-9b78-0361635e2447: switch camera to host
    failed: -110 (LP: #2067364)
    - mei: vsc: Don't stop/restart mei device during system suspend/resume
    - SAUCE: media: ivsc: csi: don't count privacy on as error
    - SAUCE: media: ivsc: csi: add separate lock for v4l2 control handler
    - SAUCE: media: ivsc: csi: remove privacy status in struct mei_csi
    - SAUCE: mei: vsc: Enhance IVSC chipset stability during warm reboot
    - SAUCE: mei: vsc: Enhance SPI transfer of IVSC rom
    - SAUCE: mei: vsc: Utilize the appropriate byte order swap function
    - SAUCE: mei: vsc: Prevent timeout error with added delay post-firmware
      download

  * failed to probe camera sensor on Dell XPS 9315: ov01a10 i2c-OVTI01A0:00:
    failed to check hwcfg: -22 (LP: #2070251)
    - ACPI: utils: Make acpi_handle_path() not static
    - ACPI: property: Ignore bad graph port nodes on Dell XPS 9315
    - ACPI: property: Polish ignoring bad data nodes
    - ACPI: scan: Ignore camera graph port nodes on all Dell Tiger, Alder and
      Raptor Lake models

  * Update amd_sfh for AMD strix series (LP: #2058331)
    - HID: amd_sfh: Increase sensor command timeout
    - HID: amd_sfh: Improve boot time when SFH is available
    - HID: amd_sfh: Extend MP2 register access to SFH
    - HID: amd_sfh: Set the AMD SFH driver to depend on x86

  * RFIM and SAGV Linux Support for G10 models (LP: #2070158)
    - drm/i915/display: Add meaningful traces for QGV point info error handling
    - drm/i915/display: Extract code required to calculate max qgv/psf gv point
    - drm/i915/display: extract code to prepare qgv points mask
    - drm/i915/display: Disable SAGV on bw init, to force QGV point recalculation
    - drm/i915/display: handle systems with duplicate psf gv points
    - drm/i915/display: force qgv check after the hw state readout

  * Update amd-pmf for AMD strix series (LP: #2058330)
    - platform/x86/amd/pmf: Differentiate PMF ACPI versions
    - platform/x86/amd/pmf: Disable debugf...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/6.8.0-1012.14 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure' to 'verification-done-noble-linux-azure'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure' to 'verification-failed-noble-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi/6.8.0-1009.10 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-raspi' to 'verification-done-noble-linux-raspi'. If the problem still exists, change the tag 'verification-needed-noble-linux-raspi' to 'verification-failed-noble-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-riscv-6.8/6.8.0-40.40.1~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-riscv-6.8' to 'verification-done-jammy-linux-riscv-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-riscv-6.8' to 'verification-failed-jammy-linux-riscv-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi-realtime/6.8.0-2008.8 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-raspi-realtime' to 'verification-done-noble-linux-raspi-realtime'. If the problem still exists, change the tag 'verification-needed-noble-linux-raspi-realtime' to 'verification-failed-noble-linux-raspi-realtime'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency/6.8.0-40.40.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-lowlatency' to 'verification-done-noble-linux-lowlatency'. If the problem still exists, change the tag 'verification-needed-noble-linux-lowlatency' to 'verification-failed-noble-linux-lowlatency'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-6.8/6.8.0-1010.10~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-ibm-6.8' to 'verification-done-jammy-linux-ibm-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-ibm-6.8' to 'verification-failed-jammy-linux-ibm-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-08-09 05:10 EDT-------
As pointed out before, I consider the verification done. Bug can be closed.

[Frank Heimes (fheimes)]

Many thanks for the successful verification, Barbara!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency-hwe-6.8/6.8.0-40.40.1~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.8' to 'verification-done-jammy-linux-lowlatency-hwe-6.8'. If the problem still exists, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.8' to 'verification-failed-jammy-linux-lowlatency-hwe-6.8'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!