[SRU] trac crashes while importing a library in Ubuntu 20.04 (ImportError: cannot import name 'soft_unicode' from 'jinja2.utils')

[Lars Becker]

[ Impact ]

trac is completely unusable as it will fail with the below error whenever any user tries to run "tracd" or "trac-admin":

Traceback (most recent call last):
  File "/usr/bin/tracd", line 33, in <module>
    sys.exit(load_entry_point('Trac==1.5.3', 'console_scripts', 'tracd')())
  File "/usr/bin/tracd", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/trac/web/standalone.py", line 35, in <module>
    from trac.web.auth import BasicAuthentication, DigestAuthentication
  File "/usr/lib/python3/dist-packages/trac/web/auth.py", line 29, in <module>
    from trac.web.chrome import Chrome, INavigationContributor
  File "/usr/lib/python3/dist-packages/trac/web/chrome.py", line 40, in <module>
    from trac.mimeview.api import RenderingContext, get_mimetype
  File "/usr/lib/python3/dist-packages/trac/mimeview/__init__.py", line 14, in <module>
    from trac.mimeview.api import *
  File "/usr/lib/python3/dist-packages/trac/mimeview/api.py", line 70, in <module>
    from trac.resource import Resource
  File "/usr/lib/python3/dist-packages/trac/resource.py", line 21, in <module>
    from trac.util.presentation import classes
  File "/usr/lib/python3/dist-packages/trac/util/presentation.py", line 26, in <module>
    from jinja2.utils import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'jinja2.utils' (/usr/lib/python3/dist-packages/jinja2/utils.py)

[ Test Plan ]

* Simple test:
  - Execute "tracd --help" or "tracd"
  - Execute "trac-admin --help" or "trac-admin"
  - If its not fixed then both the commands will fail as the error above
  - With fixed package both the commands will show the help message.

* More comprehensive test: (execute the following as root)
  - mkdir -p /var/lib/trac
  - trac-admin /var/lib/trac/myproject initenv
  - Enter some project name when prompted
  - Leave default sqlite DB (just press enter)
  - echo -n "tracadmin:example.com:" > /var/lib/trac/myproject/users.htdigest
  - trac-admin /var/lib/trac/myproject permission add tracadmin TRAC_ADMIN
  - tracd --port 8080 --auth=*,/var/lib/trac/myproject/users.htdigest,example.com /var/lib/trac/myproject
  - open http://127.0.0.1:8080 in browser, it should show the project name and clicking on that should show the welcome screen

[ Where problems could occur ]

It is not changing any part of the code, its only importing the module with the new name and so there is minimum chance of any regression.

[racb] We're adjusting imports of soft_unicode/soft_str. Problems might manifest in module imports, or in the assumption that one of these is an exact replacement of the other.

[ Other Info ]

- The Ubuntu patch is a combination of portions of two upstream but related changes.
- One of the upstream changes mentions about an AssertionError for jinja, so it will be great if this can be tested by a real user of trac. The testplan only checks the server starts and is working.

[ Original Bug Description ]

Trac 1.5.3 which comes with Ubuntu 20.04 crashes because of problems with a library. The problem was fixed upstream: https://trac.edgewall.org/ticket/13404

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sudip Mukherjee (sudipmuk)]

I can reproduce the problem in Jammy. Also tested and confirmed that Mantic does not show the problem.
This package is not present in Focal, so not applicable.

[Sudip Mukherjee (sudipmuk)]

Debdiff attached for Jammy. I am not really sure if two upstream urls are allowed according to the dep3 header or not, but Lintian did not give any error and I also checked with "patchedit check". But if its not allowed I can break the patch into two different patches for each upstream url.

[Robie Basak (racb)]

This looks good. Thanks!

I added a line to "Where problems could occur".

I suggest changing the changelog entry to simply:

  * Fix crash on startup (LP: #2024325).

Saying that we backported the fix seems superfluous to me. The people who read this are likely to be non-technical, and those technical can always review the diff and/or dep3 headers to understand the code changes if they're interested. But what we do want to do is make it clear to users that we are actually fixing the bug that they have experienced. "error with library import" sounds like something that could just be an edge case and describes the root cause that the user might not know as opposed to the actual user experience. "crash on startup" makes it clear that if a user was affected by that then that's what we're addressing by this fix.

Please let me know if you're OK with this fix and I'll go ahead and sponsor. After that, it will await review by the SRU team.

[Sudip Mukherjee (sudipmuk)]

Thanks. Thats sounds ok, do you want me to upload new debdiff with the updated changelog?

[Robie Basak (racb)]

No need - I can alter it trivially :)

[Robie Basak (racb)]

Uploaded. Thanks!

[Andreas Hasenack (ahasenack)]

Looking at the patch:

Description: make compatible with markupsafe 2.1.0
Origin: backport, https://trac.edgewall.org/changeset/17575 and https://trac.edgewall.org/changeset/17543
Bug-Ubuntu: https://launchpad.net/bugs/2024325
Last-Update: 2024-01-29
---

--- trac-1.5.3+dfsg.orig/trac/util/html.py
+++ trac-1.5.3+dfsg/trac/util/html.py
@@ -25,6 +25,10 @@ from html import entities
 from html.parser import HTMLParser

 from markupsafe import Markup, escape as escape_quotes
+try:
+ from markupsafe import soft_str as soft_unicode
+except ImportError:
+ from markupsafe import soft_unicode

 try:
     from babel.support import LazyProxy
--- trac-1.5.3+dfsg.orig/trac/util/presentation.py
+++ trac-1.5.3+dfsg/trac/util/presentation.py
@@ -23,11 +23,11 @@ import re

 from jinja2 import Markup, Undefined, contextfilter, evalcontextfilter
 from jinja2.filters import make_attrgetter
-from jinja2.utils import soft_unicode

 from trac.core import TracError
 from .datefmt import to_utimestamp, utc
-from .html import Fragment, classes, html_attribute, styles, tag
+from .html import (Fragment, Markup, classes, html_attribute, soft_unicode,
+ styles, tag)
 from .text import javascript_quote

 __all__ = ['captioned_button', 'classes', 'first_last', 'group', 'istext',

a) In presentation.py, you are leaving the "from jinja2 import Markup" bit, and again, later down, importing it again via "from .html import Markup", which is a change you added.

That comes from https://trac.edgewall.org/changeset/17543, but note that that diff also *removed* the Markup import from jinja2:
-from jinja2 import Markup, Undefined, contextfilter, evalcontextfilter
+from jinja2 import Undefined, contextfilter, evalcontextfilter

Could you please review these changes, and the import changes overall in the patch?

b) If I understood this correctly, this is the big import change: soft_unicode is no longer imported from jinja2.utils, but now comes from markupsafe *either* as soft_unicode, or renamed from soft_str. What's the story here? What guarantees do we have that these modules are accurate replacements? Did jinja2.utils remove soft_unicode in some version, and if yes, in which?

[Sudip Mukherjee (sudipmuk)]

> a) In presentation.py, you are leaving the "from jinja2 import Markup" bit, and again, later down, importing it again via "from .html import Markup", which is a change you added.
>
> That comes from https://trac.edgewall.org/changeset/17543, but note that that diff also *removed* the Markup import from jinja2:
> -from jinja2 import Markup, Undefined, contextfilter, evalcontextfilter
> +from jinja2 import Undefined, contextfilter, evalcontextfilter
>
>
> Could you please review these changes, and the import changes overall in the patch?

Looks like I messed up here. I added the old debdiff and forgot to add the final debdiff which had this change. Adding the debdiff here for your reference, and will be great if you can reject the upload please.

>
> b) If I understood this correctly, this is the big import change:
> soft_unicode is no longer imported from jinja2.utils, but now comes from
> markupsafe *either* as soft_unicode, or renamed from soft_str. What's
> the story here? What guarantees do we have that these modules are
> accurate replacements? Did jinja2.utils remove soft_unicode in some
> version, and if yes, in which?

Yes, jinja2.utils removed soft_unicode in v3.0.0 and you can see the change at:
https://github.com/pallets/jinja/commit/b0015c72d5acbf93b9d99a1ce6167889338db85b#diff-99868302c0e75a5c181912aab0e396266e2ce25fd7b0b8f29684df74eef49984L718

The original function in jinja2.utils was just doing:
def soft_unicode(s):
    from markupsafe import soft_unicode
    return soft_unicode(s)

So, it was basically just returning soft_unicode from markupsafe.

From the pov of markupsafe, soft_unicode has been renamed as soft_str and you can see the change at:
https://github.com/pallets/markupsafe/blob/2.0.1/src/markupsafe/_native.py#L70

So, these should be accurate replacements.

[Robie Basak (racb)]

Re-sponsored.

[Andreas Hasenack (ahasenack)]

Initially I thought the patch was still more complicated than it had to be. After all, we just needed to switch the import of soft_unicode/soft_str from jinja2 to markupsafe. But the other part of the fix also changes the import of Markup. Before, html.py was importing it from markupsafe, and presentation.py was importing Markup from jinja2. That's two different sources for the same module, and while it might work, it's confusing and down the road could stop working. Right now jinja2.utils is exporting markupsafe.Markup, so essentially they are the same. But confusing, and jinja2 could change that some point in the future.

All good, I just wanted to record it here in case someone came along and also wondered why not just fix the soft_unicode/soft_str import, and leave the rest alone.

[Andreas Hasenack (ahasenack)]

Oh, never mind my suggestion of a new test plan, you have a second more comprehensive test :)
I'll delete my comment

[Andreas Hasenack (ahasenack)]

Hello Lars, or anyone else affected,

Accepted trac into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/trac/1.5.3+dfsg-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lars Becker (lrsbckr)]

Thanks Andreas. We needed a maintainable solution and containerized our trac instance. It's no longer running under Ubuntu. I'm afraid I can't test the solution.

[Sudip Mukherjee (sudipmuk)]

I can confirm that the trac package in jammy-proposed has fixed the bug for me.

Test done:

Test-1: "tracd --help" and "trac-admin --help" both worked and displayed the help message.

Test-2: Followed the testplan as mentioned in the "More comprehensive test:" and confirmed that the project is viewable on browser.

Test result: the errors with trac has been fixed.

Package tested:

$ dpkg -l trac
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=====================-============-=========================================================================
ii trac 1.5.3+dfsg-1ubuntu0.1 all Enhanced wiki and issue tracking system for software development projects

[David Guilbeau (coderdave)]

The trac/jammy-proposed,now 1.5.3+dfsg-1ubuntu0.1 package fixed trac-admin crashing for me.

trac-admin used to crash and give the error:
ImportError: cannot import name 'soft_unicode' from 'jinja2.utils' (/usr/lib/python3/dist-packages/jinja2/utils.py

[Launchpad Janitor (janitor)]

This bug was fixed in the package trac - 1.5.3+dfsg-1ubuntu0.1

---------------
trac (1.5.3+dfsg-1ubuntu0.1) jammy; urgency=medium

  * Fix crash on startup (LP: #2024325)

 -- Sudip Mukherjee <email address hidden> Mon, 29 Jan 2024 21:16:53 +0000

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for trac has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.