login.html needed with debug.ini in WSGI?

[Martijn Faassen]

Apparently login.html should be used to login if debug.ini is used with paster. I do not know why. I don't understand why this should be so, and it'd be a major cause for people tripping up.

We should add a test that demonstrates this behavior somewhere, probably in grokui.admin.

[Uli Fouquet (uli-gnufix)]

I think this is caused by the way authentication works: in 'normal' use basic-auth is enabled, which is triggered, when someone requests a page she is not authenticated to get. Somewhere in the publishing process (don't know where) then an exception is raised, which might be caught by the Zope publisher(?) and is turned into a basic-auth request by changing the HTTP status of the response. So an exception happens, but the user does not see it (instead the basic-auth-window pops up).

When WSGI comes in, then you can let exceptions of any kind be handled by another middleware in the pipeline which is, what deploy.ini does: it starts a separate middleware which cares for exceptions during requests.

When both things come together (basic-auth + debugger-middleware), then also this normally not seen Unauthorized exception is passed to the middleware and you get the debugging screen instead of the basic-auth pop-up. This behaviour is correct from the logical point of view.

What Michael recommended is to use the login page in that case, which as a public view does not raise any exceptions but also sets the cookies according to the entered credentials. Therefore it works, but basic-auth does not.

So much for the problem, as far as I understood it. Michael, you might be able to tell more about the dirty details. Is there a possibility to bypass the middleware for certain requests?

[Sylvain Viollon (thefunny)]

It is what I understood as well. I think it will a nicer feature that in the debugger middleware to be able (if it's not already the case) to ignore some exceptions. But that's implies changes in Paster, as far as I known, so a new release of it. We should propose that to the Paster people, maybe send a patch to them.

We might have other case where we want ignore such exceptions, specially if two layers use them to communicates together.

[Uli Fouquet (uli-gnufix)]

Thanks for the comment, Sylvain!

Rethinking the problem it looks strange to me, that the two use-cases of authentication behave so different.

In the basic-auth case it looks to me that the raised exception is something like a signal to some 'plugin' or whatever. I vaguely remember some statement from Christian Theune, that things couldn't be handled differently (say: without raising an exception) with basic-auth, but I can't remember, what the reason was, actually. Wouldn't it be cleaner when also basic-auth authentication would be triggered without a raised exception?

So, before telling paster how to behave more subtle with exceptions I would like to see, whether it is not possible to tackle this problem inside the Zope publishing machinery. An advanced paster-side exception-handling could be useful anyway as you pointed out for other usecases.

What do you think?

[Martijn Faassen (faassen)]

I think any change to the publisher machinery will be a bigger one
than telling Paster's exception handling middleware that some
exceptions are to be ignored by it.

On the longer term we could explore changes to the publication
machinery of Zope though.

[Sylvain Viollon (thefunny)]

I agree that looking at the publisher is far more complicated that telling to Paster to ignore it.

However, we can still first tell that to Paster, and let us have more time to figure a clean way in the publisher to remove that exception. I don't think it's the only exception used, and I don't think we can fix it in one night (otherwise it would have been implemented without exception probably, at least at the place we got it). And people are using it as a feature, raise Unauthorized in your code then you don't want the current user to be there (so getting ride of it might be catching it before it get back to the WSGI layer).

And if we say that that issue block the release of Grok 1.0, well, we might prefer the first solution instead of releasing Grok in 6 months.

(I don't say that we should not look at the publisher at all, but we can do both).

[Martijn Faassen (faassen)]

Yes, agreed with Sylvain: let's make paster ignore any "special"
exceptions from Zope such as Unauthorized that aren't really to be
treated as errors. That's a 1.0 topic. Fixing Zope's publication
machinery is something for later (see also Shane's work).

[Michael Haubenwallner (d2m)]

Same debug.ini works fine with zopeproject.
It simply calls/redirects to @@login.html.

Maybe we could look into what exactly happens inside grokui.admin?

[Uli Fouquet (uli-gnufix)]

Okay, as discussed on the list I will fix grokui.admin not to emit an Unauthorized exception when the ZODB root is called.

Michael, may I take this bug over so that it shows up in my ToDo list?

[Michael Haubenwallner (d2m)]

Uli sure, please do

[Uli Fouquet (uli-gnufix)]

Thanks, Michael! Your hint with `zopeproject` was valuable.

[Martijn Faassen (faassen)]

On Thu, Mar 5, 2009 at 4:27 PM, Uli Fouquet <email address hidden> wrote:
> Okay, as discussed on the list I will fix grokui.admin not to emit an
> Unauthorized exception when the ZODB root is called.

I still don't get it. I don't agree with this approach.

Unauthorized exceptions will be raised by all sorts of applications
and they *shouldn't* result in brokenness. Instead we should remove
the exception traceback formatter if it leads to problems and we can't
fix it.

[Martijn Faassen (faassen)]

Please roll back the fix to grokui.admin. I think it's a symptom of a
problem with the WSGI setup that we should fix

[Uli Fouquet (uli-gnufix)]

Okay, I reopen the bug to have it invesitgated further. I am not sure about the rollback, however, because a more intelligent handling of requests in the index view might be okay. I'll update (rollback) the grokui.admin docs anyway.

Beside this I think there is nothing wrong with the approach to let users slip through that call the index page with the WSGI debugger enabled. Let's call it a convenience feature. There is nothing to stop users from creating their own debug.ini if we do not deliver one with grokproject.

[Martijn Faassen (faassen)]

My main point is that raising Unauthorized shouldn't end up in the
debugger but instead should have the behavior it had pre-WSGI, which
as far as I know is trigger user authentication.

I understand that the reaction of the system (with the debugger
installed) is now different correct?

[Uli Fouquet (uli-gnufix)]

Don't know, whether the system reaction is really that different from pre-WSGI. It's difficult to say, because there was nothing like a pipeline before. When I needed a debugger in pre-WSGO times, I used pdb, triggered somewhere in the code.

But I really would like to learn what's going on there in detail and will look into that.

If someone has a hint for other 'pre-WSGI' debuggers with Zope I'd appreciate that!

[Martijn Faassen (faassen)]

The pre-WSGI story didn't have a debug.

But when Unauthorized is raised, the debugger isn't supposed to kick
in at all, is it? It's just supposed to authorize the user. If that is
currently broken, then we need to fix this.

To compare you could create an old-style grok project and see what
happens when you raise Unauthorized.

[Uli Fouquet (uli-gnufix)]

I'll try to explain the difference.

First of all: there seems to be no (important) difference in how WSGI and non-WSGI apps are handled (which becomes obvious, if you think of the deploy.ini setup, which does not end with exceptions)

Both, the deploy.ini setup and debug.ini setup call the Zope publisher differently.

deploy.ini: zope.publisher.publish(request, handle_errors=True)

debug.ini: zope.publisher.publish(request, handle_errors=False)

So, what happens in publish then? I pasted the relevant piece here:

  http://python.pastebin.com/m100c4df9

As you can see, after the publication handles the exception (i.e. finds a view for it, sets status code, etc.) in one case (if handle_errors is False) the exception is reraised while in the other case it is not (line 18).

In all cases a view is found and the status set correctly (for Unauthorized at least), but in the debug case the next element in the pipeline (the debugger middleware) takes over afterwards and does not post back the already finished response, but does its debugger stuff.

In the regular-mode instead the exception (turned into HTML) is returned to the browser. In both cases the exception is turned into HTML but only in regular use (handle_errors==True), this page is sent back as response to the browser.

So much for a first analysis.

Where can we hook into it now?

The debugger middleware might be too late in processing. It could check, whether the exception it gets is an Unauthorized, but it would not get the already finished response object, etc. It sounds very crude to me to handle things on this level.

One could instead check, whether the publisher itself could handle exceptions smarter: I'd expect from it to only reraise exceptions in case of 'Internal Server Error'. For this one had to modify the publish function and lots of side effects may arise. At least it would be nice if one could tell the publisher, which exceptions to reraise: none, all or only important ones (internal server errors). This looks like the right location for me to fix.

One could also think of a workaround in the Grok publisher: it should be able to catch the most Unauthorized exceptions and could turn them into an exception view (what the publisher normally does). This sounds also much like a workaround with unwanted side effects.

I tend to opt for fixing this in zope.publisher.

But well, maybe someone has got an idea?

[Uli Fouquet (uli-gnufix)]

I'll have another look into the whole paste stuff. Maybe there is cool way to go. Please give one day.

[Uli Fouquet (uli-gnufix)]

I looked into paste, repoze and z3c.evalexception. The result is, that it is possible to let certain exceptions slip through, but they had to be handled by another middleware then, which also wouldn't get the result (HTTP response) the Zope publisher created before.

When the middleware catches an exception, it won't get the response.

To sum it up: it looks like the only possiblity to get the usual Zope-created response from an encapsulating middleware like the debugger would be to snoop around in the call-stack of the exception, likewise 'going back into Zope'.

I don't think this is a real option.

[Martijn Faassen (faassen)]

Thanks very much for doing all this research, Uli, I'm sorry I was a
bit short earlier. I'll review your answers in more detail later, but
I take it that basically, because we want to use the exception
middleware, we cannot let Zope handle its own exceptions, and thus
Unauthorized is treated like all the others.

[Uli Fouquet (uli-gnufix)]

Right. Unauthorized is an exception like any other for the publisher in that respect.

It might be worth to note, that Shane's efforts towards a WSGI oriented publisher also consider a new pipeline var 'zope.response' to be forwarded in the pipeline.

If we had this available in the debugger middleware, we could easily configure the debugger to return the Zope generated Unauthorized page in case of Unauthorized. That would solve our problems in the right way, I think.

We might should support Shane in his cool project, although it won't help fixing the bug until 1.0a1.

[Reinout van Rees (reinout)]

Selected for europython

[Uli Fouquet (uli-gnufix)]

Thanks to Martijn we made some real progress on that issue during Europython.

The solution is now targeted in zope.publisher, blessed by Martijn and Christian Theune and works like this:

We introduce a new interface 'IReRaiseException' in zope.publisher, that certain exception types can be adapted to.

The adapter should return 'True' or 'False' upon being called depending on whether we want this exception type to reach any middleware debugger on publishing or not.

The changes are now introduced to the zope.publisher trunk and then backported to the zope.publisher version currently used by Grok. Afterwards we will introduce an appropriate adapter into Grok that will deny reraising Unauthorized exceptions so that people can still login with basic-auth when using the WSGI debugger middleware.

[Uli Fouquet (uli-gnufix)]

Fix is incorporated in zope.publisher trunk and 3.4 maintenance branch.

Added an appropriate adapter in grok branch 'ulif-paster-loginhtml' (with tests). To make that really work, a maintenance release of zope.publisher 3.4 has to be done. I have no permissions to do so.

Michael, could you do it?

[Michael Haubenwallner (d2m)]

Ulif, i just granted you owner role on zope.publisher

[Uli Fouquet (uli-gnufix)]

Thanks a lot, Michael! Just released and updated branch.

[Uli Fouquet (uli-gnufix)]

Sorry, this fix is not released yet, only committed.

[Reinout van Rees (reinout)]

Note to jkolman: the 3.4.9 zope.publisher pin (http://pypi.python.org/pypi/zope.publisher/3.4.9) has already been done in grok trunk's version.cfg.