Improve Firefox/XULRunner Support

[Thomas Berezansky]

The initial commit message is reproduced below the link to the branch.

This is a significant enough change that I feel that more than just one person should review it. It has been being used on Jason Stephenson's development machine for a while now, though.

As a note, I included the work from the following bugs while building/testing the branch for this one:

Bug #921740 - External sites can be opened in the staff client

This has a component that needed to be re-written to support later XULRunner versions, and would *not* have worked in Extension mode in any satisfactory manner.

In addition, this covers Bug #628378 (Remote XUL support going away in Gecko 2) and may help correct Bug #491537 (xulrunner 1.9 (and Firefox 3.5) prints garbage to certain printers) by introducing a significantly later XULRunner version.

In the interest of documenting what has and has not been tested, I touched on the following:

Basic Circ Functions, Basic Admin functions (a number of config menus in particular), Automatic Updates, Installer Building, basic receipt printing, Vandelay loads (but I didn't test much beyond that), and minimal reporter testing. I touched on MARC View/Edit when errors were being generated there. Generally I bounced through random parts of the system I knew enough about to check things in, looking for anything that wasn't working, and tried to hit interfaces I didn't know enough about to ensure that they at least appeared to load without errors. I don't have a comprehensive list of them, though.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/tsbere/new_xulrunner

Commit message:

Improve Firefox/XULRunner Support

XULRunner/Firefox 4+ change the rules significantly, and to support them
significant changes have to be made in Evergreen.

This commit covers a number of things:

1 - Remote XUL Issues

Later versions of XULRunner disable support for Remote XUL without using a
whitelist entry, and whitelisting may stop working eventually as well. The
solution here is the addition of a new protocol wrapper that allows Remote
XUL to act like Local XUL.

The wrapper is oils:// and acts like HTTPS for the most part. The "host" the
client connects to is always "remote", however, and use of the wrapper has
multiple effects.

The first of these effects is the enabling of Remote XUL content, with as
close to the Local XUL security level as I could manage. This means that
nearly anything Local XUL can do, Remote XUL should be able to do too.

The second of these effects is the disabling of JavaScript-visible cookies.
XUL doesn't have cookies, and as such a side effect of the wrapper allowing
remote content to appear to be local content is the lack of cookies. This
does *not* affect server side visibility of cookies, however, so things like
TPac can still see cookies. This required fixing a lot of places that were
built to read data like authtokens out of cookies.

The third of these effects is the "hiding" of the hostname for anything
going through the wrapper. This prevents oils_persist from saving things
correctly. I thus made oils_persist check for the wrapper and use the data
stash to get the hostname instead. I suspect that we can get rid of the
location.hostname check entirely at this point, but have not done so.

The wrapper also sets a request header of OILS-Wrapper, value of 'true'.
That allows the server to know that the request came through the wrapper. I
used that ability to have TPac go into "Staff" mode when the wrapper is in
use. This will enhance extension use, where the same authtoken will be seen
by both the staff client and Firefox browsing.

I willi also note that I rigged the wrapper to be easily adjusted for a
second, hard-coded host for selfcheck use, using oils://selfcheck instead of
oils://remote, that could be very easily turned into a minimal Firefox
extension. That would enable the selfcheck code to have permission to work
with printers and such without enablePrivilege (see next section).

2 - enablePrivilege Deprecation

The enablePrivilege security model has been removed and no longer functions,
and the staff client depended heavily upon it.

The Remote XUL fix provided us with a way to fix the enablePrivilege issues,
however. Because the Remote XUL has the Local XUL security context it
doesn't need enablePrivilege anymore. However, because enablePrivilege no
longer functions for those things *not* loaded through the wrapper some
things had to be changed.

For one, because if you are loaded through the wrapper you can't see
cookies, and if you can see cookies you can't do "XUL" things, the "XUL"
cookie has been removed. However, the wrapper can be detected by the
protocol you are using, so I added protocol checking to the isXUL checks I
know of. I am unsure about what, if anything, to do about the IAMXUL flag
that is set by the browser code, though.

For two, the JavaScript OPAC needs to be passed through the wrapper for some
staff functions to work currently. This has the additional side effect of
effectively forcing SSL mode for JSPac regardless of other settings.

For three, I went ahead and removed all enablePrivilege calls I could find
outside of the selfcheck code. As they no longer work they were just extra
cruft.

3 - Component changes

Component loading has changed significantly, and the easiest way to
accommodate the changes was to re-write our components. The command line
handler and force external components (the latter from another branch of
mine) were simple enough, especially as they aren't interacted with by other
code in the system (and as I wrote them I understood them anyway). While I
was at it I made the force external component more extension-friendly.

The data stash, however, was using a hack to accomplish what a singleton
component could do more easily. That and I couldn't get it to function as
it was. My solution was to re-write it as a singleton, removing the
interface definition that was not needed. As I changed how it functioned so
significantly I had to update all calls to it.

In addition, I hid a window loading routine in the data stash component for
easier loading of a *single* main Evergreen window from the Firefox
overlays.

4 - Extension changes

First off, Venkman, the DOM Inspector, and Chrome List are not guaranteed
compatible with XULRunner anymore. As such, I have removed the out of date
versions. Instead, I have added routines for loading Venkman and the DOM
Inspector as Extensions. This keeps them easy to update to later versions,
and makes us play much nicer when we are an extension. Also, now the
buttons/menu items for loading these interfaces disable themselves if the
extensions don't appear to be loaded.

A side effect of that was that Windows installers compressed with zlib were
unhappy. I switched to solid lzma, which in theory will get us smaller file
sizes too.

Secondly, Extensions have changed significantly in Firefox 4+, and Evergreen
had to be adjusted to deal with some of the changes. I adjusted the overlay
to take into account the new application menu, and I re-wrote the hotkey set
loading routines to deal with the fact that extensions don't get extracted
by default.

5 - Staff Client Build Process

XULRunner has some changes that needed accounting for, such as a root level
chrome.manifest being needed. Extension mode, as mentioned above, needed
tweaks as well, and other issues were run into along the way. Both XULRunner
and Firefox version support has been extended to 12.* and the base XULRunner
download has been bumped to 12.0.

In addition to the changes needed to support later XULRunner versions, I
also included alternate application name support and support for OS-specific
branding folders. Mainly because I ended up using all of them in testing,
and decided to leave them in afterwards.

6 - Miscellaneous fixes

MARC View was building a data URL and dumping it into an attribute without
escaping it. Apparently this made XULRunner unhappy, so I added in escaping.

7 - Miscellaneous tweaks

In addition to everything else, while I was running through my various tests
I changed a few other things.

I enabled the navigation buttons on the OPAC interface embedded in the
patron overview for hold placement. With Remote XUL having the same security
context as Local XUL they now work.

I removed all non-SSL authtoken cookie assignments I could find, and flagged
all the SSL ones I could find as being "secure" cookies. On the subject of
cookies, I also told the logoff code to not erase every cookie, as those
running in extension mode might dislike all cookies going away. I know that
during my testing I disliked it, anyway.

I removed the code that messes with the user agent. Extensions are basically
not allowed to do that, and I don't think we need, or should rely on, a
modified user agent elsewhere, so adding a check specifically for when we
are not an extension didn't make a lot of sense to me.

I also made it so that the automatic login command line options will work
after a logoff (but they won't trigger a logoff) during a testing run where
I was logging off, clearing cache, and logging in again repeatedly.

To deal with CSS changes I changed most (but not all) popup elements to
menupop elements. Popups in later XULRunners are transparent by default.

Also for CSS changes, a menubar CSS entry in global.css was causing issues
with menu visibility on Linux. Things look fine without it, so I went ahead
and removed it outright.

In the reporter I found that "class" was being used as a variable name, but
that is a reserved keyword. I changed it to aClass.

To make things easier for url_prefix use I made it so that the urls hash is
checked by the url_prefix function. It looks for names up to the first of
several characters: /, ?, |. If urls contains that name it is replaced into
place. For | the | itself is removed in this process.

[Thomas Berezansky (tsbere)]

I have set up a test server for those interested:

http://new-xulrunner.mvlcstaff.org/

Staff client can be downloaded from:

http://new-xulrunner.mvlcstaff.org/updates/manualupdate.html

Superuser is evergreen/open-ils, it has the Concerto and sample user data from https://bugs.launchpad.net/evergreen/+bug/926281 loaded.

For reference, the list of staff users that data set loads appears to be (with apologies for formatting):

 Library | Profile Type | username | password
---------+----------------------------+----------------+------------------
 BR1 | Catalogers | br1bbrown | beverlyb1234
 BR1 | Catalogers | br1jmcginnis | josem1234
 BR1 | Circulators | br1iwalton | inezw1234
 BR1 | Circulators | br1mtownsend | maryt1234
 BR1 | Acquisitions | br1breid | barbarar1234
 BR1 | Acquisitions | br1mroberts | michaelr1234
 BR1 | Acquisitions Administrator | br1krush | keithr1234
 BR1 | Acquisitions Administrator | br1lfarrell | lindaf1234
 BR1 | Cataloging Administrator | br1dbeck | dianab1234
 BR1 | Cataloging Administrator | br1mclark | montyc1234
 BR1 | Circulation Administrator | br1awilliams | annaw1234
 BR1 | Circulation Administrator | br1mneal | melvinn1234
 BR1 | Local Administrator | br1csmith | cathys1234
 BR1 | Local Administrator | br1wrandall | williamr1234
 BR1 | Serials | br1epalmer | elvap1234
 BR1 | Serials | br1mmeeks | michellem1234
 BR1 | System Administrator | br1rjackson | ronaldj1234
 BR1 | System Administrator | br1sbrock | scottb1234
 BR1 | Global Administrator | br1vcampbell | vincentc1234
 BR1 | Data Review | br1mmartin | marym1234
 BR1 | Data Review | br1vford | vivianf1234
 BR1 | Volunteers | br1ajoseph | alexj1234
 BR1 | Volunteers | br1imccoy | idam1234
 BR2 | Catalogers | br2jclark | joannec1234
 BR2 | Catalogers | br2sschmidt | scotts1234
 BR2 | Circulators | br2mbarber | michealb1234
 BR2 | Circulators | br2tcruz | tomc1234
 BR2 | Acquisitions | br2krowland | kennethr1234
 BR2 | Acquisitions | br2thansen | terranceh1234
 BR2 | Acquisitions Administrator | br2jroberts | johnr1234
 BR2 | Acquisitions Administrator | br2sheath | stevenh1234
 BR2 | Cataloging Administrator | br2awright | alyssaw1234
 BR2 | Cataloging Administrator | br2psantiago | paiges1234
 BR2 | Circulation Administrator | br2cdodson | cleod1234
 BR2 | Circulation Administrator | br2rsmith | roberts1234
 BR2 | Local Administrator | br2dlawson | doral1234
 BR2 | Local Administrator | br2lvargas | laurav1234
 BR2 | Serials | br2mgonzales | maryg1234
 BR2 | Serials ...

[Kathy Lussier (klussier)]

Hi Tom,

I ran across some trouble while starting to test acq on your server. I was logged in with the super user account.

I retrieved Record ID 101 (BTW, when I try doing a catalog search for the record, I'm getting an internal server error)
I click Actions for this record < View/place orders

I receive a pop-up saying:

Excetpion: Session Expired
Please enter the credentials for the new login session.

with a prompt asking to enter my user name and password.

When I enter the user name and password, I receive the same pop-up message. I need to exit the window to stop the pop-ups.

For the purposes of documenting what has been tested, I also successfully loaded an order record using the following steps:

I set up a fund source, fund, and provider with holdings tag and subfield.

In the Load MARC Order Records interface, I selected create purchase order, entered a provider code, identified SYS1 as the context org unit, added the order to a selection list called "test," browsed for the file, and clicked upload.

On your server, the order records and acquisitions copies successfully loaded, and I was able to access the PO and selection list. However, when I tested a similar process on a server with http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/bshum/acq-vandelay-matching-and-import loaded, I received a user name and password prompt like the one I reported above. Since neither branch is in master yet, I don't know when that particular issue can be tackled, but I thought it would be worthwhile to report it sooner rather than later.

Kathy

[Thomas Berezansky (tsbere)]

For the record, I have looked at Kathy's issues and believe I have resolved them in the new xulrunner branch.

I also rebased it and tacked on a commit to bump the version numbers up to 11. And updated the test server with a blowout and reload.

[Kathy Lussier (klussier)]

Hi Tom,

I am unable to load the following admin interfaces with the new xul client:

* Permissions (No permissions are listed, just a question mark in each column)
* Permission Groups (the interface loads, but there is nothing in the Permission Groups folder)
* Copy Statuses (No statuses are listed, just a question mark in each column)
* Organizational Unit Types (the interface loads, but there are nothing in the OU Types folder)
* Organizational Units (Just sits on the screen that you typically see as the interface is loading)

Also, when I try to go to "User Permission Editor" on a patron record, I get the following error:

{"fileName":"https://new-xulrunner.mvlcstaff.org/xul/2_2newxulrunner_20120326092941/server/patron/user_edit.js","lineNumber":376}

I apologize in advance if they are unrelated to the new xul work, but they are all interfaces that I can confirm were working in the somewhat recent past.

Kathy

[Kathy Lussier (klussier)]

I have one more display issue to add. When using MARC View on a record, (Actions for this Record < MARC View), the client only displays the first few MARC fields.

[Thomas Berezansky (tsbere)]

I think I have fixed the various interfaces that don't work at all (authtoken issues. bah.).

MARC View requires more digging, but I think I have a general idea where the problem lies.

[Thomas Berezansky (tsbere)]

And I have pushed a fix for the MARC View issue. Blowing out the test server to throw the new code in now, it should be back up in a few minutes.

[Thomas Berezansky (tsbere)]

I have updated the description, the branch, and will update the test server shortly.

[Thomas Berezansky (tsbere)]

And the test server is updated and back online.

[Jason Stephenson (jstephenson)]

If anyone wants to help with testing this on their own, or on their own data:

http://blog.mvlcstaff.org/2012/05/testing-1-2-3.html

[Thomas Berezansky (tsbere)]

Added a commit to the branch to fix the printer settings editor. Newer XULRunner versions don't seem to link binding the messagecatalog from HTML files.

[Kathy Lussier (klussier)]

I have tested the batch import interfaces on the testing server, including the following:

* Uploading MARC records
* Inspecting queues and performing actions from the queue (filtering, copying to bucket, importing, exports)
* Viewing and creating record display attributes, merge/overlay profiles, record match sets, import item attributes.

The only bug I found was something I was able to replicate in a standard 2.2 install, so I would say this area of the staff client looks pretty good.

[Kathy Lussier (klussier)]

Tom,

I'm not sure if this is related to newxul or not, but I'm not seeing this behavior on other servers. I took the following steps:

1. Retrieve a record.
2. Select Actions for this Record < View/Place Orders
3. Wait for the screen to fully load.
4. Click on "Add to Selection" list.

The progress meter just hangs at 0%. I'm using the superuser account.

Kathy

[Thomas Berezansky (tsbere)]

At this point it appears to be working for me, though it wasn't on the test server initially. I think ACQ changes that went in today may have resolved the issue.

I have since re-loaded the test server to get the latest changes to master.

[Kathy Lussier (klussier)]

When uploading order records via acquisitions, I am seeing a "session expired" error similar to one I had reported earlier in other parts of acquisitions. I took the following steps:

Acquisitions < Load MARC Order Records
Select Create Purchase Order
bru is the provider
BR1 is the context org unit
Type the name of a new list in "Add to Selection List"
Create a new queue with any name
Test is the Record Match Set
Match-Only Merge is the Merge Profile
Select Import Non-Matching Records and Merge on Best Match

I select a file to upload - if you need to replicate this, the file in the Vandelay section at http://open-ils.org/dokuwiki/doku.php?id=qa:eg_test_cases should work here.

I click upload, and the file uploads without any problem.

If I then click on"View Purchase Order," "View Selection List" or "View Import Queue", I get the "Session Expired" box. If I enter credentials or click cancel, the box just reappears again.

[Thomas Berezansky (tsbere)]

I have added a commit to fix that issue and updated the test server.

[Kathy Lussier (klussier)]

Thanks Tom! The latest commit did the trick.

I only found one other small issue in acquisitions that I wasn't able to replicate on a clean copy of master. The return button on the lineitem worksheet does not work. A progress bar comes up, but then the screen just reloads. You can replicate it with the following steps:

Go to Acquisitions < Purchase orders
You can find a PO by PO ID = 2
Click on the PO ID to access the PO
Click on the worksheet link that's in the purple "Line Items" section of the PO. The "Return" button in the upper left corner of the screen is the one where I encountered trouble.

Otherwise, everything else in acquisitions looks good. I'll move on to serials next.

[Thomas Berezansky (tsbere)]

I suspect that is because document.referrer doesn't work with oils:// urls. Or any non-http and non-https urls in general.

Given that the interface in question has a "Go Back" button I am not sure it is that big of a deal, though. Perhaps at some point we should remove the button?

I will note that it looks like *all* of the uses of that method of providing a way "back" seem to be from the new ACQ changes from Bill Erickson that went into master recently. There are 4 apparent uses of the technique in the code base right now. I don't know if there is a good way to make them all work again, but the one you point out is the only one that does not appear to be in an if block before it shows up.

[Kathy Lussier (klussier)]

Just ran through serials, creating subscriptions, distributions, prediction patterns and issuances in both views, generating predictions in both views, and creating a special issue in the alternate serials control view.

The only problem I found was that the "Pattern Code Wizard" does not load in either interface. If you need to replicate this, the most direct way to get there is probably by going through the Serial Control View.

1. Retrieve record 111
2. Go to Actions for this Menu < Serial Control View.
3. Click the "Subscriptions" tab.
4. In the left pane, open up BR1, then Subscription: #1, and then Captions/Patterns.
5. Click Caption / Pattern #1.
6. Click the "Pattern Code Wizard" button in the bottom right corner.

An empty window appears instead of a window with the Pattern Code Wizard interface.

[Thomas Berezansky (tsbere)]

I have fixed the issue reported in #19, and the issues I saw because of it (the pattern wizard would crash out, and not have enough height to be used). Couple more commits to the branch, updated the test server.

[Kathy Lussier (klussier)]

Thanks Tom! The pattern code wizard looks good in the serial control view. I walked through each of the screens without any apparent trouble.

There still was a problem with height in the alternate serial control view. Thanks for working through this!

[Kathy Lussier (klussier)]

Fiona Borthwick reported on the dev list that the following areas seem to be working as expected:

patron management - creation, editing, searching
bill management - adding bills, partial payments, looking at bill history etc
checkout of items
checkin of items

Many thanks to Fiona for helping out with testing!

[Kathy Lussier (klussier)]

The pattern code wizard is now displaying correctly in both serials interfaces. Thanks Tom!

[Kathy Lussier (klussier)]

Took a cursory look at cataloging today. No trouble adding volumes and items through the unified volume/copy editor and with the separate volume and copy editors. I also could create a MARC record and use the shortcut keys in the editor.

The only trouble I came across was related to adding authorities from the MARC record editor. After entering my name in the 100 field, I right-click on my name. The authority popup menu that comes up is nearly transparent and difficult to read.

Normally, if I select "Create immediately," I get a popup message that says "created new authority record." The message doesn't appear, and I don't think the authority record is actually created because I couldn't find it later when trying to search authority.

Also, normally, if I select "Create and edit," I get a pop up window with a New MARC editor to edit and save the authority record. This popup did not appear.

[Thomas Berezansky (tsbere)]

I will push a fix shortly for the transparent menus.

The "Create immediately" issue appears to be related to the maintain_901 change in master, and not to the new xulrunner work.

[Thomas Berezansky (tsbere)]

Ok, there were some other issues that I pushed fixes for after testing some maintain_901 fixes (and pushed those fixes).

window.open wasn't working due to not having an unsecured chrome "frame" to load the content into. The solution to that was two-fold:

1 - Create said unsecured frame

2 - Tell most of our window.open calls to use the "chrome" flag (this is ignored if we are doing so from a point where we don't have rights to)

I have updated the test server with these changes and the fix to maintain_901.

[Dan Scott (denials)]

Tested on Fedora 17 x86-64 with "make linux-client".

I had to install an extra i686 library (dbus-glib.i686), then run it with "LD_LIBRARY_PATH=xulrunner ./evergreen" to pick up libxul.so and others from the right location.

I deliberately misconfigured the hostname at connection time ("localhosty") and the status & version shown were both just a red "0"; I expected a red "404". Come to think of it, the HTTP status codes are useful for developers / sysadmins but probably not at all useful to normal humans, in a separate patch we should replace the green "200: OK" with "Good" and failure with "Problem connecting" or something like that.

On load, the tab was showing the raw XUL directory contents rather than the index.xhtml page.

Confirmed working under Local Administration:
  * Closed Date Editor
  * Patrons with Negative Balances (displays cleanly - didn't have test subjects)

Confirmed working under Search:
  * Search for Patrons
  * Retrieve patron by Database ID

Confirmed working under Circulation:
  * Bill Patron
  * Apply Payment
  * Mark Lost
  * Edit Due Date
  * Register Patron
  * Edit Patron
  * Apply Standing Penalty
  * Acquisition Requests
  * Test Password

Confirmed working under workstation administration:
  * External editor command
  * Global font/sound settings
  * Custom toolbars (just switching cat/circ)

Confirmed working under Server Administration
  * Z3950 Servers

The one problem I ran into was in trying to run the user permission editor to edit the permissions on a newly created Cataloguer user. The editor began to load, but failed to display completely with an error dialog: {"fileName":"oils://remote/xul/server/server/patron/user_edit.js","lineNumber":376}

[Dan Scott (denials)]

"Edit Volumes" initially opened very narrow (about 20em wide) but seemed to remember the size on subsequent invocations after I resized it.

[Dan Scott (denials)]

Import via Z39.50, Create new MARC Record, basic authority work within the MARC Editor worked well.

[Dan Scott (denials)]

I signed off on this and committed it to master. Many thanks, Thomas! (We'll open a bug to track the user permission editor error separately).

[Dan Scott (denials)]

See bug # 1030907 for the user permission editor problem.