Ubuntu
software-center package

Ubuntu Software Center "Get Software" screen contents not read in Orca

Bug #433104 reported by Matthew Paul Thomas on 2009-09-19

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	software-center (Ubuntu)	Fix Released	High	Unassigned
	webkit (Ubuntu)	Confirmed	Undecided	Unassigned

Bug Description

Orca 2.29.91, Ubuntu Software Center 1.1.12, Ubuntu Lucid

1. Launch Ubuntu Software Center.
2. In Accerciser, navigate to software-center path 0,0,1,1,0,0,0,1,0,0,0,1,1 -- i.e. "software-center" > "Ubuntu Software Center" > (filler) > (split pane) > (page tab list) > first (page tab) > (panel) > (filler) > (page tab list) > first (page tab) > (scroll pane) > (document frame) > (panel) > second (paragraph).
3. In the "Interface Viewer" tab, expand "Text" and notice that it reads "Accessories" as it should.
4. Launch Orca.
5. Tab to the "Accessories" link.

What should happen: Orca says "Accessories".
What actually happens: Orca says "panel".

The list of departments currently uses an HTML table. (Yes, maybe not the best choice, but anyway...) JavaScript code intercepts Tab, Shift Tab, or arrow keys, to focus the appropriate table cell. Each table cell has this structure:
    <td>
      <p><img src="department-icon" /></p>
      <p>Accessories</p>
    </td>

Oddly the <td> does not show up in the Accerciser hierarchy, though the two paragraphs inside it do.

[Originally reported by "Kyle" in <http://twitter.com/Kyle4Jesus/status/4101268542>.]

See original description

Tags:

Matthew Paul Thomas (mpt) on 2009-09-19

description:	updated
description:	updated

Revision history for this message

Michael Vogt (mvo) wrote on 2009-09-21:

There is little html to dump in that view, the document is created via JS mostly.

Changed in software-store (Ubuntu):
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

Michael Vogt (mvo) wrote on 2009-09-21:

Here is the actual html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />
<title></title>
<script type="text/javascript">
  function changeTitle(title) {
    document.title = title;
  }
  function addCategory(name, img) {
    oT = document.getElementsByTagName("table")[0];
    newTr = document.getElementsByTagName("tr")[0];
    newTd = document.createElement("td");
    newTd.setAttribute("tabindex", 0)
    newTd.className = "category_td";
    // we need a <p> around the img so that we can align it
    newTextAroundImg = document.createElement("p");
    newImg = document.createElement("img");
    newImg.src = img;
    newImg.setAttribute("width", ${icon_size})
    newImg.setAttribute("height", ${icon_size})
    newTextAroundImg.appendChild(newImg)
    newText = document.createElement("p");
    newText.align = "center";
    click_cb = 'changeTitle("call:on_category_clicked:' + name + '");';
    newTd.setAttribute('onclick', click_cb);
    key_cb = 'if (event.which == 13) { changeTitle("call:on_category_clicked:' + name + '"); };'
    newTd.setAttribute('onkeydown', key_cb);
    newText.appendChild(document.createTextNode(name))
    newTd.appendChild(newTextAroundImg);
    newTd.appendChild(newText);
    newTr.appendChild(newTd);
    oT.appendChild(newTr);
  }
</script>
  <style type="text/css">
    body {background-color:#b6bdc4; margin: 0; padding: 1em; color:#000;}
    h2 {
      background-color: #7D91aa;
      border-style: solid;
      border-width: 0 0 1px 0;
      border-color: #E2E7EC #404a57 #404a57 #E2E7EC;
      color: #fff;
      font-size: 100%;
      font-weight: normal;
      padding: 0.25em 0.5em;
      margin: 0;
      text-shadow: #404a57 0 -1px 1px;
    }
/* debug border */
/* .category_td {border-width:1px;border-style:solid;} */
/* fixed width does not work, it does not ellipsis but overflows the box */
/* .category_td {padding:8px; margin:8px; width:6em; display:inline-block; text-overflow: ellipsis;} */

    .category_td {
      padding: 0 8px;
      display:inline-block;
      text-overflow: ellipsis;
      vertical-align: top;
      width: 8em !important;
    }
    .category_td:hover {cursor: pointer;}
    .category_td:active {background-color: #7D91aa;}

.category_td p { font-size:90%; text-align:center; }
</style>
</head>

<body>
  <h2>$header</h2>
  <table>
    <tr></tr>
  </table>
</body>
</html>

Here is the actual html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
 <meta http-equiv="Content-type" content="text/html;charset=UTF-8" />
 <title></title>
 <script type="text/javascript">
  function changeTitle(title) { 
    document.title = title; 
  }
  function addCategory(name, img) {
    oT = document.getElementsByTagName("table")[0];
    newTr = document.getElementsByTagName("tr")[0];
    newTd = document.createElement("td");
    newTd.setAttribute("tabindex", 0)
    newTd.className = "category_td";
    // we need a <p> around the img so that we can align it
    newTextAroundImg = document.createElement("p");
    newImg = document.createElement("img");
    newImg.src = img;
    newImg.setAttribute("width", ${icon_size})
    newImg.setAttribute("height", ${icon_size})
    newTextAroundImg.appendChild(newImg)
    newText = document.createElement("p");
    newText.align = "center";
    click_cb = 'changeTitle("call:on_category_clicked:' + name + '");';
    newTd.setAttribute('onclick', click_cb);
    key_cb = 'if (event.which == 13) { changeTitle("call:on_category_clicked:' + name + '"); };'
    newTd.setAttribute('onkeydown', key_cb);
    newText.appendChild(document.createTextNode(name))
    newTd.appendChild(newTextAroundImg);
    newTd.appendChild(newText);
    newTr.appendChild(newTd);
    oT.appendChild(newTr);
  }
 </script>
  <style type="text/css">
    body {background-color:#b6bdc4; margin: 0; padding: 1em; color:#000;}
    h2 {
      background-color: #7D91aa;
      border-style: solid;
      border-width: 0 0 1px 0;
      border-color: #E2E7EC #404a57 #404a57 #E2E7EC;
      color: #fff;
      font-size: 100%;
      font-weight: normal;
      padding: 0.25em 0.5em;
      margin: 0;
      text-shadow: #404a57 0 -1px 1px;
    }
 /* debug border */
 /* .category_td {border-width:1px;border-style:solid;} */
 /* fixed width does not work, it does not ellipsis but overflows the box */
 /* .category_td {padding:8px; margin:8px; width:6em; display:inline-block; text-overflow: ellipsis;} */

.category_td p { font-size:90%; text-align:center; }
 </style>
 </head>

<body>
  <h2>$header</h2>
  <table>
    <tr></tr>
  </table>
</body>
</html>

Revision history for this message

arky (arky) wrote on 2009-10-19:

Screenshot-Accerciser Accessibility Explorer.png Edit (132.1 KiB, image/png)

Unable to reproduce this bug in software-center 1.0. Orca screen reader reads out the names of departments correctly. Accercisor shows that applications exposes the a11y label/icon information properly.

Revision history for this message

Matthew Paul Thomas (mpt) wrote on 2009-10-20:

Marking Incomplete until we get steps to reproduce the problem.

Changed in software-center (Ubuntu):
status:	Confirmed → Incomplete

Revision history for this message

arky (arky) wrote on 2009-10-21:

Regression:

In software-center (1.0.1) the departments are reported as 'panel'.

Original bug:
Initially when orca starts it does say software-center as 'html content'.

~~~~~~~~~~~~~~~~~~~~~~~
SPEECH OUTPUT: 'html content.'
generate braille for app.name='software-center' name='None' role='document frame' state='enabled focused sensitive showing visible' relations='' (args={'formatType': 'unfocused', 'role': ROLE_DOCUMENT_FRAME, 'mode': 'braille', 'recursing': True}) using '(includeContext and (ancestors + (rowHeader and [Region(" " + asString(rowHeader))]) + (columnHeader and [Region(" " + asString(columnHeader))]) + (radioButtonGroup and [Region(" " + asString(radioButtonGroup))]) + [Region(" ")]) or []) + [Component(obj, asString(label + displayedText + value + roleName + required))] + (nodeLevel and [Region(" " + asString(nodeLevel))])'
includeContext=True
ancestors=[]
rowHeader=[]
columnHeader=[]
radioButtonGroup=[]
label=[]
displayedText=['']
value=['']
roleName=['HtmlPane']
required=[]
nodeLevel=[]
generate braille results:
  Region: ' ', 0
  Component: 'HtmlPane', 0
BRAILLE LINE: ' HtmlPane'
     VISIBLE: ' HtmlPane', cursor=2
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Matthew Paul Thomas (mpt) on 2009-10-23

Changed in software-center (Ubuntu):
status:	Incomplete → Confirmed

arky (arky) on 2010-02-13

tags:

added: a11y

Revision history for this message

Matthew Paul Thomas (mpt) wrote on 2010-02-24:

I know there are still accessibility issues with WebKit-GTK, but this doesn't seem to be one of them. This screenshot shows that Accerciser correctly finds the "Accessories" text inside the WebKit view. But when tabbing to the link, Orca reads it only as "panel". So I think this is probably a bug in Orca, rather than WebKit or USC.

Revision history for this message

Matthew Paul Thomas (mpt) wrote on 2010-02-24:

screenshot Edit (90.6 KiB, image/png)

Matthew Paul Thomas (mpt) on 2010-02-24

description:	updated
summary:	- Main "Get Free Software" screen is apparently inaccessible + Ubuntu Software Center "Get Software" screen contents not read in Orca

Matthew Paul Thomas (mpt) on 2010-02-24

description:

updated

Revision history for this message

Matthew Paul Thomas (mpt) wrote on 2010-02-24:

Marking as Incomplete for software-center until/unless we know there's something we need to change in the USC code.

If it would help, we could switch from using <td> elements to using <li> or similar, if there's something in WebKit-GTK accessibility that has been implemented for some HTML elements but not others.

Changed in software-center (Ubuntu):
status:	Confirmed → Incomplete

Revision history for this message

WillieWalker (william-walker) wrote on 2010-02-24:

Matthew and I looked at this side by side today. Something seems odd and we've CC'd Joanie on this bug.

We set up accerciser to listen for events and then arrowed between the labeled icons in the software center's main pane. When looking at the events in Orca and accerciser, I'm noticing object:state-changed:focused events coming from WebKitGtk for a "panel" of some sort, but cannot seem to find this panel in the accerciser hierarchy. As such, it looks like there might be some sort of bottom up vs. top down hierarchy issue.

Having said that, even if that is fixed, it looks like focus is being put on the panel which I would assume might be the table cell that contains the icon and text. If this is going to end up being how things are done, Orca might need a script to drill down into these things to discover text to present to the user. Not sure.

Joanie - what do you think?

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-24:

#10

Will, et al.:

If I had to bet money, I'd say it's not a bottom up vs top down hierarchy issue.

Amongst the various and sundry tasks we are working on within WebKitGtk a11y is "collapsing/flattening" the accessible hierarchy so that instead of things like:

->Panel
   ->Paragraph
       -> text (this is)
       -> text (bold)
       -> text (text)

We have:

-> Paragraph

whose accessible text contains the string 'this is bold text'.

This happens to be a major change which is still very much a work in progress. Getting the events (and for that matter the states) all straightened out is still taking place. When it's done -- and for that matter, when the rest of the work necessary to make WebKitGtk accessible is done -- it will be wonderful, because I think we will find many things (including this Software Center) start to JustWork with a minimum amount of scripting support needed on an app-by-app basis. And some things might JustWork with no scripting support.

Unfortunately, WebKitGtk a11y work is not yet done. (I don't mean the changes I refer to above. I mean period.)

Distros which choose to employ tools that rely upon it might be able to hack around things which have not yet been implemented in WebKitGtk by providing their own custom scripts for Orca, but they should keep in mind that things are changing rapidly.

I hope this helps shed some light on things.

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-24:

#11

Also, the Orca meta bug for WebKitGtk is:
https://bugs.webkit.org/show_bug.cgi?id=25531

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-24:

#12

Oh, and one more thing since I see reference to table cells. The accessible table hierarchy in WebKitGtk is -- and has always been, independent of the hierarchy collapsing -- completely and utterly borked. See https://bugs.webkit.org/show_bug.cgi?id=30895.

That's on my to-do list as well.

Revision history for this message

Gabe Gorelick (gabegorelick) wrote on 2010-02-24:

#13

Actually, the real orca package has been called gnome-orca since Dapper. Confusing, I know.

affects:

orca (Ubuntu) → gnome-orca (Ubuntu)

Revision history for this message

Matthew Paul Thomas (mpt) wrote on 2010-02-24:

#14

Joanmarie, would it help if we avoided using either paragraphs or table cells, by switching to <ul><li>Accessories</li><li>Education</li><li>Games</li>... instead?

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-24:

#15

Probably. :-) List items are being exposed in a far more normal fashion (though at the moment, they have a child text object for the bullet/number).

This weekend (when I have time to work on WebKitGtk), I will prioritize two issues identified in this bug:

1. Ensuring that the correct object emits the event (comment #9)
2. Seeing if I can unbork the accessible table hierarchy

Matthew, what is the cut-off point for Ubuntu including a newer release of WebKitGtk?

Revision history for this message

Alexander Sack (asac) wrote on 2010-02-24:

#16

> Matthew, what is the cut-off point for Ubuntu including a newer release of WebKitGtk?

since we are past feature freeze we first need to see what comes with such a release before estimating/considering such an upgrade. If its a bug fix only release its easier to get in; if it adds new features and changes API etc. its harder to get in.

Revision history for this message

Matthew Paul Thomas (mpt) wrote on 2010-02-24:

#17

In the meantime, let's try switching software-center to use styled <ul> and <li>s instead of a <table>.

Changed in software-center (Ubuntu):
status:	Incomplete → Triaged
affects:	gnome-orca (Ubuntu) → webkit (Ubuntu)
Changed in webkit (Ubuntu):
status:	New → Confirmed

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-26:

#18

Triaging now.....

W.r.t. this:

> Oddly the <td> does not show up in the Accerciser hierarchy,
> though the two paragraphs inside it do.

This is due to the fact that WebKit decides not to expose "layout tables" as tables. Your table has been heuristically declared a "layout table," thus your cells are not cells and your table is not a table. :-)

WebKit a11y work is largely the work of the folks at Apple. They've done lots of great things which we can ultimately use. However, sometimes what makes sense for VoiceOver users doesn't jive with what we expect in GNOME-based ATs. This is such a case.

My intent is to make a platform specific change to cause such tables and cells to be exposed as such, leaving it up to individual ATs to decide how to present the content. With any luck, that might be sufficient to cause Orca to present the right info even with a borked table hierarchy (which is also on my list for this weekend). Such a change, I believe, will be small, simple, and safe. So even if there have been API changes, you could patch your version of WebKitGtk to fix the issue.

All a long way of saying the following: If it's not too late, hold off on the list conversion and let me see what I can do between now and Monday evening.

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-26:

#19

Bug for the primary issue filed upstream:
https://bugs.webkit.org/show_bug.cgi?id=35418
[Gtk] Every table, including layout tables, should be exposed as a table

I've also attached a proposed fix. As expected, it's a small, isolated patch. With that change, and using Lucid's Software Center:

* The objects Software Center gives focus to (the hidden panels Will pointed out), are now table cells which are, of course, not hidden. :-)

* Because the objects claiming focus are table cells, Orca is looking for the actual contents of each cell which claims focus and presenting them.

Translation: If you Tab or Arrow amongst the items, Orca no longer says "panel"; it speaks the contents as expected.

What is not yet quite right:

* When the table first gets focus, Orca speaks all of the items. I believe this is the result of the bogus table hierarchy, which is the other item I mentioned in this bug that I intend to address this weekend. If that isn't it, it's an extraneous object of ATK_ROLE_PANEL in the hierarchy between the Table and the Table Cells. Either way, it's on my list.

Michael, it would be super if you could verify that the patch attched to upstream bug works for you -- and is something that, worst case scenario, you could manually apply to the version of WebKitGtk which you'll be shipping. Thanks!

Revision history for this message

Joanmarie (joanmarie-diggs-deactivatedaccount) wrote on 2010-02-26:

#20

And by Michael, I mean Matthew. I'm really, really sorry!

Revision history for this message

Michael Vogt (mvo) wrote on 2010-04-01:

#21

I just tested this with the latest software-center and the latest webkit in lucid and it appears that it works fine now. I can navigate in the table and it reads the correct category names for me.

Michael Vogt (mvo) on 2010-04-01