Firefox fails acid3.acidtests.org

[Uqbar]

Compliance test at http://acid3.acidtests.org/ is failing since a few releases, now.
The reported failure says:

Test 35 failed: expected '0' but got '1' - root element, with no parent node, claims to be a :first-child

Previous versions used to pass 100 tests out of 100.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: firefox 53.0+build6-0ubuntu0.16.04.1
Uname: Linux 4.11.0-041100-lowlatency x86_64
AddonCompatCheckDisabled: False
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: enzo 2238 F.... pulseaudio
BuildID: 20170418123315
Channel: Unavailable
CurrentDesktop: KDE
Date: Thu May 11 08:54:48 2017
Extensions: extensions.sqlite corrupt or missing
ForcedLayersAccel: False
IfupdownConfig:
 # interfaces(5) file used by ifup(8) and ifdown(8)
 auto lo
 iface lo inet loopback
IncompatibleExtensions: Unavailable (corrupt or non-existant compatibility.ini or extensions.sqlite)
IpRoute:
 default via 192.168.255.254 dev enp2s0 proto static metric 100
 169.254.0.0/16 dev enp2s0 scope link metric 1000
 192.168.255.0/24 dev enp2s0 proto kernel scope link src 192.168.255.42 metric 100
IwConfig:
 lo no wireless extensions.

 enp2s0 no wireless extensions.
Locales: extensions.sqlite corrupt or missing
Plugins: Shockwave Flash - /usr/lib/flashplugin-installer/libflashplayer.so
PrefSources: prefs.js
Profiles: Profile0 (Default) - LastVersion=53.0/20170418123315 (In use)
RfKill:
 0: hci0: Bluetooth
  Soft blocked: no
  Hard blocked: no
RunningIncompatibleAddons: False
SourcePackage: firefox
Themes: extensions.sqlite corrupt or missing
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/22/2013
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: L02 v02.01
dmi.board.asset.tag: CZC3523T17
dmi.board.name: 18EB
dmi.board.vendor: Hewlett-Packard
dmi.chassis.asset.tag: CZC3523T17
dmi.chassis.type: 6
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvrL02v02.01:bd09/22/2013:svnHewlett-Packard:pnHPProDesk490G1MT:pvr:rvnHewlett-Packard:rn18EB:rvr:cvnHewlett-Packard:ct6:cvr:
dmi.product.name: HP ProDesk 490 G1 MT
dmi.sys.vendor: Hewlett-Packard

[Bill Gianopoulos (wgianopoulos)]

Under both Aurora and nightly ACID3 test 35 is failing with this error:

Test 35 failed: expected '0' but got '1' - root element, with no parent node, claims to be a :first-child

Current relase and Beta are NOT affected.

[Bill Gianopoulos (wgianopoulos)]

This fails under WIndows as well.

[Mats-l (mats-l)]

Is this an intentional change from bug 1300374?

[Bill Gianopoulos (wgianopoulos)]

(In reply to Mats Palmgren (:mats) from comment #2)
> Is this an intentional change from bug 1300374?

If this is an intentional change, and the reasons for the change are legitimate, then we should ask to have the test changed. That said, Google Chrome still gets 100% on this test. I will try backing out that change to see how it impacts the test results.

[Bill Gianopoulos (wgianopoulos)]

I have verified that restoring this code:

if (!parent) {
    return false;
}

from edgeChildMatches

which was removed by the patch for bug 1300374 does restore the 100% result in the Acid3 test.

I also wonder why this was not known , if it was not, because I thought we included the Acid3 test in the our test suite.

[Emilio (emiliocobos)]

Yes, this is intentional, since the spec changed. If it breaks stuff we can of course restore the old behavior (though I won't say Acid3 is representative of actual content, and this is the first report related to that bug AFAIK).

I think we should get the test changed, but I don't know how feasible is it.

Also, no, I don't think we include Acid3 in our test suite (otherwise that patch would've never landed).

[Ms2ger (ms2ger)]

Fixing Acid3 has historically not been an issue. I asked Hixie <https://lists.w3.org/Archives/Public/www-archive/2016Oct/0005.html>.

[Bmo-7 (bmo-7)]

(In reply to Emilio Cobos Álvarez [:emilio] from comment #5)
> I think we should get the test changed, but I don't know how feasible is it.
Will this change cause any webcompat issue here between Blink and us ? Following the spec is correct but still concerned about breakage of any websites due to compatibility issues.

[Emilio (emiliocobos)]

Hi Astley,

As I said before, I don't think Acid3 is representative of actual content, so I don't think sites rely on this.

That being said, yes, it's a difference, but this is the first report related to that change in two months, and it's in a fairly artificial test.

[Bmo-7 (bmo-7)]

So could we change to Tech Evangelism team to help the fix on ACID3 test cases?

[Ms2ger (ms2ger)]

No need. I'll try to poke again this week.

[Bill Gianopoulos (wgianopoulos)]

Problem is what to recommend doing with the test. I would hate to have another test out of this suite end up being an everyone passes, yet if we change the test to expect the new behavior and make the old behavior fail, we end up needing to uplift bug 1300374 everywhere including ESR, otherwise we end up with release builds failing ACID3. I am sure other browser vendors are int he same boat here. Permitting both behaviors is kind of the same as making it an everyone passes test. Can we suggest somehow putting in a current date check to permit old behavior as pass before some future date and make new behavior pass?

[Bmo-7 (bmo-7)]

(In reply to Bill Gianopoulos [:WG9s] from comment #11)
> Can we suggest somehow putting
> in a current date check to permit old behavior as pass before some future
> date and make new behavior pass?

It's not possible to uplift the fix to all releases.
Instead, I think it all depends on how the test case would be updated in order to fit the spec change.

[Ms2ger (ms2ger)]

https://github.com/w3c/csswg-drafts/issues/695

[Onnebula9 (onnebula9)]

I can confirm that this applies equally to the current aurora build of SeaMonkey 2.49a2.

[Bmo-7 (bmo-7)]

Before there is any response or conclusion from WG, do we really want this behavior change being shipped along the way to release channel users ? What's the pros to keep this change even though there might be some webcompat issues hitting us in the future?

[Emilio (emiliocobos)]

Tab seems to agree that this is the way forward to do this[1], and I'm trying to get the change landed on Blink too.

[1]: https://github.com/w3c/csswg-drafts/issues/695#issuecomment-263930658

[Bzbarsky (bzbarsky)]

For what it's worth, we (and other browsers) used to have our current behavior until Acid3 was published, iirc. Then we changed to make acid3 pass in this one weird edge case.

The right way to fix the test is to remove this part of it completely, imo.

[Annevk (annevk)]

As I mentioned in bug 1300374 comment 21, I replied to Ian: https://lists.w3.org/Archives/Public/www-archive/2017Jan/0014.html.

[Emilio (emiliocobos)]

FYI, I landed the equivalent patch on Blink[1], so it shouldn't be considered a webcompat issue anymore.

[1]: https://codereview.chromium.org/2588643004/

[Ryanvm (ryanvm)]

Sounds like this is wontfix for 52+ then, Boris?

[Bzbarsky (bzbarsky)]

Yes, I think so. We should keep pushing on Ian, but if it's not just us making the change that makes it much simpler to make the case the test is broken if he refuses to change it.

[Ryanvm (ryanvm)]

Calling this WONTFIX from a Firefox standpoint and moving the bug over to Tech Evangelism at this point, then. Thanks!

[Alice0775-t (alice0775-t)]

*** Bug 1345994 has been marked as a duplicate of this bug. ***

[Uqbar (uqbar)]

I reported this also on mainstream, just in case it's not Ubuntu-related

https://bugzilla.mozilla.org/show_bug.cgi?id=1363980

[Kohei-yoshino (kohei-yoshino)]

*** Bug 1363980 has been marked as a duplicate of this bug. ***

[Hans Joachim Desserud (hjd)]

Thanks for reporting.

The upstream issue was marked as a duplicate of https://bugzilla.mozilla.org/show_bug.cgi?id=1311329, so I've taken the liberty of adding a bug watch for this issue.

Looks like the actual issue occurs because the specification has changed, but the test hasn't been updated, see discussion on the upstream issue for details.

[Norbert (nrbrtx)]

Got 98/100 with 54.0+build3-0ubuntu0.16.04.1.

[Vader-24 (vader-24)]

"Calling this WONTFIX from a Firefox standpoint and moving the bug over to TECH EVANGELISM at this point, then. Thanks!"
Well, this is embarrassing...

[Bzbarsky (bzbarsky)]

What exactly is embarrassing? We're following the spec. The test is wrong per current spec text, but the test author refuses to change it. Who is supposed to be embarrassed and why?

[Vader-24 (vader-24)]

Where is bug in the test and where is this specification, that specifies 1 instead 0? If the test is wrond the title shall be "INVALID instead of tech evangelism. The test 72 is also a bug in test benchmark?

What is embarrassing? Race Hazard in Firefox. Please see topic for test 72 and please multiple run the test. Why the result is 98+(rand()%2) instead of stable 99 or 98?

[Bzbarsky (bzbarsky)]

> Where is bug in the test and where is this specification, that specifies 1 instead 0?

Did you read this bug report? That's covered in the links in the first several comments.

> The test 72 is also a bug in test benchmark?

This bug report is about test 35. It has nothing to do with test 72.

That said, I just tried in Firefox 54 release and I can't reproduce any test 72 failures; I get a stable score of 99 over dozens of runs. If you can reproduce a problem there, please file a separate bug with steps to reproduce and cc me.

[Marius-spix (marius-spix)]

How can I find out which test is failing? On my FF 53.0.3, the test result is 99%, the third bucket is gray. Hovering over the A converts the cursor to a help cursor, this state keeps permanently if you click on the A. No dialog appears, however and refreshing the page gives an 1% score. This is very weird.

[Bzbarsky (bzbarsky)]

Please stop asking all possible Acid3 questions in this bug... But in general, clicking the A is supposed to show an alert that explains which tests failed, and does for me on Firefox 53.0.3 with a clean profile.

[Marius-spix (marius-spix)]

Thank you. After resolving 1381336, clicking on the A works for me now, too.

Failed 1 tests.
Test 26 passed, but took 61ms (less than 30fps)
Test 35 failed: expected '0' but got '1' - root element, with no parent node, claims to be a :first-child
Total elapsed time: 0.35s

[Ms2ger (ms2ger)]

Sent another email: http://lists.w3.org/Archives/Public/www-archive/2017Jul/0002.html

[Ms2ger (ms2ger)]

Apparently the test is no longer being updated: <http://lists.w3.org/Archives/Public/www-archive/2017Jul/0003.html>. Not sure if there's much point in keeping this bug open.

[Miket-y (miket-y)]

(In reply to :Ms2ger (⌚ UTC+1/+2) from comment #33)
> Apparently the test is no longer being updated:
> <http://lists.w3.org/Archives/Public/www-archive/2017Jul/0003.html>. Not
> sure if there's much point in keeping this bug open.

Given that update, let's close. Thanks.

"Acid3, in particular, contains some controversial tests and no longer reflects the consensus of the Web standards it purports to test, especially when it comes to issues affecting mobile browsers. The tests remain available for historical purposes and for use by browser vendors."