[devops] Most of test report threads weren't updated due to gdata.service.RequestError: {'status': 403, 'body': 'It looks like someone else already deleted this cell.', 'reason': 'Forbidden'}

[Andrey Sledzinskiy]

Please look at - /view/5.1_swarm/job/5.1-test-reports/48/console . Most of threads weren't updated because of
Create column b49-thread-1-on-5.1.1-17
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/jenkins/workspace/5.1-test-reports/utils/jenkins/report-exporter/__main__.py", line 64, in <module>
    build_column = sheet.get_build_column_name(build.number, build.get_iso_number())
  File "utils/jenkins/report-exporter/spreadsheet.py", line 74, in get_build_column_name
    self.table.SetFields(fields)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/gdata/spreadsheet/text_db.py", line 329, in SetFields
    self.spreadsheet_key, self.worksheet_id)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/gdata/spreadsheet/service.py", line 278, in UpdateCell
    converter=gdata.spreadsheet.SpreadsheetsCellFromString)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/gdata/service.py", line 1395, in Put
    media_source=media_source, converter=converter)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/gdata/service.py", line 1358, in PostOrPut
    'reason': server_response.reason, 'body': result_body}
gdata.service.RequestError: {'status': 403, 'body': 'It looks like someone else already deleted this cell.', 'reason': 'Forbidden'}

[Mike Scherbakov (mihgen)]

This is critical issue as it's basically very, very hard to get overall tests status. It's not easy by design, and with this bug it becomes much worse. Let's figure out asap how we can fix it.

[Igor Shishkin (teran)]

Some yesterday fixes could help, still checking.

Set High because there is existent workaround and work in progress.

[Aleksandra Fedorova (bookwar)]

There are three types of errors in test-reports jobs:

1) {'status': 403, 'body': 'It looks like someone else already deleted this cell.', 'reason': 'Forbidden'}

Appears when there are no more empty columns in the worksheet. Can be fixed by adding more empty columns via Goggle Spreadsheet interface.

2) {'status': 409, 'body': "...", 'reason': 'Conflict'}

Appears in os_patching jobs because there is indeed a conflict. Test results we get from os_patching jobs don't have unique keys for different test cases:

http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.os_patching/lastCompletedBuild/artifact/nosetests.xml

And this needs to be fixed on the tests side.

3) Other failures can be caused by rearranging test cases inside test groups. Usually they can be fixed by removing broken '<init>' row from the worksheet.

[Mike Scherbakov (mihgen)]

Please document a workaround.

[Mike Scherbakov (mihgen)]

mihgen: hi bookwar, are you working on https://bugs.launchpad.net/fuel/+bug/1393711 or do you have any workaround?
[3:56pm] mihgen: I usually just go to that report and check it there. We really need to make it up to date for 5.1.1 release
[4:00pm] bookwar: mihgen: i did those steps explained in 1) and 3). And reports work for all 5.1 jobs except os_patching. And I don't have workaround for those two

[Mike Scherbakov (mihgen)]

Do we expect to close this in 5.1.1 or it's Ok to live with workarounds, and we can fix it finally in 6.0?

[Dmitry Borodaenko (angdraug)]

+1 to Mike: I don't see why we would hold back 5.1.1 code freeze just for this bug

[Aleksandra Fedorova (bookwar)]

These workaround generally work, unless google servers stop to respond to requests by any reason.

This bug won't be fixed for 5.1.1 and for 6.0 the only way to resolve this issue is to rewrite report-exporter from scratch.

[Nastya Urlapova (aurlapova)]

Does it mean, that we will get to incorrect results for 5.1.1 swarm?

[Aleksandra Fedorova (bookwar)]

It means that to get correct results in 5.1-test-reports you'll need to retrigger the build or to apply one of those workarounds and then retrigger the build.

[Nastya Urlapova (aurlapova)]

Aleksandra, just implement retrigger if first run are failed.

[Nastya Urlapova (aurlapova)]

and all your mentioned errors in exception list.

[Aleksandra Fedorova (bookwar)]

I don't think that automatic rebuild will help here as all exceptions I've listed require someone to fix the spreadsheet manually.

I've set up notification on failure for this job so devops team will watch its status and see if it needs to be retriggered or if any other actions are needed.

[Igor Shishkin (teran)]

It's impossible to fix for sure, so Won't fix.
Anyway Sasha have created notifications to monitor this job.

[Nastya Urlapova (aurlapova)]

Igor, issue was confirmed around month, and suddenly you marked it as Won't Fix w/o targeting to the next milestone. I guess you understand clear that is very important not only for QA team, but also for developers and management.

[Nastya Urlapova (aurlapova)]

+ discussion from other task that related to reporting https://bugs.launchpad.net/fuel/+bug/1399160

[Igor Shishkin (teran)]

Nastya, the bug you have showed is a probable solution. We're going there to implement.
This bug couldn't be fixed. It's technically impossible to fix the way you want it here.

So here are the two ways:
 - fully rewrite the script reports that data (that we've discussed under your maintenance)
 - move to the plugin you're suggesting in the linked bug.

We're currently simply getting notifications from the job reports the data and fixing as soon as it possible. But we couldn't fix it.

So bug is not going to be fixed.