.pyc files should be cleaned up after updates

[Dmitry Belyaninov]

Detailed bug description:

https://bugs.launchpad.net/fuel/+bug/1592309 this issue was fixed; however when we are trying to verify the fix we observing the same behavior (logs below)

Expected results:
Issue was fixed and does not reproduces

Actual result:
Issue reproduces but source code was updated.

Root cause: .pyc files wasn't cleaned up after updates so old code is executing.

Workaround:
Delete pyc file(s) then restart services
Impact:
100% swarm tests - we actually receiving non-consistent updates

====== logs =====

[root@nailgun ~]# rpm -qf /usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py
fuel-nailgun-9.0.0-1.mos8762.noarch

app.log:

  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 285, in content_json
    cls.__class__,
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 574, in PUT
    :http: * 202 (task successfully executed)
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 249, in execute
    self.check_no_running_deployment(self.cluster)

base.py:

572 """:returns: JSONized Task object.
573
574 :http: * 202 (task successfully executed)
575 * 400 (invalid object data specified)
576 * 404 (environment is not found)
577 * 409 (task with such parameters already exists)
578 """

Steps to reproduce:

install 9.1 env with latest repos
create and start deploy from UI
add some additional node to cluster(cli)
start deploy again(cli)

[root@nailgun ~]# fuel --env 1 deploy-changes
500 Server Error: Internal Server Error for url: http://10.109.0.2:8000/api/v1/clusters/1/changes?dry_run=0 (Unexpected exception, please check logs)

Should be "400 Error" with problem description.

Reproducibility:
few last snapshots
SWARMs affected too

[Vladimir Khlyunev (vkhlyunev)]

critical cause it affects all swarm tests

[Alexander Tsamutali (astsmtl)]

I tried to install fuel-nailgun from 9.0 (mos8743) and then updated it to currently proposed version for 9.1 (mos8768). Base.py, base.pyc and base.pyo were replaced with new versions. Deeper RCA is required here.

[Georgy Kibardin (gkibardin)]

When I'm install 9.0, then add repos, then do yum update -y && bootstrap_admin_node.sh I easily reproduce https://bugs.launchpad.net/fuel/+bug/1615576.
However, this fixes as soon as I restart nailgun and receiverd.
The problem is that bootstrap_admin_node.sh doesn't restart services, they use partially updated installation and we have a total mess as a result.
This also seems to be happen on 9.x BVT.

[Ivan Berezovskiy (iberezovskiy)]

We have script for master node update (https://github.com/openstack/fuel-library/blob/stable/mitaka/files/fuel-utils/updates/update-master-node.sh). It restarts master node services. Could you please try it?

[Georgy Kibardin (gkibardin)]

I think it isn't necessary - I'm sure it will work.

[Georgy Kibardin (gkibardin)]

BTW, why don't you call bootstrap_admin_node.sh? I expect it is called so that new packages could be installed, for instance, Timmy.

[Kyrylo Galanov (kgalanov)]

I doubt that the issue is valid at all.
*pyc files are packaged in rpm.

Please do similar checks on your env.

# rpm -ql fuel-nailgun-9.0.0-1.mos8790.noarch | grep handlers/base.py
/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py
/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.pyc
/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.pyo

# rpm -ql fuel-nailgun-9.0.0-1.mos8790.noarch | grep py\$ | wc -l
501
# rpm -ql fuel-nailgun-9.0.0-1.mos8790.noarch | grep pyc\$ | wc -l
500

[Dmitry Belyaninov (dbelyaninov)]

It seems that there is additional (may be original) problem.

[root@nailgun ~]# update-master-node.sh
-bash: update-master-node.sh: command not found
[root@nailgun ~]# ls /usr/share/fuel-utils/update-master-node.sh
/usr/share/fuel-utils/update-master-node.sh
[root@nailgun ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/.local/bin:/root/bin

So it is impossible to restart some services, and old code is still running.

We should move "update-master-node.sh" from /usr/share to /usr/bin

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/363029

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/359325
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=90062fc254fa3d988e1b6bbfe3ca1009dabf5fb8
Submitter: Jenkins
Branch: stable/mitaka

commit 90062fc254fa3d988e1b6bbfe3ca1009dabf5fb8
Author: Vladimir Khlyunev <email address hidden>
Date: Tue Aug 23 19:23:19 2016 +0300

    Use new way to apply 9.x

    https://review.openstack.org/#/c/346119 was merged and now
    we have to switch our tests to production-style updating

    Change-Id: Ifd3aabc7db24ce5e5213438992ab7eeef459cae5
    Partial-bug:1616188

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/363029
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=4151f14af98b290f3cdde13f552c15f9d3e008fe
Submitter: Jenkins
Branch: master

commit 4151f14af98b290f3cdde13f552c15f9d3e008fe
Author: Vladimir Khlyunev <email address hidden>
Date: Tue Aug 23 19:23:19 2016 +0300

    Use new way to apply 9.x

    https://review.openstack.org/#/c/346119 was merged and now
    we have to switch our tests to production-style updating

    Change-Id: Ifd3aabc7db24ce5e5213438992ab7eeef459cae5
    Partial-bug:1616188

[Alexander Kurenyshev (akurenyshev)]

All reviews were merged -> Fix Commited

[Dmitry Belyaninov (dbelyaninov)]

For 9.1 snapshot #205

2016-08-31 23:46:03,060 - INFO environment.py:625 -- Searching for updates..
2016-08-31 23:46:03,060 - INFO environment.py:629 -- Performing yum clean and update commands
2016-08-31 23:46:25,304 - INFO environment.py:635 -- Packages were updated successfully
2016-08-31 23:46:25,306 - INFO environment.py:663 -- 39 package(s) were updated
2016-08-31 23:46:25,306 - INFO environment.py:665 -- Applying updates via update-master-node.sh
2016-08-31 23:50:58,995 - INFO environment.py:673 -- Update successful