[Fuel Upgrade] Provisioning is done with old package versions (mcagents, network checker)

[Andrey Sledzinskiy]

{
    "build_id": "2014-07-23_15-32-11",
    "ostf_sha": "c1b60d4bcee7cd26823079a86e99f3f65414498e",
    "build_number": "70",
    "auth_required": false,
    "api": "1.0",
    "nailgun_sha": "06b9becc1e6831351aac693290ac619c92e56e99",
    "production": "docker",
    "fuelmain_sha": "7b03801a70506b1c9efc2aec8a6203bc916a5b89",
    "astute_sha": "fd9b8e3b6f59b2727b1b037054f10e0dd7bd37f1",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "fb0e84c954a33c912584bf35054b60914d2a2360"
}
Steps:
1. Install fuel from 5.0-26 ISO
2. Create simple cluster with default settings - 1 contrller, 1 compute, 1 cinder
3. Deploy cluster
4. After successful deployment upload upgrade script to master
5. Run upgrade
6. After successful upgrade create new cluster with new release - HA, CentOS, Neutron Vlan, Cinder for volumes, Murano, 3 controllers, 2 compute, 1 cinder nodes
7. Deploy cluster
8. After successful deployment run network verification

Expected - network verification passed
Actual - verification failed with
[372] Error running RPC method verify_networks: 7a348c46-9c1e-42d6-b7ae-26295943e904: MCollective call failed in agent 'net_probe', method 'multicast_listen', failed nodes:
ID: 7 - Reason: Attempted to call action 'multicast_listen' for 'net_probe' but it's not declared in the DDL
ID: 5 - Reason: Attempted to call action 'multicast_listen' for 'net_probe' but it's not declared in the DDL
ID: 4 - Reason: Attempted to call action 'multicast_listen' for 'net_probe' but it's not declared in the DDL
ID: 9 - Reason: Attempted to call action 'multicast_listen' for 'net_probe' but it's not declared in the DDL
ID: 8 - Reason: Attempted to call action 'multicast_listen' for 'net_probe' but it's not declared in the DDL
ID: 6 - Reason: Attempted to call action 'multicast_listen' for 'net_probe' but it's not declared in the DDL
, trace: ["/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/mclient.rb:116:in `check_results_with_retries'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/mclient.rb:62:in `method_missing'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/network.rb:77:in `multicast_verification'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:229:in `multicast_verification'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:88:in `multicast_verification'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:70:in `block in verify_networks'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:68:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:68:in `verify_networks'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:128:in `dispatch_message'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:91:in `block in dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `each_with_index'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:77:in `block in perform_main_job'"]

Logs are attached

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109262

[Dima Shulyak (dshulyak)]

This issue related to usage of old ubuntu/centos images by cobbler

[Dima Shulyak (dshulyak)]

we need to modify

"repo_metadata": "{\"nailgun\":\"http://10.108.35.2:8080/ubuntu/fuelweb/x86_64 precise main\"

probably by adding new info in provisioning_serializer

[Dima Shulyak (dshulyak)]

In /var/lib/cobbler/config/systems.d/node-6.json

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109262
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3de611c59af339ff31366968e0d6834b9cf5b113
Submitter: Jenkins
Branch: master

commit 3de611c59af339ff31366968e0d6834b9cf5b113
Author: Dima Shulyak <email address hidden>
Date: Thu Jul 24 15:18:35 2014 +0300

    Disable multicast verification

    Will be enabled after resolving issues with
    upgrading network checker and mcagents

    Change-Id: Ibf402662f2402141a78c990a473de76ee2e83ae5
    Partial-Bug: #1348130

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109531
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=368ac101f36bf954921bc17c66bf763deb3faa5c
Submitter: Jenkins
Branch: master

commit 368ac101f36bf954921bc17c66bf763deb3faa5c
Author: Vladimir Sharshov <email address hidden>
Date: Fri Jul 25 13:37:26 2014 +0400

    Support custom package repositories for different fuel versions

    Change-Id: Ib4028941bfa5d858b4bf1f9e5768907477c8b1e0
    Implements: blueprint patch-openstack
    Closes-Bug: #1348130
    Related-Bug: #1348331

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-07-30_18-02-18",
    "ostf_sha": "b4c5efa51909404fd9ec1d0bbc38a31b200e1d6d",
    "build_number": "83",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "aed9ca4c68fee08a61c96c1d7271b6b42d31eb01",
    "production": "docker",
    "fuelmain_sha": "3f6b6e9f318f9ea4bd336b9250967dd41866fcbb",
    "astute_sha": "b16efcec6b4af1fb8669055c053fbabe188afa67",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "1bf6f4a96ddee3ee7297f9016650af26beb02129"

}

[Dmitry Pyzhov (dpyzhov)]

We will not deliver this fix in 5.0.1. And we will not support upgrades of master node in 5.0 branch after 5.0.1 release. So there is no point in backporting this fix to stable/5.0.

[Meg McRoberts (dreidellhasa)]

What should the release notes say about this issue (and the related https://bugs.launchpad.net/fuel/+bug/1348331?? Is there a work-around?

[Andrey Sledzinskiy (asledzinskiy)]

Issue isn't reproduced on old and new envs after upgrade to 5.0.1 - so I think 5.0.1 isn't afeected by this bug