Fuel for OpenStack

Fuel web backend throws warning "Invalid MAC is specified" with nic bonding

Series newton
Bug #1467505

Bug #1467505 reported by Claude Durocher on 2015-06-22

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Confirmed	Medium	Fuel Sustaining	Fuel for OpenStack 10.0
6.1.x	Won't Fix	Medium	Fuel Python (Deprecated)	Fuel for OpenStack 6.1-updates
7.0.x	Won't Fix	Medium	Fuel Python (Deprecated)	Fuel for OpenStack 7.0-updates
8.0.x	Won't Fix	Medium	Fuel Python (Deprecated)	Fuel for OpenStack 8.0
Mitaka	Won't Fix	Medium	Fuel Python (Deprecated)	Fuel for OpenStack 9.0
Newton	Confirmed	Medium	Fuel Sustaining	Fuel for OpenStack 10.0

Bug Description

Environment

{"build_id": "2015-06-19_05-40-59", "build_number": "280", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-19_05-40-59", "build_number": "280", "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3", "feature_groups": ["experimental"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3", "feature_groups": ["experimental"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"}

Steps to reproduce

-Deploy 3 controller nodes, 3 compute nodes and 3 ceph nodes on Ubuntu 14.04
-For the controller nodes, bond 3 nics for the public interface using balance_rr (no bonding for compute or ceph nodes)
-After successfull deployment, connect to Fuel master node with webui
-Fuel web backend logs generates those warnings very minutes or so (not sure if this has any impacts on Openstack operation):

[7f589714b740] (base) Invalid MAC is specified
[pid: 636|app: 0|req: 4035/47925] 10.20.0.3 () {36 vars in 524 bytes} [Mon Jun 22 12:44:15 2015] PUT /api/nodes/agent/ => generated 53 bytes in 33 msecs (HTTP/1.1 400) 4 headers in 187 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6188/47926] 10.20.0.3 () {36 vars in 513 bytes} [Mon Jun 22 12:44:15 2015] POST /api/nodes/ => generated 79 bytes in 10 msecs (HTTP/1.1 403) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6189/47927] 10.20.0.8 () {36 vars in 524 bytes} [Mon Jun 22 12:44:16 2015] PUT /api/nodes/agent/ => generated 25 bytes in 33 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6190/47928] 10.17.74.53 () {46 vars in 983 bytes} [Mon Jun 22 12:44:20 2015] GET /api/logs?node=&source=app&level=INFO&from=3378687&to=3596156&_=1434977033855 => generated 789 bytes in 184 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6191/47929] 10.20.0.10 () {36 vars in 525 bytes} [Mon Jun 22 12:44:22 2015] PUT /api/nodes/agent/ => generated 25 bytes in 34 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6192/47930] 10.17.74.53 () {46 vars in 915 bytes} [Mon Jun 22 12:44:22 2015] GET /api/nodes/allocation/stats?_=1434977033856 => generated 30 bytes in 33 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 634|app: 0|req: 3411/47931] 10.17.74.53 () {46 vars in 897 bytes} [Mon Jun 22 12:44:22 2015] GET /api/notifications?_=1434977033857 => generated 69926 bytes in 245 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6193/47932] 10.20.0.6 () {36 vars in 524 bytes} [Mon Jun 22 12:44:22 2015] PUT /api/nodes/agent/ => generated 25 bytes in 30 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6194/47933] 10.20.0.12 () {36 vars in 525 bytes} [Mon Jun 22 12:44:24 2015] PUT /api/nodes/agent/ => generated 25 bytes in 33 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 641|app: 0|req: 3319/47934] 10.20.0.13 () {36 vars in 525 bytes} [Mon Jun 22 12:44:25 2015] PUT /api/nodes/agent/ => generated 25 bytes in 34 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 637|app: 0|req: 12570/47935] 10.17.74.53 () {46 vars in 983 bytes} [Mon Jun 22 12:44:26 2015] GET /api/logs?node=&source=app&level=INFO&from=3378687&to=3597147&_=1434977033858 => generated 65 bytes in 8 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 639|app: 0|req: 8856/47936] 10.20.0.15 () {36 vars in 525 bytes} [Mon Jun 22 12:44:30 2015] PUT /api/nodes/agent/ => generated 25 bytes in 34 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 640|app: 0|req: 5407/47937] 10.17.74.53 () {46 vars in 983 bytes} [Mon Jun 22 12:44:31 2015] GET /api/logs?node=&source=app&level=INFO&from=3378687&to=3598763&_=1434977033859 => generated 65 bytes in 5 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 640|app: 0|req: 5408/47938] 10.17.74.53 () {44 vars in 832 bytes} [Mon Jun 22 12:44:32 2015] GET /api/version?_=1434977072580 => generated 1192 bytes in 13 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 640|app: 0|req: 5409/47939] 10.17.74.53 () {46 vars in 887 bytes} [Mon Jun 22 12:44:32 2015] GET /api/settings?_=1434977072581 => generated 1355 bytes in 29 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 640|app: 0|req: 5410/47940] 10.17.74.53 () {46 vars in 891 bytes} [Mon Jun 22 12:44:33 2015] GET /api/clusters/1?_=1434977072582 => generated 233 bytes in 12 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6195/47941] 10.17.74.53 () {46 vars in 913 bytes} [Mon Jun 22 12:44:33 2015] GET /api/clusters/1/attributes?_=1434977072583 => generated 17478 bytes in 37 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 636|app: 0|req: 4036/47942] 10.17.74.53 () {46 vars in 907 bytes} [Mon Jun 22 12:44:33 2015] GET /api/tasks?cluster_id=1&_=1434977072585 => generated 1465 bytes in 153 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 634|app: 0|req: 3412/47943] 10.17.74.53 () {46 vars in 907 bytes} [Mon Jun 22 12:44:33 2015] GET /api/nodes?cluster_id=1&_=1434977072584 => generated 44107 bytes in 333 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 635|app: 0|req: 4146/47944] 10.17.74.53 () {46 vars in 891 bytes} [Mon Jun 22 12:44:33 2015] GET /api/releases/2?_=1434977072587 => generated 31117 bytes in 28 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 638|app: 0|req: 6196/47945] 10.17.74.53 () {46 vars in 951 bytes} [Mon Jun 22 12:44:33 2015] GET /api/clusters/1/network_configuration/neutron?_=1434977072586 => generated 2437 bytes in 73 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 636|app: 0|req: 4037/47946] 10.17.74.53 () {46 vars in 915 bytes} [Mon Jun 22 12:44:33 2015] GET /api/nodes/allocation/stats?_=1434977072588 => generated 30 bytes in 27 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 637|app: 0|req: 12571/47947] 10.17.74.53 () {46 vars in 895 bytes} [Mon Jun 22 12:44:33 2015] GET /api/logs/sources?_=1434977072590 => generated 37722 bytes in 19 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 641|app: 0|req: 3320/47948] 10.17.74.53 () {46 vars in 897 bytes} [Mon Jun 22 12:44:33 2015] GET /api/notifications?_=1434977072589 => generated 69926 bytes in 66 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)

Tags:

Revision history for this message

Claude Durocher (claude-d) wrote on 2015-06-22:

I forgot to mention : ip 10.17.74.53 is the address of the WS accessing Fuel webui in the error logs.

Andrey Maximov (maximov) on 2015-06-22

Changed in fuel:
assignee:	nobody → Fuel Python Team (fuel-python)

Revision history for this message

Aleksey Kasatkin (alekseyk-ru) wrote on 2015-06-22:

Claude, thank you for the report. Please provide diagnostic snapshot if possible.
Do you see any nodes as offline?

"Invalid MAC is specified" - it is from here:
https://github.com/stackforge/fuel-web/blob/3830bdcb28ec050eed399fe782cc3dd5fbf31bde/nailgun/nailgun/api/v1/validators/node.py#L204-211

It's needed to see snapshot to estimate whether this affects operation or reproduce the situation.

Sergey Vasilenko (xenolog) on 2015-06-23

Changed in fuel:
status:	New → Incomplete

Revision history for this message

Claude Durocher (claude-d) wrote on 2015-06-23: Re: [Bug 1467505] Re: Fuel web backend throws warning "Invalid MAC is specified" with nic bonding

Download full text (8.4 KiB)

AFAIK, the deployed cloud works ok but I'm still testing.

Note that this error is thrown with different types of bonding (I also
tested with LACP and balance-alb).

Send a private email to Alesksey for a link to the snapshot.

Le mar. 23 juin 2015 à 04:05, Sergey Vasilenko <email address hidden>
a écrit :

AFAIK, the deployed cloud works ok but I'm still testing.

Note that this error is thrown with different types of bonding (I also
tested with LACP and balance-alb).

Send a private email to Alesksey for a link to the snapshot.

Le mar. 23 juin 2015 à 04:05, Sergey Vasilenko <1467505@bugs.launchpad.net>
a écrit :

> ** Changed in: fuel
>        Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1467505
>
> Title:
>   Fuel web backend throws warning "Invalid MAC is specified" with nic
>   bonding
>
> Status in Fuel: OpenStack installer that works:
>   Incomplete
>
> Bug description:
>   Environment
>
>   {"build_id": "2015-06-19_05-40-59", "build_number": "280",
>   "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id":
>   "2015-06-19_05-40-59", "build_number": "280", "api": "1.0", "fuel-
>   library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab",
>   "nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3",
>   "feature_groups": ["experimental"], "openstack_version":
>   "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha":
>   "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha":
>   "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha":
>   "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1",
>   "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"}}},
>   "auth_required": true, "api": "1.0", "fuel-library_sha":
>   "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha":
>   "126e50e624b1b97043ab87dbf1298312b9771df3", "feature_groups":
>   ["experimental"], "openstack_version": "2014.2.2-6.1", "production":
>   "docker", "python-fuelclient_sha":
>   "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha":
>   "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha":
>   "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1",
>   "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"}
>
>   Steps to reproduce
>
>   -Deploy 3 controller nodes, 3 compute nodes and 3 ceph nodes on Ubuntu
> 14.04
>   -For the controller nodes, bond 3 nics for the public interface using
> balance_rr (no bonding for compute or ceph nodes)
>   -After successfull deployment, connect to Fuel master node with webui
>   -Fuel web backend logs generates those warnings very minutes or so (not
> sure if this has any impacts on Openstack operation):
>
>   [7f589714b740] (base) Invalid MAC is specified
>   [pid: 636|app: 0|req: 4035/47925] 10.20.0.3 () {36 vars in 524 bytes}
> [Mon Jun 22 12:44:15 2015] PUT /api/nodes/agent/ => generated 53 bytes in
> 33 msecs (HTTP/1.1 400) 4 headers in 187 bytes (2 switches on core 0)
>   [pid: 638|app: 0|req: 6188/47926] 10.20.0.3 () {36 vars in 513 bytes}
> [Mon Jun 22 12:44:15 2015] POST /api/nodes/ => generated 79 bytes in 10
> msecs (HTTP/1.1 403) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 638|app: 0|req: 6189/47927] 10.20.0.8 () {36 vars in 524 bytes}
> [Mon Jun 22 12:44:16 2015] PUT /api/nodes/agent/ => generated 25 bytes in
> 33 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 638|app: 0|req: 6190/47928] 10.17.74.53 () {46 vars in 983 bytes}
> [Mon Jun 22 12:44:20 2015] GET
> /api/logs?node=&source=app&level=INFO&from=3378687&to=3596156&_=1434977033855
> => generated 789 bytes in 184 msecs (HTTP/1.1 200) 4 headers in 185 bytes
> (2 switches on core 0)
>   [pid: 638|app: 0|req: 6191/47929] 10.20.0.10 () {36 vars in 525 bytes}
> [Mon Jun 22 12:44:22 2015] PUT /api/nodes/agent/ => generated 25 bytes in
> 34 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 638|app: 0|req: 6192/47930] 10.17.74.53 () {46 vars in 915 bytes}
> [Mon Jun 22 12:44:22 2015] GET /api/nodes/allocation/stats?_=1434977033856
> => generated 30 bytes in 33 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 634|app: 0|req: 3411/47931] 10.17.74.53 () {46 vars in 897 bytes}
> [Mon Jun 22 12:44:22 2015] GET /api/notifications?_=1434977033857 =>
> generated 69926 bytes in 245 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 638|app: 0|req: 6193/47932] 10.20.0.6 () {36 vars in 524 bytes}
> [Mon Jun 22 12:44:22 2015] PUT /api/nodes/agent/ => generated 25 bytes in
> 30 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 638|app: 0|req: 6194/47933] 10.20.0.12 () {36 vars in 525 bytes}
> [Mon Jun 22 12:44:24 2015] PUT /api/nodes/agent/ => generated 25 bytes in
> 33 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 641|app: 0|req: 3319/47934] 10.20.0.13 () {36 vars in 525 bytes}
> [Mon Jun 22 12:44:25 2015] PUT /api/nodes/agent/ => generated 25 bytes in
> 34 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 637|app: 0|req: 12570/47935] 10.17.74.53 () {46 vars in 983 bytes}
> [Mon Jun 22 12:44:26 2015] GET
> /api/logs?node=&source=app&level=INFO&from=3378687&to=3597147&_=1434977033858
> => generated 65 bytes in 8 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 639|app: 0|req: 8856/47936] 10.20.0.15 () {36 vars in 525 bytes}
> [Mon Jun 22 12:44:30 2015] PUT /api/nodes/agent/ => generated 25 bytes in
> 34 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
>   [pid: 640|app: 0|req: 5407/47937] 10.17.74.53 () {46 vars in 983 bytes}
> [Mon Jun 22 12:44:31 2015] GET
> /api/logs?node=&source=app&level=INFO&from=3378687&to=3598763&_=1434977033859
> => generated 65 bytes in 5 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 640|app: 0|req: 5408/47938] 10.17.74.53 () {44 vars in 832 bytes}
> [Mon Jun 22 12:44:32 2015] GET /api/version?_=1434977072580 => generated
> 1192 bytes in 13 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on
> core 0)
>   [pid: 640|app: 0|req: 5409/47939] 10.17.74.53 () {46 vars in 887 bytes}
> [Mon Jun 22 12:44:32 2015] GET /api/settings?_=1434977072581 => generated
> 1355 bytes in 29 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on
> core 0)
>   [pid: 640|app: 0|req: 5410/47940] 10.17.74.53 () {46 vars in 891 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/clusters/1?_=1434977072582 => generated
> 233 bytes in 12 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on
> core 0)
>   [pid: 638|app: 0|req: 6195/47941] 10.17.74.53 () {46 vars in 913 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/clusters/1/attributes?_=1434977072583
> => generated 17478 bytes in 37 msecs (HTTP/1.1 200) 4 headers in 185 bytes
> (2 switches on core 0)
>   [pid: 636|app: 0|req: 4036/47942] 10.17.74.53 () {46 vars in 907 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/tasks?cluster_id=1&_=1434977072585 =>
> generated 1465 bytes in 153 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 634|app: 0|req: 3412/47943] 10.17.74.53 () {46 vars in 907 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/nodes?cluster_id=1&_=1434977072584 =>
> generated 44107 bytes in 333 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 635|app: 0|req: 4146/47944] 10.17.74.53 () {46 vars in 891 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/releases/2?_=1434977072587 => generated
> 31117 bytes in 28 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches
> on core 0)
>   [pid: 638|app: 0|req: 6196/47945] 10.17.74.53 () {46 vars in 951 bytes}
> [Mon Jun 22 12:44:33 2015] GET
> /api/clusters/1/network_configuration/neutron?_=1434977072586 => generated
> 2437 bytes in 73 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on
> core 0)
>   [pid: 636|app: 0|req: 4037/47946] 10.17.74.53 () {46 vars in 915 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/nodes/allocation/stats?_=1434977072588
> => generated 30 bytes in 27 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 637|app: 0|req: 12571/47947] 10.17.74.53 () {46 vars in 895 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/logs/sources?_=1434977072590 =>
> generated 37722 bytes in 19 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>   [pid: 641|app: 0|req: 3320/47948] 10.17.74.53 () {46 vars in 897 bytes}
> [Mon Jun 22 12:44:33 2015] GET /api/notifications?_=1434977072589 =>
> generated 69926 bytes in 66 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2
> switches on core 0)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1467505/+subscriptions
>
--

Claude
---
"Quand le dernier arbre aura été abattu, la dernière rivière empoisonnée et
le dernier poisson péché, alors l'homme s'apercevra que l'argent ne se
mange pas."

Aleksey Kasatkin (alekseyk-ru) on 2015-06-26

Changed in fuel:
status:	Incomplete → Confirmed

Alexander Kislitsky (akislitsky) on 2015-06-26

Changed in fuel:
milestone:	none → 7.0
importance:	Undecided → Medium

Revision history for this message

Aleksey Kasatkin (alekseyk-ru) wrote on 2015-06-26:

Here's the link to the dump file :

https://drive.google.com/file/d/0B1CFwOEG9nMtRWw0eTBTNk5ySms/view?usp=sharing

Revision history for this message

Vladimir Sharshov (vsharshov) wrote on 2015-08-05:

It does not affect deployment. Moving to 8.0.

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-09-30:

Won't Fix for 7.0-updates and 6.1-updates because of Medium importance

Dmitry Pyzhov (dpyzhov) on 2015-10-12

Changed in fuel:
milestone:	7.0-updates → 8.0
no longer affects:	fuel/8.0.x

Dmitry Pyzhov (dpyzhov) on 2015-10-22

tags:

added: area-python

Revision history for this message

Alexander Kislitsky (akislitsky) wrote on 2016-01-15:

We passed SCF in 8.0. Moving the bug to 9.0.

Changed in fuel:
milestone:	8.0 → 9.0

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-01-22:

Download full text (5.7 KiB)

Reproduced:

[root@nailgun pl]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

Scenario:
Install the database plugin on the master node.
Create Ubuntu, Neutron Vlan, all ceph cluster.
Add 3 nodes with contoller/ceph role.
Add 3 nodes with db role.
Add 1 compute node.
Run network verification.
Deploy changes.
Run network verification.
Run OSTF tests.
Add 1 db node and deploy changes. <- Error here

12:54:13.471686 #1720] ERROR -- : HTTP PUT failed: #<HTTP::Message:0x000000025316d0 @http_header=#<HTTP::Message::Headers:0x000000025316a8 @http_version="1.1", @body_size=0, @chunked=false, @request_method="PUT", @request_uri=#<URI::HTTPS:0x00000002531b58 URL:https://10.109.0.2:8443/api/nodes/agent/>, @request_query=nil, @request_absolute_uri=nil, @status_code=400, @reason_phrase="Bad Request", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Server", "nginx"], ["Date", "Thu, 21 Jan 2016 12:54:13 GMT"], ["Content-Type", "text/html; charset=utf-8"], ["Transfer-Encoding", "chunked"], ["Connection", "keep-alive"], ["Cache-Control", "store, no-cache, must-revalidate, post-check=0, pre-check=0"], ["Pragma", "no-cache"], ["Expires", "Thu, 01 Jan 1970 00:00:00 GMT"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate subject=/C=US/ST=California/O=Fuel/OU=Fuel Deployment <email address hidden>, issuer=/C=US/ST=California/O=Fuel/OU=Fuel Deployment <email address hidden>, serial=11524342662413337222, not_before=2016-01-15 12:55:53 UTC, not_after=2026-01-12 12:55:53 UTC>, @http_body=#<HTTP::Message::Body:0x00000002531630 @body="{\"message\": \"Invalid MAC is specified\", \"errors\": []}", @size=0, @positions=nil, @chunk_size=nil>>

Reproduced:

[root@nailgun pl]# cat /etc/fuel/version.yaml 
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

12:54:13.471686 #1720] ERROR -- : HTTP PUT failed: #<HTTP::Message:0x000000025316d0 @http_header=#<HTTP::Message::Headers:0x000000025316a8 @http_version="1.1", @body_size=0, @chunked=false, @request_method="PUT", @request_uri=#<URI::HTTPS:0x00000002531b58 URL:https://10.109.0.2:8443/api/nodes/agent/>, @request_query=nil, @request_absolute_uri=nil, @status_code=400, @reason_phrase="Bad Request", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Server", "nginx"], ["Date", "Thu, 21 Jan 2016 12:54:13 GMT"], ["Content-Type", "text/html; charset=utf-8"], ["Transfer-Encoding", "chunked"], ["Connection", "keep-alive"], ["Cache-Control", "store, no-cache, must-revalidate, post-check=0, pre-check=0"], ["Pragma", "no-cache"], ["Expires", "Thu, 01 Jan 1970 00:00:00 GMT"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate subject=/C=US/ST=California/O=Fuel/OU=Fuel Deployment Team/CN=fuel.master.local/emailAddress=root@fuel.master.local, issuer=/C=US/ST=California/O=Fuel/OU=Fuel Deployment Team/CN=fuel.master.local/emailAddress=root@fuel.master.local, serial=11524342662413337222, not_before=2016-01-15 12:55:53 UTC, not_after=2026-01-12 12:55:53 UTC>, @http_body=#<HTTP::Message::Body:0x00000002531630 @body="{\"message\": \"Invalid MAC is specified\", \"errors\": []}", @size=0, @positions=nil, @chunk_size=nil>>

Before deployment:
[root@nailgun pl]# fuel nodes
id | status   | name             | cluster | ip          | mac               | roles | pending_roles | online | group_id
---|----------|------------------|---------|-------------|-------------------|-------|---------------|--------|---------
11 | discover | Untitled (4a:f5) | None    | 10.109.0.4  | 64:6e:02:35:4a:f5 |       |               | True   | None    
12 | discover | Untitled (aa:d8) | None    | 10.109.0.22 | 64:9b:29:77:aa:d8 |       |               | True   | None    
13 | discover | Untitled (02:cd) | None    | 10.109.0.23 | 64:1f:f0:0a:02:cd |       |               | True   | None    
17 | discover | Untitled (e3:02) | None    | 10.109.0.24 | 64:a3:78:67:e3:02 |       |               | True   | None    
16 | discover | Untitled (09:36) | None    | 10.109.0.25 | 64:e0:87:ea:09:36 |       |               | True   | None    
15 | discover | Untitled (11:d6) | None    | 10.109.0.27 | 64:2e:1e:34:11:d6 |       |               | True   | None    
18 | discover | Untitled (93:1b) | None    | 10.109.0.20 | 64:28:16:9b:93:1b |       |               | True   | None    
10 | discover | Untitled (2a:c7) | None    | 10.109.0.8  | 64:bc:0d:fc:2a:c7 |       |               | True   | None    
14 | discover | Untitled (a8:2a) | None    | 10.109.0.26 | 64:bc:cd:27:a8:2a |       |               | True   | None

After deployment:
[root@nailgun pl]# fuel nodes
id | status      | name             | cluster | ip          | mac               | roles                | pending_roles | online | group_id
---|-------------|------------------|---------|-------------|-------------------|----------------------|---------------|--------|---------
17 | provisioned | Untitled (e3:02) | 2       | 10.109.0.24 | 64:a3:78:67:e3:02 | standalone-database  |               | True   | 2       
14 | ready       | Untitled (a8:2a) | 2       | 10.109.0.26 | 64:bc:cd:27:a8:2a | standalone-database  |               | True   | 2       
16 | ready       | Untitled (09:36) | 2       | 10.109.0.25 | 22:0a:96:52:c4:d9 | standalone-database  |               | True   | 2       
15 | ready       | Untitled (11:d6) | 2       | 10.109.0.27 | 64:2e:1e:34:11:d6 | standalone-database  |               | True   | 2       
10 | error       | Untitled (2a:c7) | 2       | 10.109.0.8  | 64:bc:0d:fc:2a:c7 | ceph-osd, controller |               | True   | 2       
18 | discover    | Untitled (93:1b) | None    | 10.109.0.20 | 64:28:16:9b:93:1b |                      |               | True   | None    
12 | error       | Untitled (aa:d8) | 2       | 10.109.0.22 | 06:7b:e1:27:d3:c8 | ceph-osd, controller |               | True   | 2       
11 | error       | Untitled (4a:f5) | 2       | 10.109.0.4  | 3a:98:f3:96:8f:91 | ceph-osd, controller |               | True   | 2       
13 | ready       | Untitled (02:cd) | 2       | 10.109.0.23 | 64:1f:f0:0a:02:cd | compute              |               | True   | 2

Dignostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B1CktchMwAXHaWxXS0tEQU5LR28/view?usp=sharing

Also env snapshot is available. Please contact me.

Revision history for this message

Vladimir Khlyunev (vkhlyunev) wrote on 2016-01-22:

Raise to high because the issue was reproduced by acceptance testing - see @Dmitry's comment above

Revision history for this message

Dmitry Pyzhov (dpyzhov) wrote on 2016-01-22:

#10

This message doesn't break anything and cannot be High according our triage rules. Moving back to medium.

no longer affects:	fuel/mitaka
Changed in fuel:
importance:	High → Medium

Dmitry Pyzhov (dpyzhov) on 2016-03-03

tags:

added: team-network

Revision history for this message

Bug Checker Bot (bug-checker) wrote on 2016-03-28: Autochecker

#11

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags:

added: need-info

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.