Fuel for OpenStack

rsync puppet modules partial failure breaks deployment

Bug #1322577 reported by Sergii Golovatiuk on 2014-05-23

This bug affects 6 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Vladimir Kuklin	Fuel for OpenStack 5.1

Bug Description

{"build_id": "2014-05-23_03-53-39", "mirantis": "yes", "build_number": "19", "ostf_sha": "5c479f04c35127576d35526650ec83b104f9a33d", "nailgun_sha": "bd09f89ef56176f64ad5decd4128933c96cb20f4", "production": "docker", "api": "1.0", "fuelmain_sha": "db2d153e62cb2b3034d33359d7e3db9d4742c811", "astute_sha": "9a0d86918724c1153b5f70bdae008dea8572fd3e", "release": "5.0", "fuellib_sha": "2ed4fbe1e04b85e83f1010ca23be7f5da34bd492"}

How to reproduce:
Ubuntu
Multi-node with HA
Neutron with GRE segmentation

4 -nodes for controllers
4-nodes for computes

Here is a log from agent

2014-05-23 10:24:32 ERR

10:24:32.375928 #974] ERROR -- : undefined method `[]' for nil:NilClass

2014-05-23 10:24:32 ERR

10:24:32.047275 #974] ERROR -- : HTTP PUT failed: #<HTTP::Message:0x7fe4163bf518 @http_body=#<HTTP::Message::Body:0x7fe4163bf4a0 @positions=nil, @body="Invalid MAC is specified", @size=0, @chunk_size=nil>, @http_header=#<HTTP::Message::Headers:0x7fe4163bf4f0 @body_encoding=nil, @chunked=false, @http_version="1.1", @reason_phrase="Bad Request", @dumped=false, @request_uri=#<URI::HTTP:0x7fe4163bf928 URL:http://10.108.0.2:8000/api/nodes/agent/>, @body_date=nil, @status_code=400, @header_item=[["Server", "nginx/1.0.15"], ["Date", "Fri, 23 May 2014 10:24:32 GMT"], ["Content-Type", "application/json"], ["Transfer-Encoding", "chunked"], ["Connection", "keep-alive"], ["Cache-Control", "store, no-cache, must-revalidate, post-check=0, pre-check=0"], ["Pragma", "no-cache"], ["Expires", "Thu, 01 Jan 1970 00:00:00 GMT"]], @request_method="PUT", @body_charset=nil, @request_absolute_uri=nil, @is_request=false, @body_size=0, @body_type=nil, @request_query=nil>, @peer_cert=nil>

Tags:

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-05-23:

log.tar.gz Edit (1.4 MiB, application/x-tar)

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-05-23:

Screen Shot 2014-05-23 at 2.07.03 PM.png Edit (397.5 KiB, image/png)

Vladimir Kuklin (vkuklin) on 2014-05-23

Changed in fuel:
milestone:	none → 5.1
importance:	Undecided → Critical
assignee:	nobody → Fuel OSCI Team (fuel-osci)

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-05-23:

pastebin from nailgunagent

http://pastebin.com/BLWVRQTb

Mike Scherbakov (mihgen) on 2014-05-23

no longer affects:

fuel/5.1.x

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2014-05-23:

Looks like rsync has no retry built in. Passing to Vladimir Sharshov to see if we can try to repeat this task with astute+mcollective. If not, we'll wrap it in some (beautiful) bash.

summary:

- Ubuntu deployment failed
+ rsync puppet modules partial failure breaks deployment

Matthew Mosesohn (raytrac3r) on 2014-05-23

Changed in fuel:
assignee:	Fuel OSCI Team (fuel-osci) → Vladimir Sharshov (vsharshov)
importance:	Critical → High
status:	New → Confirmed

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2014-05-23:

issue here: http://pastie.org/private/ujmgezjpofat1gqep5via
rsync fails to copy /etc/puppet/manifests on 2 of the nodes (but the rest is ok)
We should drop the priority because this only affects virtual deployments, but we should try to fix with a retry option for rsync (or optional retry within astute if necessary)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-03: Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/97443

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-03: Fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/97443
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=6ee3a5afdf36355fe38de964d65a62444dde957c
Submitter: Jenkins
Branch: master

commit 6ee3a5afdf36355fe38de964d65a62444dde957c
Author: Vladimir Sharshov <email address hidden>
Date: Tue Jun 3 13:58:48 2014 +0400

Add rsync retries mechanism to prevent temporary error with it

Change-Id: I5d3d4928e5b7adf415896afcf4ad4c02c0c0cb32
Closes-Bug: #1322577

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-01: Fix proposed to fuel-astute (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/103816

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-01: Fix merged to fuel-astute (stable/5.0)

Reviewed: https://review.openstack.org/103816
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=948c21e4e25318dbdbff125a0ede78e40f5d2def
Submitter: Jenkins
Branch: stable/5.0

commit 948c21e4e25318dbdbff125a0ede78e40f5d2def
Author: Vladimir Sharshov <email address hidden>
Date: Tue Jun 3 13:58:48 2014 +0400

Add rsync retries mechanism to prevent temporary error with it

    Change-Id: I5d3d4928e5b7adf415896afcf4ad4c02c0c0cb32
    Closes-Bug: #1322577
    (cherry picked from commit 6ee3a5afdf36355fe38de964d65a62444dde957c)

Revision history for this message

Evgeny Kozhemyakin (ekozhemyakin) wrote on 2014-07-10:

#10

The issue is still appears in v5.0.1.
https://answers.launchpad.net/fuel/+question/251076

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-11: Fix proposed to fuel-library (stable/5.0)

#11

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/106481

Revision history for this message

Joshua Dotson (tns9) wrote on 2014-07-11:

#12

Is it possible that the problem is the xinetd setup specified in the server.pp Puppet file isn't actually being deployed in the Docker container? I'm on 5.0.1.

Take a look:

http://paste.openstack.org/show/86166/

Revision history for this message

Joshua Dotson (tns9) wrote on 2014-07-11:

#13

Looks like my paste was truncated. Trying again:

http://paste.openstack.org/show/86167/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-12: Fix merged to fuel-library (master)

#14

Reviewed: https://review.openstack.org/106480
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b27bd563eddc456b5e15f800c562f4a664e3fe92
Submitter: Jenkins
Branch: master

commit b27bd563eddc456b5e15f800c562f4a664e3fe92
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Jul 11 19:49:36 2014 +0000

Increase xinetd connections for rsync

    - Increase connections to 512 with 10 second rate
    - Set per_source to UNLIMITED
    - Specify IPv4 flag

    Change-Id: Ia007301eca8e006e9792c73d0d35350588c48b96
    Closes-bug: 1330495
    Closes-bug: 1322577

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-12: Fix merged to fuel-library (stable/5.0)

#15

Reviewed: https://review.openstack.org/106481
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c0c0c135ef85bcf2c05f1b79dee65e61660e1f37
Submitter: Jenkins
Branch: stable/5.0

commit c0c0c135ef85bcf2c05f1b79dee65e61660e1f37
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Jul 11 19:49:36 2014 +0000

Increase xinet.d connections for rsync

    - Increase connections to 512 with 10 second rate
    - Set per_source to UNLIMITED
    - Specify IPv4 flag

    Change-Id: Ia007301eca8e006e9792c73d0d35350588c48b96
    Closes-bug: 1330495
    Closes-bug: 1322577

Revision history for this message

Sergii Golovatiuk (sgolovatiuk) wrote on 2014-07-12: Re: [Bug 1322577] Re: rsync puppet modules partial failure breaks deployment

#16

Joshua,

Proposed patch should improve rsync stability. It will allow to deploy up
to 20-30 nodes simultaneously. Also Vladimir Sharshov added retry
functionality to astude module which helps also. Personally, I deployed 1
controller with 15 compute nodes successfully.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Sat, Jul 12, 2014 at 12:40 AM, Joshua Dotson <email address hidden> wrote:

> Looks like my paste was truncated. Trying again:
>
> http://paste.openstack.org/show/86167/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1322577
>
> Title:
> rsync puppet modules partial failure breaks deployment
>
> Status in Fuel: OpenStack installer that works:
> Fix Committed
> Status in Fuel for OpenStack 5.0.x series:
> Fix Committed
>
> Bug description:
> {"build_id": "2014-05-23_03-53-39", "mirantis": "yes", "build_number":
> "19", "ostf_sha": "5c479f04c35127576d35526650ec83b104f9a33d",
> "nailgun_sha": "bd09f89ef56176f64ad5decd4128933c96cb20f4",
> "production": "docker", "api": "1.0", "fuelmain_sha":
> "db2d153e62cb2b3034d33359d7e3db9d4742c811", "astute_sha":
> "9a0d86918724c1153b5f70bdae008dea8572fd3e", "release": "5.0",
> "fuellib_sha": "2ed4fbe1e04b85e83f1010ca23be7f5da34bd492"}
>
> How to reproduce:
> Ubuntu
> Multi-node with HA
> Neutron with GRE segmentation
>
>
> 4 -nodes for controllers
> 4-nodes for computes
>
> Here is a log from agent
>
> 2014-05-23 10:24:32 ERR
>
> 10:24:32.375928 #974] ERROR -- : undefined method `[]' for
> nil:NilClass
>
> 2014-05-23 10:24:32 ERR
>
> 10:24:32.047275 #974] ERROR -- : HTTP PUT failed:
> #<HTTP::Message:0x7fe4163bf518
> @http_body=#<HTTP::Message::Body:0x7fe4163bf4a0 @positions=nil,
> @body="Invalid MAC is specified", @size=0, @chunk_size=nil>,
> @http_header=#<HTTP::Message::Headers:0x7fe4163bf4f0
> @body_encoding=nil, @chunked=false, @http_version="1.1",
> @reason_phrase="Bad Request", @dumped=false,
> @request_uri=#<URI::HTTP:0x7fe4163bf928
> URL:http://10.108.0.2:8000/api/nodes/agent/>, @body_date=nil,
> @status_code=400, @header_item=[["Server", "nginx/1.0.15"], ["Date",
> "Fri, 23 May 2014 10:24:32 GMT"], ["Content-Type",
> "application/json"], ["Transfer-Encoding", "chunked"], ["Connection",
> "keep-alive"], ["Cache-Control", "store, no-cache, must-revalidate,
> post-check=0, pre-check=0"], ["Pragma", "no-cache"], ["Expires", "Thu,
> 01 Jan 1970 00:00:00 GMT"]], @request_method="PUT", @body_charset=nil,
> @request_absolute_uri=nil, @is_request=false, @body_size=0,
> @body_type=nil, @request_query=nil>, @peer_cert=nil>
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1322577/+subscriptions
>

Joshua,

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Sat, Jul 12, 2014 at 12:40 AM, Joshua Dotson <josh@knoesis.org> wrote:

> Looks like my paste was truncated. Trying again:
>
> http://paste.openstack.org/show/86167/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1322577
>
> Title:
>   rsync puppet modules partial failure breaks deployment
>
> Status in Fuel: OpenStack installer that works:
>   Fix Committed
> Status in Fuel for OpenStack 5.0.x series:
>   Fix Committed
>
> Bug description:
>   {"build_id": "2014-05-23_03-53-39", "mirantis": "yes", "build_number":
>   "19", "ostf_sha": "5c479f04c35127576d35526650ec83b104f9a33d",
>   "nailgun_sha": "bd09f89ef56176f64ad5decd4128933c96cb20f4",
>   "production": "docker", "api": "1.0", "fuelmain_sha":
>   "db2d153e62cb2b3034d33359d7e3db9d4742c811", "astute_sha":
>   "9a0d86918724c1153b5f70bdae008dea8572fd3e", "release": "5.0",
>   "fuellib_sha": "2ed4fbe1e04b85e83f1010ca23be7f5da34bd492"}
>
>   How to reproduce:
>   Ubuntu
>   Multi-node with HA
>   Neutron with GRE segmentation
>
>
>   4 -nodes for controllers
>   4-nodes for computes
>
>   Here is a log from agent
>
>   2014-05-23 10:24:32     ERR
>
>   10:24:32.375928 #974] ERROR -- : undefined method `[]' for
>   nil:NilClass
>
>   2014-05-23 10:24:32     ERR
>
>   10:24:32.047275 #974] ERROR -- : HTTP PUT failed:
>   #<HTTP::Message:0x7fe4163bf518
>   @http_body=#<HTTP::Message::Body:0x7fe4163bf4a0 @positions=nil,
>   @body="Invalid MAC is specified", @size=0, @chunk_size=nil>,
>   @http_header=#<HTTP::Message::Headers:0x7fe4163bf4f0
>   @body_encoding=nil, @chunked=false, @http_version="1.1",
>   @reason_phrase="Bad Request", @dumped=false,
>   @request_uri=#<URI::HTTP:0x7fe4163bf928
>   URL:http://10.108.0.2:8000/api/nodes/agent/>, @body_date=nil,
>   @status_code=400, @header_item=[["Server", "nginx/1.0.15"], ["Date",
>   "Fri, 23 May 2014 10:24:32 GMT"], ["Content-Type",
>   "application/json"], ["Transfer-Encoding", "chunked"], ["Connection",
>   "keep-alive"], ["Cache-Control", "store, no-cache, must-revalidate,
>   post-check=0, pre-check=0"], ["Pragma", "no-cache"], ["Expires", "Thu,
>   01 Jan 1970 00:00:00 GMT"]], @request_method="PUT", @body_charset=nil,
>   @request_absolute_uri=nil, @is_request=false, @body_size=0,
>   @body_type=nil, @request_query=nil>, @peer_cert=nil>
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1322577/+subscriptions
>

Vladimir Kuklin (vkuklin) on 2014-08-14

Changed in fuel:
status:	Fix Committed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-14: Fix proposed to fuel-library (master)

#17

Fix proposed to branch: master
Review: https://review.openstack.org/114246

Changed in fuel:
assignee:	Vladimir Sharshov (vsharshov) → Vladimir Kuklin (vkuklin)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-14: Fix proposed to fuel-library (stable/5.0)

#18

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114247

Vladimir Kuklin (vkuklin) on 2014-08-14

no longer affects:

fuel/5.0.x

Vladimir Kuklin (vkuklin) on 2014-08-15

tags:

added: release-notes

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-15: Change abandoned on fuel-library (stable/5.0)

#19

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/114247

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-18: Fix merged to fuel-library (master)

#20

Reviewed: https://review.openstack.org/114246
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b2acea613501200d6320bcecd3ef47637c2173c6
Submitter: Jenkins
Branch: master

commit b2acea613501200d6320bcecd3ef47637c2173c6
Author: Vladimir Kuklin <email address hidden>
Date: Thu Aug 14 17:28:09 2014 +0400

Set rsync parameters for nailgun rsync server

Set xinetd parameters for nailgun rsyncd server
as it is configured by separate template

Change-Id: I2e3ccb467d03d4c6f84865b76b649f9455442785
Closes-bug: #1322577

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-09-02: Fix proposed to puppet-modules/puppetlabs-rsync (0.4.0-mos)

#21

Fix proposed to branch: 0.4.0-mos
Change author: Denis Egorenko <email address hidden>
Review: https://review.fuel-infra.org/11071

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-09-21: Fix merged to puppet-modules/puppetlabs-rsync (0.4.0-mos)

#22

Reviewed: https://review.fuel-infra.org/11071
Submitter: Ivan Berezovskiy <email address hidden>
Branch: 0.4.0-mos

Commit: fba3de1a9adbfeff27cae2999e3e1b6b6625f005
Author: Denis Egorenko <email address hidden>
Date: Fri Sep 18 08:54:00 2015

Increase xinetd connections for rsync

- Increase connections to 512 with 10 second rate
- Set per_source to UNLIMITED
- Specify IPv4 flag

Closes-bug: 1330495
Closes-bug: 1322577

Change-Id: Ia007301eca8e006e9792c73d0d35350588c48b96

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.