rsync: on remote machine: --sparse-block=1024: unknown option on centos stream 8

Bug #1984035 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned

Bug Description

Image build jobs are not pushing overcloud images to image server on cs8 wallaby.
testproject on IBM cloud:
https://logserver.rdoproject.org/54/44454/2/check/periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-wallaby/c4d9a33/job-output.txt
```
mv overcloud-hardened-uefi-full.qcow2 overcloud-hardened-uefi-full.qcow2.md5 31a61a906cdcd8fccc322d597aa1dd5f
2022-08-09 03:33:52.146492 | primary | + '[' -f ironic-python-agent.tar ']'
2022-08-09 03:33:52.146521 | primary | + rsync --verbose --archive --no-perms --no-owner --no-group --delay-updates --relative 31a61a906cdcd8fccc322d597aa1dd5f <email address hidden>:/var/www/html/images/centos8/wallaby/rdo_trunk
2022-08-09 03:33:52.830177 | primary | Warning: Permanently added 'images.rdoproject.org,38.102.83.152' (ECDSA) to the list of known hosts.
2022-08-09 03:33:53.423201 | primary | rsync: on remote machine: --sparse-block=1024: unknown option
2022-08-09 03:33:53.443311 | primary | rsync error: syntax or usage error (code 1) at main.c(1582) [server=3.1.3]
2022-08-09 03:33:53.543934 | primary | rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-08-09 03:33:53.544006 | primary | rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]
2022-08-09 03:33:53.545872 | primary | cp: cannot stat '/home/zuul/*repo_setup*log': No such file or directory
2022-08-09 03:33:53.760662 | primary | ok: Runtime: 0:00:01.501241
```
testproject on Vexxhost
https://logserver.rdoproject.org/67/44467/1/check/periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-wallaby/43b9123/job-output.txt
```
+ '[' -f overcloud-hardened-uefi-full.qcow2 ']'
2022-08-09 04:59:37.419573 | primary | + mv overcloud-hardened-uefi-full.qcow2 overcloud-hardened-uefi-full.qcow2.md5 31a61a906cdcd8fccc322d597aa1dd5f
2022-08-09 04:59:37.421745 | primary | + '[' -f ironic-python-agent.tar ']'
2022-08-09 04:59:37.421796 | primary | + rsync --verbose --archive --no-perms --no-owner --no-group --delay-updates --relative 31a61a906cdcd8fccc322d597aa1dd5f <email address hidden>:/var/www/html/images/centos8/wallaby/rdo_trunk
2022-08-09 04:59:37.633780 | primary | Warning: Permanently added 'images.rdoproject.org,38.102.83.152' (ECDSA) to the list of known hosts.
2022-08-09 04:59:37.974967 | primary | rsync: on remote machine: --sparse-block=1024: unknown option
2022-08-09 04:59:37.995126 | primary | rsync error: syntax or usage error (code 1) at main.c(1582) [server=3.1.3]
2022-08-09 04:59:38.095734 | primary | rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-08-09 04:59:38.095794 | primary | rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]
2022-08-09 04:59:38.099624 | primary | cp: cannot stat '/home/zuul/*repo_setup*log': No such file or directory
2022-08-09 04:59:38.564844 | primary | ok: Runtime: 0:00:00.749122
2022-08-09 04:59:38.590928 |
```

We also checked the cs9 wallaby logs, it is passing
https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-buildimage-overcloud-full-wallaby/968ce65/job-output.txt
```
+ rsync --verbose --archive --no-perms --no-owner --no-group --delay-updates --relative 1f9075f3a633f50d5203cdc95045b150 <email address hidden>:/var/www/html/images/centos9/wallaby/rdo_trunk
2022-08-08 17:46:35.805859 | primary | Warning: Permanently added 'images.rdoproject.org' (ED25519) to the list of known hosts.
2022-08-08 17:46:36.134698 | primary | building file list ... done
2022-08-08 17:46:36.135645 | primary | 1f9075f3a633f50d5203cdc95045b150/
2022-08-08 17:46:36.135680 | primary | 1f9075f3a633f50d5203cdc95045b150/overcloud-full.tar
2022-08-08 17:46:39.890451 | primary | 1f9075f3a633f50d5203cdc95045b150/overcloud-full.tar.md5
2022-08-08 17:46:39.916754 | primary |
2022-08-08 17:46:39.916883 | primary | sent 832,807,699 bytes received 61 bytes 185,068,391.11 bytes/sec
2022-08-08 17:46:39.916896 | primary | total size is 832,604,213 speedup is 1.00
2022-08-08 17:46:39.918858 | primary | + rsync --verbose --archive --no-perms --no-owner --no-group --delay-updates --relative --delete --include 'tripleo-ci-testing**' --exclude '*' ./ <email address hidden>:/var/www/html/images/centos9/wallaby/rdo_trunk/
2022-08-08 17:46:39.954473 | primary | Warning: Permanently added 'images.rdoproject.org' (ED25519) to the list of known hosts.
2022-08-08 17:46:40.125146 | primary | building file list ... done
2022-08-08 17:46:40.126290 | primary | deleting tripleo-ci-testing
2022-08-08 17:46:40.126329 | primary | ./
2022-08-08 17:46:40.148323 | primary |
2022-08-08 17:46:40.148403 | primary | sent 94 bytes received 45 bytes 92.67 bytes/sec
2022-08-08 17:46:40.148413 | primary | total size is 0 speedup is 0.00
2022-08-08 17:46:40.148826 | primary | + ln -s 1f9075f3a633f50d5203cdc95045b150 tripleo-ci-testing
2022-08-08 17:46:40.150646 | primary | + rsync -av tripleo-ci-testing <email address hidden>:/var/www/html/images/centos9/wallaby/rdo_trunk
2022-08-08 17:46:40.188068 | primary | Warning: Permanently added 'images.rdoproject.org' (ED25519) to the list of known hosts.
2022-08-08 17:46:40.343543 | primary | sending incremental file list
2022-08-08 17:46:40.371711 | primary | tripleo-ci-testing -> 1f9075f3a633f50d5203cdc95045b150
2022-08-08 17:46:40.393513 | primary |
2022-08-08 17:46:40.393590 | primary | sent 104 bytes received 19 bytes 246.00 bytes/sec
2022-08-08 17:46:40.393600 | primary | total size is 32 speedup is 0.26
2022-08-08 17:46:40.394007 | primary | + popd
2022-08-08 17:46:40.394081 | primary | ~/workspace
2022-08-08 17:46:40.394131 | primary | + echo ======== UPLOAD CLOUD IMAGES COMPLETE
2022-08-08 17:46:40.394157 | primary | ======== UPLOAD CLOUD IMAGES COMPLETE
```

It seems the issue is related to CS8 wallaby. Adding a bug to track the issue.

Revision history for this message
chandan kumar (chkumar246) wrote :

on cs8 train
https://logserver.rdoproject.org/18/44418/11/check/periodic-tripleo-centos-8-buildimage-overcloud-full-train/c6e3672/job-output.txt
```
 + rsync --verbose --archive --no-perms --no-owner --no-group --delay-updates --relative 8cf307aefe47066dfa2b89be39b174f8 <email address hidden>:/var/www/html/images/centos8/train/rdo_trunk
2022-08-05 09:48:55.037039 | primary | Warning: Permanently added 'images.rdoproject.org,38.102.83.152' (ECDSA) to the list of known hosts.
2022-08-05 09:48:55.623050 | primary | building file list ... done
2022-08-05 09:48:55.672366 | primary | 8cf307aefe47066dfa2b89be39b174f8/
2022-08-05 09:48:59.824317 | primary | 8cf307aefe47066dfa2b89be39b174f8/overcloud-full.tar
2022-08-05 09:50:39.800332 | primary | 8cf307aefe47066dfa2b89be39b174f8/overcloud-full.tar.md5
2022-08-05 09:50:40.217910 | primary |
2022-08-05 09:50:40.217935 | primary | sent 1,130,308,085 bytes received 275,757 bytes 10,615,810.72 bytes/sec
2022-08-05 09:50:40.217942 | primary | total size is 1,550,714,933 speedup is 1.37
2022-08-05 09:50:40.218353 | primary | + rsync --verbose --archive --no-perms --no-owner --no-group --delay-updates --relative --delete --include 'tripleo-ci-testing**' --exclude '*' ./ <email address hidden>:/var/www/html/images/centos8/train/rdo_trunk/
2022-08-05 09:50:40.496357 | primary | Warning: Permanently added 'images.rdoproject.org,38.102.83.152' (ECDSA) to the list of known hosts.
2022-08-05 09:50:41.066819 | primary | building file list ... done
2022-08-05 09:50:41.118387 | primary | deleting tripleo-ci-testing
2022-08-05 09:50:41.118419 | primary | ./
2022-08-05 09:50:41.273307 | primary |
2022-08-05 09:50:41.273360 | primary | sent 94 bytes received 45 bytes 92.67 bytes/sec
2022-08-05 09:50:41.273367 | primary | total size is 0 speedup is 0.00
2022-08-05 09:50:41.273724 | primary | + ln -s 8cf307aefe47066dfa2b89be39b174f8 tripleo-ci-testing
2022-08-05 09:50:41.274754 | primary | + rsync -av tripleo-ci-testing <email address hidden>:/var/www/html/images/centos8/train/rdo_trunk
2022-08-05 09:50:41.522449 | primary | Warning: Permanently added 'images.rdoproject.org,38.102.83.152' (ECDSA) to the list of known hosts.
2022-08-05 09:50:42.062515 | primary | sending incremental file list
2022-08-05 09:50:42.136025 | primary | tripleo-ci-testing -> 8cf307aefe47066dfa2b89be39b174f8
2022-08-05 09:50:42.234249 | primary |
2022-08-05 09:50:42.234280 | primary | sent 105 bytes received 19 bytes 82.67 bytes/sec
2022-08-05 09:50:42.234289 | primary | total size is 32 speedup is 0.26
2022-08-05 09:50:42.234728 | primary | + popd
2022-08-05 09:50:42.234746 | primary | ~/workspace
2022-08-05 09:50:42.234753 | primary | + echo ======== UPLOAD CLOUD IMAGES COMPLETE
2022-08-05 09:50:42.234758 | primary | ======== UPLOAD CLOUD IMAGES COMPLETE
2022-08-05 09:50:42.236453 | primary | cp: cannot stat '/home/zuul/*repo_setup*log': No such file or directory
2022-08-05 09:50:42.395870 | primary | ok: Runtime: 0:01:47.694461
```
It is working fine.

Revision history for this message
chandan kumar (chkumar246) wrote :
summary: - rsync: on remote machine: --sparse-block=1024: unknown option
+ rsync: on remote machine: --sparse-block=1024: unknown option on centos
+ stream 8
Revision history for this message
chandan kumar (chkumar246) wrote :

downgrading rsync to rsync-3.1.3-14 seems to fix the issue.

tags: added: promotion-blocker
Revision history for this message
chandan kumar (chkumar246) wrote :
Revision history for this message
chandan kumar (chkumar246) wrote :

44471: [Temp] Downgrade rsync to 3.1.3-14 | https://review.rdoproject.org/r/c/config/+/44471

Revision history for this message
chandan kumar (chkumar246) wrote :
Download full text (6.7 KiB)

After applying the fix https://review.rdoproject.org/r/c/testproject/+/44468, the job is back to green.
https://logserver.rdoproject.org/68/44468/1/check/periodic-tripleo-centos-8-buildimage-overcloud-full-train/7117efe/job-output.txt
```
2022-08-09 10:03:51.750559 | primary | + echo 'UPLOAD_URL is <email address hidden>:/var/www/html/images/centos8/train/rdo_trunk'
2022-08-09 10:03:51.750580 | primary | UPLOAD_URL is <email address hidden>:/var/www/html/images/centos8/train/rdo_trunk
2022-08-09 10:03:51.750589 | primary | + [[ centos8 =~ centos8 ]]
2022-08-09 10:03:51.750603 | primary | + sudo dnf -y downgrade rsync-3.1.3-14.el8
2022-08-09 10:03:52.315816 | primary | Last metadata expiration check: 0:32:43 ago on Tue 09 Aug 2022 09:31:09 AM UTC.
2022-08-09 10:03:53.514378 | primary | Dependencies resolved.
2022-08-09 10:03:53.520671 | primary | ================================================================================
2022-08-09 10:03:53.520705 | primary | Package Architecture Version Repository Size
2022-08-09 10:03:53.520714 | primary | ================================================================================
2022-08-09 10:03:53.520721 | primary | Downgrading:
2022-08-09 10:03:53.520730 | primary | rsync x86_64 3.1.3-14.el8 quickstart-centos-base 405 k
2022-08-09 10:03:53.520736 | primary |
2022-08-09 10:03:53.520743 | primary | Transaction Summary
2022-08-09 10:03:53.520750 | primary | ================================================================================
2022-08-09 10:03:53.520756 | primary | Downgrade 1 Package
2022-08-09 10:03:53.520763 | primary |
2022-08-09 10:03:53.521108 | primary | Total download size: 405 k
2022-08-09 10:03:53.521416 | primary | Downloading Packages:
2022-08-09 10:03:53.634260 | primary | rsync-3.1.3-14.el8.x86_64.rpm 3.5 MB/s | 405 kB 00:00
2022-08-09 10:03:53.634730 | primary | --------------------------------------------------------------------------------
2022-08-09 10:03:53.634979 | primary | Total 3.5 MB/s | 405 kB 00:00
2022-08-09 10:03:53.635604 | primary | Running transaction check
2022-08-09 10:03:53.643912 | primary | Transaction check succeeded.
2022-08-09 10:03:53.644058 | primary | Running transaction test
2022-08-09 10:03:53.709510 | primary | Transaction test succeeded.
2022-08-09 10:03:53.713461 | primary | Running transaction
2022-08-09 10:03:54.027744 | primary | Preparing : 1/1
2022-08-09 10:03:54.143478 | primary | Downgrading : rsync-3.1.3-14.el8.x86_64 1/2
2022-08-09 10:03:54.331151 | primary | Cleanup : rsync-3.1.3-16.el8.x86_64 2/2
2022-08-09 10:03:54.635264 | primary | Running scriptlet: rsync-3.1.3-16.el8.x86_64 2/2
2022-08-09 10:03:54.635413 | primary | Verifying : rsync-3.1.3-14.el8.x86_64 1/2
2022-08-09 10:03:54.949653 | primary | Verifying : rsync-3.1.3-16.el8.x86_64 2/2
2022...

Read more...

Revision history for this message
Marios Andreou (marios-b) wrote :

no longer blocked after https://review.rdoproject.org/r/c/config/+/44471

but waiting on new release of rsync with the fix (via https://bugzilla.redhat.com/show_bug.cgi?id=2043753) so we can unpin ^^

Revision history for this message
Alan Pevec (apevec) wrote (last edit ):

https://bugzilla.redhat.com/show_bug.cgi?id=2043753

fixed in version rsync-3.1.3-17.el8

@Marios please review my blind revert via Gerrit UI: https://review.rdoproject.org/r/c/config/+/48954

Revision history for this message
Alan Pevec (apevec) wrote :

Train promoted so latest rsync in CS8 is good

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.