Cinder LVM driver and ipv6 broken

Bug #1696866 reported by Ben Nemec
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Gorka Eguileor
OpenStack Compute (nova)
Invalid
Undecided
Unassigned
oslo.config
Invalid
Undecided
Unassigned
tripleo
Fix Released
Medium
Ben Nemec

Bug Description

The following traceback is seen in the cinder volume logs if you deploy with the default LVM driver and ipv6:

2017-06-08 21:50:40.665 38554 CRITICAL cinder [req-a7aff57e-2042-48f2-b1eb-962f79ccda01 - - - - -] ConfigFileValueError: Value for option iscsi_ip_address is not valid: [fd00:fd00:fd00:3000::12] is not a valid host address
2017-06-08 21:50:40.665 38554 ERROR cinder Traceback (most recent call last):
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>
2017-06-08 21:50:40.665 38554 ERROR cinder sys.exit(main())
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 120, in main
2017-06-08 21:50:40.665 38554 ERROR cinder launcher.wait()
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 581, in wait
2017-06-08 21:50:40.665 38554 ERROR cinder self.conf.log_opt_values(LOG, logging.DEBUG)
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2799, in log_opt_values
2017-06-08 21:50:40.665 38554 ERROR cinder _sanitize(opt, getattr(group_attr, opt_name)))
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3277, in __getattr__
2017-06-08 21:50:40.665 38554 ERROR cinder return self._conf._get(name, self._group)
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2841, in _get
2017-06-08 21:50:40.665 38554 ERROR cinder value = self._do_get(name, group, namespace)
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2884, in _do_get
2017-06-08 21:50:40.665 38554 ERROR cinder % (opt.name, str(ve)))
2017-06-08 21:50:40.665 38554 ERROR cinder ConfigFileValueError: Value for option iscsi_ip_address is not valid: [fd00:fd00:fd00:3000::12] is not a valid host address

The problem appears to be that we bracket the value. If I remove the brackets then cinder volume works correctly.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/472455

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (master)

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/472455
Reason: I don't think this actually works. We need the brackets. Will have to fix oslo/netaddr/socket to get this working.

Revision history for this message
Ben Nemec (bnemec) wrote :

Okay, if I un-bracket the IP then cinder-volume works, but the volume can't be attached to the nova instance. I've attached the nova-compute log from the attempted boot.

It looks like there might be a bug in os-brick? I'm going to keep investigating, but I'll add it to the bug report.

Revision history for this message
Ben Nemec (bnemec) wrote :

I did some more digging yesterday but concluded that this is beyond my ability to debug in a reasonable timeframe. I no longer think os-brick is the problem though - it looks like the data passed in to os-brick is already bad, so I think it's something in Cinder or Nova that is not correctly handling the ipv6 iscsi address. I'm going to add both projects to this bug and let them determine who needs to fix what.

To recap, if we bracket iscsi_ip_address then oslo.config fails to validate the value, if we remove the brackets then cinder-volume starts correctly and is able to create the volume, but it can't be attached to the Nova instance. This appears to be because the iscsi port is being appended to the ipv6 address without first bracketing the address to separate the address and port portions (see the attached nova-compute logs).

Here are the RDO package versions I was testing with:

$ rpm -qa | egrep "nova|cinder"
puppet-cinder-11.1.0-0.20170605202131.257519d.el7.centos.noarch
openstack-nova-conductor-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
python2-cinderclient-2.0.1-0.20170320163530.d0790e3.el7.centos.noarch
puppet-nova-11.1.0-0.20170605232112.27baec7.el7.centos.noarch
python-cinder-11.0.0-0.20170521083557.cedf99f.el7.centos.noarch
openstack-nova-common-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-api-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-placement-api-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-cinder-11.0.0-0.20170521083557.cedf99f.el7.centos.noarch
python-nova-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-migration-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-scheduler-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-console-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
python2-novaclient-8.0.0-0.20170517113627.e1b9e76.el7.centos.noarch
openstack-nova-compute-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-novncproxy-16.0.0-0.20170521033533.99bd334.el7.centos.noarch

no longer affects: os-brick
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Actually sounds like oslo.config needs to be updated to handle IPv6 addressing to me.

Revision history for this message
Alex Schultz (alex-schultz) wrote :

I think the correct fix is for cinder to stop assuming the IP is ok to just concatenate with the port, example https://github.com/openstack/cinder/blob/a55a6b5c717c0802288ab802a8fbff287a4b6b66/cinder/volume/targets/scst.py#L233

I think technically [ivp6address] is not valid. It's a convention for URIs. I could be wrong though

Revision history for this message
Eric Harney (eharney) wrote :
Changed in cinder:
status: New → Incomplete
assignee: nobody → Eric Harney (eharney)
Revision history for this message
Eric Harney (eharney) wrote :

> I think the correct fix is for cinder to stop assuming the IP is ok to just concatenate with the port, example https://github.com/openstack/cinder/blob/a55a6b5c717c0802288ab802a8fbff287a4b6b66/cinder/volume/targets/scst.py#L233

Correct, but this is only for the scst target, and isn't what failed here.

> I think technically [ivp6address] is not valid. It's a convention for URIs. I could be wrong though

I agree.

Revision history for this message
Ben Nemec (bnemec) wrote :

I'll doublecheck, but I believe that will fix the problem too. It leaves us with the mismatch of having to put a bracketed URI in an option named _ip_address, but that's pretty far down my list of things to worry about. :-)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/472455
Reason: Cinder changed the type of the opt back so this is no longer necessary.

Revision history for this message
Ben Nemec (bnemec) wrote :

Confirmed that this fixes the tripleo use case.

Changed in tripleo:
status: In Progress → Fix Released
Sean Dague (sdague)
Changed in nova:
status: New → Invalid
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote : Bug Assignee Expired

Unassigning due to no activity for > 6 months.

Changed in cinder:
assignee: Eric Harney (eharney) → nobody
Changed in oslo.config:
status: New → Incomplete
status: Incomplete → Invalid
Changed in cinder:
assignee: nobody → Gorka Eguileor (gorka)
status: Incomplete → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/677524
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fddb9b7962ab2d4f3b6f57faef1c090b097b60c6
Submitter: Zuul
Branch: master

commit fddb9b7962ab2d4f3b6f57faef1c090b097b60c6
Author: Gorka Eguileor <email address hidden>
Date: Tue Aug 20 19:36:09 2019 +0200

    Fix LVM IPv6 target portals

    The LVM driver is reporting incorrect target_portal and target_portals
    entries for IPv6 with all the targets, as it is not enclosing the
    address in square brackets making it impossible for iscsiadm in OS-Brick
    to tell the port appart from the IP address.

    This patch makes sure that IPv6 addresses in target_portal and
    target_portals are properly formatted in the [IPv6_address]:port when
    the IPv6_address does not already include the square brackets.

    This fix affects all the targets: IET, LIO, TGT, CXT, and SCST.

    Closes-Bug: #1696866
    Change-Id: Ie7053c813af8c99b5545222d4724b71061348a09

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/678601

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/stein)

Reviewed: https://review.opendev.org/678601
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=88a87b2d4892fd8bdd3b80d1385775c5b044c368
Submitter: Zuul
Branch: stable/stein

commit 88a87b2d4892fd8bdd3b80d1385775c5b044c368
Author: Gorka Eguileor <email address hidden>
Date: Tue Aug 20 19:36:09 2019 +0200

    Fix LVM IPv6 target portals

    The LVM driver is reporting incorrect target_portal and target_portals
    entries for IPv6 with all the targets, as it is not enclosing the
    address in square brackets making it impossible for iscsiadm in OS-Brick
    to tell the port appart from the IP address.

    This patch makes sure that IPv6 addresses in target_portal and
    target_portals are properly formatted in the [IPv6_address]:port when
    the IPv6_address does not already include the square brackets.

    This fix affects all the targets: IET, LIO, TGT, CXT, and SCST.

    Closes-Bug: #1696866
    Change-Id: Ie7053c813af8c99b5545222d4724b71061348a09
    (cherry picked from commit fddb9b7962ab2d4f3b6f57faef1c090b097b60c6)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 15.0.0.0rc1

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 14.0.2

This issue was fixed in the openstack/cinder 14.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.