Cinder LVM driver and ipv6 broken

Bug #1696866 reported by Ben Nemec on 2017-06-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Gorka Eguileor
OpenStack Compute (nova)
Undecided
Unassigned
oslo.config
Undecided
Unassigned
tripleo
Medium
Ben Nemec

Bug Description

The following traceback is seen in the cinder volume logs if you deploy with the default LVM driver and ipv6:

2017-06-08 21:50:40.665 38554 CRITICAL cinder [req-a7aff57e-2042-48f2-b1eb-962f79ccda01 - - - - -] ConfigFileValueError: Value for option iscsi_ip_address is not valid: [fd00:fd00:fd00:3000::12] is not a valid host address
2017-06-08 21:50:40.665 38554 ERROR cinder Traceback (most recent call last):
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>
2017-06-08 21:50:40.665 38554 ERROR cinder sys.exit(main())
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 120, in main
2017-06-08 21:50:40.665 38554 ERROR cinder launcher.wait()
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 581, in wait
2017-06-08 21:50:40.665 38554 ERROR cinder self.conf.log_opt_values(LOG, logging.DEBUG)
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2799, in log_opt_values
2017-06-08 21:50:40.665 38554 ERROR cinder _sanitize(opt, getattr(group_attr, opt_name)))
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3277, in __getattr__
2017-06-08 21:50:40.665 38554 ERROR cinder return self._conf._get(name, self._group)
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2841, in _get
2017-06-08 21:50:40.665 38554 ERROR cinder value = self._do_get(name, group, namespace)
2017-06-08 21:50:40.665 38554 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2884, in _do_get
2017-06-08 21:50:40.665 38554 ERROR cinder % (opt.name, str(ve)))
2017-06-08 21:50:40.665 38554 ERROR cinder ConfigFileValueError: Value for option iscsi_ip_address is not valid: [fd00:fd00:fd00:3000::12] is not a valid host address

The problem appears to be that we bracket the value. If I remove the brackets then cinder volume works correctly.

Fix proposed to branch: master
Review: https://review.openstack.org/472455

Changed in tripleo:
status: Triaged → In Progress

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/472455
Reason: I don't think this actually works. We need the brackets. Will have to fix oslo/netaddr/socket to get this working.

Ben Nemec (bnemec) wrote :

Okay, if I un-bracket the IP then cinder-volume works, but the volume can't be attached to the nova instance. I've attached the nova-compute log from the attempted boot.

It looks like there might be a bug in os-brick? I'm going to keep investigating, but I'll add it to the bug report.

Ben Nemec (bnemec) wrote :

I did some more digging yesterday but concluded that this is beyond my ability to debug in a reasonable timeframe. I no longer think os-brick is the problem though - it looks like the data passed in to os-brick is already bad, so I think it's something in Cinder or Nova that is not correctly handling the ipv6 iscsi address. I'm going to add both projects to this bug and let them determine who needs to fix what.

To recap, if we bracket iscsi_ip_address then oslo.config fails to validate the value, if we remove the brackets then cinder-volume starts correctly and is able to create the volume, but it can't be attached to the Nova instance. This appears to be because the iscsi port is being appended to the ipv6 address without first bracketing the address to separate the address and port portions (see the attached nova-compute logs).

Here are the RDO package versions I was testing with:

$ rpm -qa | egrep "nova|cinder"
puppet-cinder-11.1.0-0.20170605202131.257519d.el7.centos.noarch
openstack-nova-conductor-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
python2-cinderclient-2.0.1-0.20170320163530.d0790e3.el7.centos.noarch
puppet-nova-11.1.0-0.20170605232112.27baec7.el7.centos.noarch
python-cinder-11.0.0-0.20170521083557.cedf99f.el7.centos.noarch
openstack-nova-common-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-api-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-placement-api-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-cinder-11.0.0-0.20170521083557.cedf99f.el7.centos.noarch
python-nova-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-migration-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-scheduler-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-console-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
python2-novaclient-8.0.0-0.20170517113627.e1b9e76.el7.centos.noarch
openstack-nova-compute-16.0.0-0.20170521033533.99bd334.el7.centos.noarch
openstack-nova-novncproxy-16.0.0-0.20170521033533.99bd334.el7.centos.noarch

no longer affects: os-brick
Sean McGinnis (sean-mcginnis) wrote :

Actually sounds like oslo.config needs to be updated to handle IPv6 addressing to me.

Alex Schultz (alex-schultz) wrote :

I think the correct fix is for cinder to stop assuming the IP is ok to just concatenate with the port, example https://github.com/openstack/cinder/blob/a55a6b5c717c0802288ab802a8fbff287a4b6b66/cinder/volume/targets/scst.py#L233

I think technically [ivp6address] is not valid. It's a convention for URIs. I could be wrong though

Eric Harney (eharney) wrote :
Changed in cinder:
status: New → Incomplete
assignee: nobody → Eric Harney (eharney)
Eric Harney (eharney) wrote :

> I think the correct fix is for cinder to stop assuming the IP is ok to just concatenate with the port, example https://github.com/openstack/cinder/blob/a55a6b5c717c0802288ab802a8fbff287a4b6b66/cinder/volume/targets/scst.py#L233

Correct, but this is only for the scst target, and isn't what failed here.

> I think technically [ivp6address] is not valid. It's a convention for URIs. I could be wrong though

I agree.

Ben Nemec (bnemec) wrote :

I'll doublecheck, but I believe that will fix the problem too. It leaves us with the mismatch of having to put a bracketed URI in an option named _ip_address, but that's pretty far down my list of things to worry about. :-)

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/472455
Reason: Cinder changed the type of the opt back so this is no longer necessary.

Ben Nemec (bnemec) wrote :

Confirmed that this fixes the tripleo use case.

Changed in tripleo:
status: In Progress → Fix Released
Sean Dague (sdague) on 2017-06-28
Changed in nova:
status: New → Invalid

Unassigning due to no activity for > 6 months.

Changed in cinder:
assignee: Eric Harney (eharney) → nobody
Changed in oslo.config:
status: New → Incomplete
status: Incomplete → Invalid
Changed in cinder:
assignee: nobody → Gorka Eguileor (gorka)
status: Incomplete → In Progress

Reviewed: https://review.opendev.org/677524
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fddb9b7962ab2d4f3b6f57faef1c090b097b60c6
Submitter: Zuul
Branch: master

commit fddb9b7962ab2d4f3b6f57faef1c090b097b60c6
Author: Gorka Eguileor <email address hidden>
Date: Tue Aug 20 19:36:09 2019 +0200

    Fix LVM IPv6 target portals

    The LVM driver is reporting incorrect target_portal and target_portals
    entries for IPv6 with all the targets, as it is not enclosing the
    address in square brackets making it impossible for iscsiadm in OS-Brick
    to tell the port appart from the IP address.

    This patch makes sure that IPv6 addresses in target_portal and
    target_portals are properly formatted in the [IPv6_address]:port when
    the IPv6_address does not already include the square brackets.

    This fix affects all the targets: IET, LIO, TGT, CXT, and SCST.

    Closes-Bug: #1696866
    Change-Id: Ie7053c813af8c99b5545222d4724b71061348a09

Changed in cinder:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/678601
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=88a87b2d4892fd8bdd3b80d1385775c5b044c368
Submitter: Zuul
Branch: stable/stein

commit 88a87b2d4892fd8bdd3b80d1385775c5b044c368
Author: Gorka Eguileor <email address hidden>
Date: Tue Aug 20 19:36:09 2019 +0200

    Fix LVM IPv6 target portals

    The LVM driver is reporting incorrect target_portal and target_portals
    entries for IPv6 with all the targets, as it is not enclosing the
    address in square brackets making it impossible for iscsiadm in OS-Brick
    to tell the port appart from the IP address.

    This patch makes sure that IPv6 addresses in target_portal and
    target_portals are properly formatted in the [IPv6_address]:port when
    the IPv6_address does not already include the square brackets.

    This fix affects all the targets: IET, LIO, TGT, CXT, and SCST.

    Closes-Bug: #1696866
    Change-Id: Ie7053c813af8c99b5545222d4724b71061348a09
    (cherry picked from commit fddb9b7962ab2d4f3b6f57faef1c090b097b60c6)

tags: added: in-stable-stein

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

This issue was fixed in the openstack/cinder 14.0.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers