PBR doesn't seem to copy migrate.cfg into proper locations

[Lance Bragstad]

I noticed this with keystone stable/pike. These were the steps I took

$ wget https://github.com/openstack/keystone/archive/12.0.0.tar.gz
$ wget https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt -P /tmp
$ mkdir keystone-pike
$ tar xvfz 12.0.0.tar.gz -C keystone-pike
$ cd keystone-pike/keystone-12.0.0
$ virtualenv keystone-pike
$ pip install -r requirements.txt -c /tmp/upper-constraints.txt
$ PBR_VERSION=12.0.0 python setup.py install

After that I see this error when I run `keystone-manage db_version`:

$ keystone-manage db_version
2018-01-11 22:17:46.285 8616 WARNING oslo_db.sqlalchemy.engines [-] URL mysql://keystone:***@localhost/keystone does not contain a '+drivername' portion, and will make use of a default driver. A full dbname+drivername:// protocol is rec
ommended. For MySQL, it is strongly recommended that mysql+pymysql:// be specified for maximum service compatibility
2018-01-11 22:17:46.300 8616 DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTI
ON _check_effective_sql_mode /home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:285
2018-01-11 22:17:46.302 8616 DEBUG migrate.versioning.repository [-] Loading repository /home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/common/sql/migrate_repo... __init__ /home/lbra
gstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/migrate/versioning/repository.py:76
2018-01-11 22:17:46.302 8616 CRITICAL keystone [-] Unhandled error: InvalidRepositoryError: /home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/common/sql/migrate_repo
2018-01-11 22:17:46.302 8616 ERROR keystone Traceback (most recent call last):
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/bin/keystone-manage", line 10, in <module>
2018-01-11 22:17:46.302 8616 ERROR keystone sys.exit(main())
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/cmd/manage.py", line 45, in main
2018-01-11 22:17:46.302 8616 ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1337, in main
2018-01-11 22:17:46.302 8616 ERROR keystone CONF.command.cmd_class.main()
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/cmd/cli.py", line 553, in main
2018-01-11 22:17:46.302 8616 ERROR keystone print(upgrades.get_db_version())
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/common/sql/upgrades.py", line 258, in get_db_version
2018-01-11 22:17:46.302 8616 ERROR keystone session.get_bind(), find_repo(repo), get_init_version())
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/common/sql/upgrades.py", line 199, in get_init_version
2018-01-11 22:17:46.302 8616 ERROR keystone repo = migrate.versioning.repository.Repository(abs_path)
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/migrate/versioning/repository.py", line 77, in __init__
2018-01-11 22:17:46.302 8616 ERROR keystone self.verify(path)
2018-01-11 22:17:46.302 8616 ERROR keystone File "/home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/migrate/versioning/repository.py", line 98, in verify
2018-01-11 22:17:46.302 8616 ERROR keystone raise exceptions.InvalidRepositoryError(path)
2018-01-11 22:17:46.302 8616 ERROR keystone InvalidRepositoryError: /home/lbragstad/keystone-pike/keystone-12.0.0/keystone-pike/local/lib/python2.7/site-packages/keystone/common/sql/migrate_repo
2018-01-11 22:17:46.302 8616 ERROR keystone

The migrate.cfg files typically found in keystone/common/sql/migrate_repo/ are not installed, but they are in the source. Providing a MANIFEST.ini or copying the migrate.cfg files manually work as a work around [0].

[0] http://paste.openstack.org/show/643175/

[Doug Hellmann (doug-hellmann)]

Do they need to be listed in the "data_files" section in setup.cfg?

[Lance Bragstad (lbragstad)]

Oh - that could be the case. The part I'm curious about is if anything changed in PBR that would have caused this recently. This was apparently just noticed on the stable/pike branch, but it was working when pike was released. I'm not sure how that happened though.

[Kamil Madac (kamil-madac)]

I just found out that when I clone the keystone repo, checkout stable/pike and run 'python setup.py install', then .cfg files are copied.
But if I use released package from github.com as it is written in bug desription, .cfg file are no copied.

There were do difference between setup.cfg in repo and in released package.

[Kamil Madac (kamil-madac)]

I found same behavior on glance. When cloned from git, .cfg files are installed, otherwise not.

[Ben Nemec (bnemec)]

I did a bit of poking at this, and I think the problem is that the github release tarballs don't match what we publish on releases.openstack.org. The file sizes are noticeably different, and this process works as expected if you use the official tarball instead of the github one.

My best guess would be that however github creates release tarballs from tags, it doesn't do _something_ that is required for OpenStack projects. I couldn't say what off hand though.

Since I doubt we want to duplicate our release tooling to handle github's process, maybe there is some way to have github link to the OpenStack tarballs instead of generating them itself? I'm going to send something to the dev list since I would guess that a fair number of people are trying to download OpenStack releases from github (even if we'd rather they didn't).

[Monty Taylor (mordred)]

It is required that openstack tarballs be generated in a git context using python setup.py sdist.

github produces its tarballs with git export- which is not supported.

This is not a bug - it is a fundamental design tenet of pbr.

Also, I highly recommend not wgetting the upper-constraints file from github. If you want a copy, the location is:

https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt
?h=stable/rocky

[Ben Nemec (bnemec)]

We can't fix this on the pbr side since github isn't using pbr to generate the tarball.