map_instances default batch size is too small.

[Liam Young]

Description
===========

map_instances seemingly hung for hours on a cloud with ~190000 instance records. I think the following fixes are valid (in order of preference):

1) nova_manage should examine the amount of instances that need mapping and make an informed choice about batch size if max_count is not set.
2) max_counts default should be raised. It is currently 50 and I cannot imagine what use case 50 is a good default for. For small clouds the max_count is almost irrelevant, for medium/large clouds 50 is far too low.
3) Update max_count description. It currently reads "Maximum number of instances to map" but I think it should also point out that this is the batch size that instances will be processed in.

Steps to reproduce
==================

Fire up a large number of instances on a cloud and run map_instances without max_count set:

nova-manage --config-file /etc/nova/nova.conf cell_v2 map_instances --cell_uuid <uuid>

Expected result
===============

The command should complete in a reasonable time (under an hour)

Actual result
=============

Command runs for over three hours

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   If this is from a distro please provide

# dpkg -l | grep nova
ii nova-api-os-compute 2:16.0.3-0ubuntu1~cloud0 all OpenStack Compute - OpenStack Compute API frontend
ii nova-common 2:16.0.3-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-conductor 2:16.0.3-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-placement-api 2:16.0.3-0ubuntu1~cloud0 all OpenStack Compute - placement API frontend
ii nova-scheduler 2:16.0.3-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:16.0.3-0ubuntu1~cloud0 all OpenStack Compute Python libraries

[Liam Young (gnuoy)]

Actually, it would be useful to be able to say map all the instances using this batch size, eg:

nova-manage --config-file /etc/nova/nova.conf cell_v2 map_instances --cell_uuid <uuid> --max_count all --batch_size 50000

[Matt Riedemann (mriedem)]

This is a one time operation when you're upgrading to a cells v2 deployment, so while it's unfortunate it took this long in a large cloud with several thousand instances, I'd consider this lower priority to resolve with code at this point unless the change is trivial. Therefore option 3, better documentation of the command and it's option itself, is probably what I'd lean toward.

[Matt Riedemann (mriedem)]

Alternatively, a --verbose option could be added to the command so you could at least see it's making progress by printing out each instance that is being mapped while it's running.

[Matt Riedemann (mriedem)]

I'll push up a change for the docs and I've tagged this with low-hanging-fruit so someone that wants to work on the suggestion from comment #1 can do so. It would be similar to how the --until-complete option works with the nova-manage db archive_deleted_rows command works.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/536959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536959
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2890d8db67012d7c33c40a1e36a66766740b8f8f
Submitter: Zuul
Branch: master

commit 2890d8db67012d7c33c40a1e36a66766740b8f8f
Author: Matt Riedemann <email address hidden>
Date: Tue Jan 23 15:30:47 2018 -0500

    Expand on when you might want to set --max-count for map_instances

    The default behavior for the "nova-manage cell_v2 map_instances"
    command is to map all instances in the cell in batches of 50.

    This can be slow when there are several thousand instances in the
    deployment and an operator may want to specify a higher --max-count
    value and run the command until it completes.

    This simply updates the command option description and man page to
    point this out for consideration.

    Change-Id: I59c2ed89fe02212977445f6825c6da8fedbb8ccf
    Related-Bug: #1742649