sync-index stops at the first failure when a charm is indexed.

[Abel Deuring]

Below is a modified variant of TestCharmSource.test_sync_index_script() that shows the error. (It contains changes from lp:~adeuring/charmworld/1206659-simpler-es-mapping which are not yet merged. Most of the changes from this branch are not relevant for this problem, except of the fact that earlier variants of the test in this branch failed spuriously.)

    def test_sync_index_script(self):
        factory.makeCharm(self.db, owner='a')[0]
        factory.makeCharm(self.db, owner='c')[0]
        # Create a charm which is index-incompatible with the previous one.
        factory.makeCharm(self.db, owner='b',
                          description={'text': 'whatever'})[0]
        handler = self.get_handler('sync-index')
        with patch('charmworld.models.configure_logging'):
            with patch('charmworld.models.get_ini', lambda: INI):
                with force_index_client(self.use_index_client()):
                    sync_index()
            messages = [r.getMessage() for r in handler.buffer]
            self.assertIn('Unable to index charm.', messages)
        self.assertEqual(2, len(self.index_client.api_search()))

The setup: Three charms, those owned by "a" and "c" should be indexed; indexing b's charm will fail. But only c's charm is indexed.

The cause:

class CharmSource:
    def sync_index(self):
        for charm in _find_charms(self.collection):
            yield charm['_id']
            self.index_client.index_charm(charm)

def sync_index():
    """"Entry point for sync-index script."""
    configure_logging()
    logger = logging.getLogger('sync-index')
    charm_source = CharmSource.from_settings(get_ini())
    charm_ids = charm_source.sync_index()
    while True:
        try:
            charm_id = next(charm_ids)
        except StopIteration:
            break
        except Exception:
            logger.exception('Unable to index charm.')
        else:
            logger.info('Syncing %s to Elasticsearch' % charm_id)

_find_charms() returns all charms in descending order of (owner, series, charm_name), so charm "c" is indexed first; when index_client.index_charm(charm) fails for charm "b", CharmSource.sync_index() does not catch the error. It is instead passed to the function sync_index(), where the error is logged; the next attempt to call next(charm_ids) raises StopIteration.