[incident] [devops] openstack/cinder project can't be cloned - 2h

[Dmitry Burmistrov]

https://review.fuel-infra.org/#/admin/projects/openstack/cinder
>500 Internal server error

1467707940: #cinder! + git remote set-url mirantis-cinder ssh://<email address hidden>:29418/openstack/cinder.git
1467707940: #cinder! + git fetch --prune mirantis-cinder --no-tags
1467707941: #cinder! fatal: Project not found: openstack/cinder

Looks like, the following projects should be checked too:

openstack/horizon
openstack/kolla

[Andrey Nikitin (heos)]

I see the following errors in the logs:

[2016-07-05 00:00:01,574] WARN com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project openstack/cinder
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 8cc353957e98681b5de103c05804dbcdc131793d
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2346)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
        at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:122)
        at com.google.gerrit.server.project.ProjectControl$GenericFactory.controlFor(ProjectControl.java:81)
        at com.google.gerrit.server.project.ChangeControl$GenericFactory.controlFor(ChangeControl.java:78)
        at com.google.gerrit.server.query.change.IsVisibleToPredicate.match(IsVisibleToPredicate.java:62)
        at com.google.gerrit.server.query.change.IsVisibleToPredicate.match(IsVisibleToPredicate.java:27)
        at com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:76)
        at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:121)
        at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:100)
        at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:267)
        at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:228)
        at com.google.gerrit.server.query.change.QueryProcessor.query(QueryProcessor.java:309)
        at com.google.gerrit.sshd.commands.Query.run(Query.java:95)
        at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:364)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.Threa...

[Andrey Nikitin (heos)]

Also I see the following errors there:

[2016-07-04 19:50:04,094] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/kolla
[2016-07-04 19:50:04,094] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/horizon
[2016-07-04 19:50:04,131] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/cinder
[2016-07-04 20:06:03,050] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/cinder
[2016-07-04 20:06:03,056] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/horizon
[2016-07-04 20:06:03,236] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/kolla
[2016-07-04 20:17:03,103] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/horizon
[2016-07-04 20:17:03,150] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/kolla
[2016-07-04 20:17:03,886] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Error trying to advertise history on openstack/cinder

[Andrey Nikitin (heos)]

The problem is fixed by runnig the following commands on each repository:

 -- git gc --aggressive
 -- git fsck --full --progress.

Also Lucene reindex has been started after this operations.

All is working fine right now.