[Fuel Gerrit] Wierd behavior of mos-infra/project-config repo and strange error in logs on Fuel Gerrit

[Andrey Nikitin]

I've faced the following problem on Fuel Gerrit:

heos@worksys:/tmp$ git clone ssh://<email address hidden>:29418/mos-infra/project-config project-config-test
Cloning into 'project-config-test'...
fatal: internal server error
remote: internal server error
fatal: early EOF
fatal: index-pack failed

I see the error in the logs of Fuel Gerrit at the same time:

[2016-03-02 15:18:16,358] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user mos-infra-ci account 131) during git-upload-pack '/mos-infra/project-config'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
 at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1275)
 at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:717)
 at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:628)
 at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:57)
 at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
 at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
 at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
 at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
 at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:364)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing commit 526b811e959a2d12a98e9b24bccb1dd3df9d52f4
 at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:149)
 at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:883)
 at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:145)
 at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:148)
 at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:421)
 at org.eclipse.jgit.revwalk.ObjectWalk.next(ObjectWalk.java:261)
 at org.eclipse.jgit.internal.storage.pack.PackWriterBitmapWalker.findObjects(PackWriterBitmapWalker.java:103)
 at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPackUsingBitmaps(PackWriter.java:1849)
 at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1667)
 at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:799)
 at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1384)
 at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1269)
 ... 15 more

Looks like 'mos-infra/project-config' has wrong git tree. I suppose it should be fixed by reindexing of changes on gerrit database and running garbage collector on the repository.

Also the problem affects on load of Fuel Gerrit. As I can see in htop command output one core of CPU is loaded on 100% permanently.

[Andrey Nikitin (heos)]

The problem were fixed by executing the following steps:

1. Running on raw git-repository on the disk the following comand: git gc
2. Running on raw git-repository on the disk the following comand: git fsck --full --progress

Also I ran reindex of Lucena cache on the server after the previous commands and started the gerrit.

All is working fine after those steps. The problem project could be cloned from the gerrit without errors.

[Andrey Nikitin (heos)]

The root cause of the problem is weird commit replicated into repository. That commit has 526b811e959a2d12a98e9b24bccb1dd3df9d52f4 id.