Librarian garbage collector does not cope with symlinks

Bug #562828 reported by Stuart Bishop
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Gary Poster

Bug Description

The Librarian garbage collector traverses the file store using os.walk(). We have recently split this over multiple disks using symlinks. Unfortunately, os.walk() does not follow symlinks. This generates a lot of ERROR messages, as the garbage collector does not find files on disk that exist in the database.

A followlinks argument was added to os.walk() in Python 2.6. If we can switch to 2.6 quickly, using this option is the best solution.

Otherwise, we need an os.walk() replacement that follows symlinks.

Related branches

Tom Haddon (mthaddon)
Changed in launchpad-foundations:
importance: Undecided → High
Revision history for this message
Steve McInerney (spm) wrote :

<stub> Can you run it again? I need to know the first id reported
<stub> If it is 67108865 then it is safe to leave it enabled

2010-04-14 08:24:11 ERROR LibraryFileContent 67108865 exists in the database but was not found on disk.

1st entry off the manual re-run

Revision history for this message
Stuart Bishop (stub) wrote :

So we can leave the script enabled and it will keep doing its job. If we do this, we should filter these errors from the output as we can expect them to grow by tens of thousands each day.

Revision history for this message
Max Bowsher (maxb) wrote :

Would it be practical to (temporarily) use bindmounts instead of symlinks?

Gary Poster (gary)
Changed in launchpad-foundations:
status: New → Triaged
Revision history for this message
Gary Poster (gary) wrote :

Because we will need to carefully dance to both Lucid and Py 2.6 in the data center at the same time, we will not be able to fully switch to Python 2.6 until at least 1.5 cycles from now. We'll want this addressed sooner.

The LOSAs and James regarded hacking os.walk into our scripts as a superior hack than bindmounts--the os.walk copy-n-paste-n-hack should be much easier for us to implement than bindmounts on their side.

I do not remember clearly why filtering the errors was rejected as a work-around, but it was, for the time of 1.5 cycles.

Therefore, we need to copy the 2.6 os.walk code and hack it to work in 2.5 if necessary.

I'm pushing a branch with this change and a test.

Changed in launchpad-foundations:
assignee: nobody → Gary Poster (gary)
status: Triaged → In Progress
milestone: none → 10.04
Revision history for this message
Ursula Junque (ursinha) wrote : Bug fixed by a commit
Changed in launchpad-foundations:
status: In Progress → Fix Committed
tags: added: qa-needstesting
Gary Poster (gary)
tags: added: qa-untestable
removed: qa-needstesting
Gary Poster (gary)
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers