OSError at /oops.py/ when using lp-oops

Bug #628510 reported by Michael Barnett on 2010-09-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Diogo Matsubara

Bug Description

When trying to pull up:
https://lp-oops.canonical.com/oops.py/?oopsid=1705A2026
we get an oops that starts with:
OSError at /oops.py/

[Errno 13] Permission denied: '/x/launchpad.net-logs/production/mizuho/2010-09-01'

Checking the apache logs, I see:

[Wed Sep 01 23:18:25 2010] [error] [client 122.63.10.108] Unauthorized access attempt for 'https://lp-oops.canonical.com/oops.py/?oopsid=1705A2026' by 'https://login.ubuntu.com/+id/kPbPBDC' (teams: [])

Related branches

tags: added: canonical-losa-lp
Tim Penhey (thumper) on 2010-09-03
affects: launchpad → oops-tools
Changed in oops-tools:
status: New → Triaged
importance: Undecided → High
Diogo Matsubara (matsubara) wrote :

I investigated this issue with Stefan, we found out a few things but still no
root cause:

- the librarian appinstance is creating those unreadable oops directories.

    <Chex> librarian@mizuho:/srv/launchpadlibrarian.net/production-logs/2010-09-29$ ls -la
    <Chex> total 216
    <Chex> drwx--S--- 2 librarian librarian 4096 Sep 29 09:14 .
    <Chex> drwxr-sr-x 6 librarian librarian 196608 Sep 30 20:37 ..
    <Chex> -rw------- 1 librarian librarian 4348 Sep 29 09:02 32568.LIBRARIANA1
    <Chex> -rw------- 1 librarian librarian 4348 Sep 29 09:14 33248.LIBRARIANA2

- the user running the librarian appinstance seems to have the correct
  permission to create new files.

    <Chex> librarian@mizuho:/tmp$ umask
    <Chex> 0022
    <Chex> so that looks OK

- there's no cronjob changing the permissions afeterwards.

    <Chex> I wonder if there is a cron changing the perms.. checking
    <Chex> no, there is a cron that deletes old librarian.log.* but thats it

- content of one of the oopses: https://pastebin.canonical.com/37969/

- If we find a reliable way of triggereing the oops above we could add some
  cowboy code to the librarian to log the permission at the time an oops is
  being generated. that way we would know if it's the appserver generating the
  content with the wrong permission or if it's some process later on changing
  it.

- Any other ideas to help find out the cause?

the librarian possibly is changing its own umask. if so (and its
likely) the oops writing code may want to override the perms.

-Rob

Diogo Matsubara (matsubara) wrote :

Robert, is there a way to verify if the librarian is changin its own umask with the process already running?

Robert Collins (lifeless) wrote :

Dunno :) I'd look at the twisted daemonisation code though. I suspect
it defaults to 077 (which actually we like for the librarian).

I suggest that the oops logging code, if it needs some specific
permissions, should set them.

e.g.
os.fchmod(outfile.fd, 0440)

-Rob

Diogo Matsubara (matsubara) wrote :

Today this happened again. I cowboyed the following to the production
oops-tools instance: https://pastebin.canonical.com/38351/

This means there won't be a fallback to look for the OOPS in the filesystem.
The view will try to find the OOPS in the database, and if it's not there,
it'll show a message saying the the OOPS can't be found.

The cowboyed code can be removed once the permissions in the source directories are fixed and errorlog.py is patched to set the permission in the error_dir and oops file to world readable.

Gary Poster (gary) wrote :

As Robert described, the fix for this is to change ``os.makedirs(result)`` in lp/services/log/uniquefileallocator.py to specify a more friendly permission value, and then do the same for the OOPS file generation in errorlog.py. We need both.

affects: oops-tools → launchpad-foundations
Changed in launchpad-foundations:
assignee: nobody → Diogo Matsubara (matsubara)
status: Triaged → In Progress
milestone: none → 10.10
Changed in launchpad-foundations:
milestone: 10.10 → 10.11
tags: added: qa-needstesting
Changed in launchpad-foundations:
status: In Progress → Fix Committed
tags: added: qa-untestable
removed: qa-needstesting
Curtis Hovey (sinzui) on 2010-11-17
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers