Poppy (ftp upload server) dies without explanation

[Celso Providelo]

Poppy has suddenly died more frequently that we would like and there is currently no way to find out why, since its log output is not stored. The only thing we can tell is that the claimed PID was not present anymore.

For improving the response on failures we could set up nagios to probe the pid file (/var/run/poppy/poppy.pid) every 5 minutes and change the init.d script to actually store the information logged somewhere for future inspection.

On the code front, we may need to be more verbose about upload events an their durations and incoming IP (but AFAICS this is already available).

The last occurrence of this problem was reported in https://answers.edge.launchpad.net/launchpad/+question/57903 (and I've restarted poppy manually).

[Celso Providelo (cprov)]

We can easily store the log by appending '-v --log-file=<FILE>' command line used in /etc/init.d/poppy on PPA and ubuntu machines.

Bonus if we can setup logrotate for this file (PPA already has it set for /srv/production-logs, but unfortunately lp_upload can't write on it)

The log is very poor, it only logs ftp start/end sessions with the corresponding incoming IP address. I'm not entirely sure random events will get logged, but we have to try.

[Tom Haddon (mthaddon)]

I've created a /srv/launchpad.net/production-logs/poppy folder owned by lp_upload. I've made a copy of /etc/init.d/poppy and applied some changes to it:

https://pastebin.canonical.com/14780/

But starting poppy up with this version of the initscript still sees nothing being written to the indicated logfile.

[Tom Haddon (mthaddon)]

Working version that includes writing to /srv/launchpad.net/production-logs/poppy/poppy.log now in place.

[Tom Haddon (mthaddon)]

And we now also have logrotation in place (as the lp_upload user) for the poppy logs.

[Ivan Zorin (iaz)]

Looks like, that I've clashed with this bug - that's i get from dput, when i'm trying to upload package in my ppa:
...
Uploading to ppa (via ftp to ppa.launchpad.net):
Connection failed, aborting. Check your network (111, 'Connection refused')
>

[Celso Providelo (cprov)]

Ivan,

You are right, there was a outage between 3:40 and 5:20 (UTC), we are working on it.

Sorry for the inconvenience, it should be fine now.

[Ivan Zorin (iaz)]

Ups, sorry - I've just tryed to re-upload - everything went ok.