Frequent time-outs between Launchpad and the Salesforce proxy

[Brad Crittenden]

About a month ago we began seeing frequent time-outs when Launchpad attempted to make an XMLRPC connection to the Salesforce proxy running on niobium. An example failure is at OOPS-1557M2198 .

This problem is affecting our paying customers at the point they have just given us money and thus sours the experience from the outset. It also affects our OEM internal customers such as Joey who cannot complete his tasks efficiently.

We understand this proxy is slated to be replaced but we need to find the cause of the time-outs and create a fix in the interim.

Please consider the filing of this bug as an escalation of the problem. It is occurring with increasing frequency and is having a larger impact than when first reported.

[Curtis Hovey (sinzui)]

I think the process gets hung after an applying a voucher for the first time:
1. I applyed a voucher for project A and the response was quick.
2. I saw gateway timeout trying to apply the voucher for project B a minute later
3. After 15 minutes of tries I gave up.
4. I tried to apply the voucher 3 hours later ang got a gateway timeout.
5. I asked an admin to restart the sfi-production process
6. The voucher was applied in a second.

[James Jesudason (jamesj)]

Following a request from Brad, I've added a some code to log the start of the LaunchpadSubscriptionVoucherService.getAllVouchers call and the start and end of the VoucherService.getVouchers method. This should get deployed to staging on niobium by the end of the week.

Once that has been done, perhaps you can run a couple of tests on staging to see if we can replicate the issues there.

[Brad Crittenden (bac)]

I have confirmed that testing on staging (via https://staging.launchpad.net/~bac/+vouchers) hits the staging proxy on niobium and logs messages in

/srv/salesforce/staging/staging-logs/sfiproxy.log

[James Jesudason (jamesj)]

The extra logging code has been merged into production on Launchpad and this has now been deployed to niobium.

[Curtis Hovey (sinzui)]

James, Brad.

Tom has made firewall changes based on Brad's and Stefan's conclusion from the logs that the timeouts never made a request to the saleforce proxy. I think we cam makr this bug fix released once we confirm that the timeout issue is gone.

[Brad Crittenden (bac)]

I have asked Joey to let me know if he sees more timeouts.

Please leave the additional logging in place on production as it provides very useful information.

[Brad Crittenden (bac)]

The firewall change was filed as: [rt.admin.canonical.com #39129] Firewall mods for LP

[Curtis Hovey (sinzui)]

Fixed released in launchpad-project 10.04.