puppet broken after libapache2-mod-passenger upgrade

[trent--]

On Ubuntu 12.04, after upgrading libapache2-mod-passenger from 2.2.11debian-2 to 2.2.11debian-2+deb6u1ubuntu12.04.1, puppet fails with :

Warning: Error 500 on SERVER: {"issue_kind":"RUNTIME_ERROR","message":"Server Error: undefined method `upcase' for nil:NilClass","stacktrace":["/usr/lib/ruby/vendor_ruby/puppet/network/http/route.rb:80:in `process'","/usr/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:63:in `process'","/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'","/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'","/usr/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:61:in `process'","/usr/lib/ruby/vendor_ruby/puppet/network/http/rack.rb:21:in `call'","/usr/lib/ruby/1.8/phusion_passenger/rack/request_handler.rb:92:in `process_request'","/usr/lib/ruby/1.8/phusion_passenger/abstract_request_handler.rb:207:in `main_loop'","/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:118:in `run'","/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:65:in `spawn_application'","/usr/lib/ruby/1.8/phusion_passenger/utils.rb:184:in `safe_fork'","/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:58:in `spawn_application'","/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:41:in `spawn_application'","/usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb:159:in `spawn_application'","/usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb:287:in `handle_spawn_application'","/usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb:352:in `__send__'","/usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb:352:in `main_loop'","/usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb:196:in `start_synchronously'","/usr/lib/phusion_passenger/passenger-spawn-server:61"]}

This seems to be https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812103

Quoting from that bug :
« It is a regression, there's no way for applications using
mod_passenger to work after the latest update. Not only did the update
switch to a native package and drop some documentation, but it broke
the module.
Granted, the package is safer now that it doesn't work. »

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Pitts (bpitts)]

I also believe that this update has completely broken mod_passenger.

This affected us and stopped our puppetmasters from working. All of the stuff that should be in the rack environment like REMOTE_ADDR went missing, which led to fun-to-decipher error messages like "Could not resolve : cannot interpret as address:" because request.ip was nil. I built the previous package version and downgraded to it; that got our puppetmasters working again.

[Simon Lundström (simmel)]

I agree with Brian; it looks like almost all environment variables are missing. The only ones I, think I, get from Apache is:

    "HTTPS"=>"on",
    "CONTENT_LENGTH"=>nil,

then I get some variables from libapache2-mod-shib2 which is an authentication module:

    "Shib-Session-Index"=>"deadbeef",
    "Shib-Authentication-Instant"=>"2016-05-17T07:04:18.013Z",
    "Shib-Application-ID"=>"default",
    "Shib-Session-ID"=>"_deadbeef",
    "Shib-AuthnContext-Class"=>"urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport",

[Junien F (axino)]

Hi,

So what is the path forward here ?

[Junkern (ulf-bjork)]

I downgraded to previous version of libapache2-mod-passenger, seems to get puppetmaster up and running.

apt-get install libapache2-mod-passenger=2.2.11debian-2

[Trent Lloyd (lathiat)]

Discussion about this in the upstream bug tracker, though no real progress at this stage it's tagged for investigation for the next minor release:
https://tickets.puppetlabs.com/browse/PUP-6411
https://tickets.puppetlabs.com/browse/PUP-6458
https://tickets.puppetlabs.com/browse/PUP-6424

Discussion in the Debian bug implies that the passenger patch was bad, and filters too much. Have not personally evaluated if this is true and seems no one has chimed in on that in the Debian bug as yet - though there is a random patch.

[Trent Lloyd (lathiat)]

I have confirmed that this patch seems to *completely* break passenger (v2) in precise, no variables are passed through at all.

This works as expected on xenial (Passenger 5) with the latest release including the same fix, so I am fairly sure something went wrong in backporting the patch which appears to have been done as part of squeeze-lts. Squeeze LTS concluded in February 2016 so a patch from Debian seems unlikely.

Test case for the issue:
root@precise:~# apt-get install apache2 libapache2-mod-passenger

root@preicse:~# a2enmod passenger

root@precise:~# cat /etc/apache2/sites-enabled/000-default
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 DocumentRoot /var/www/public
 ErrorLog ${APACHE_LOG_DIR}/error.log
 CustomLog ${APACHE_LOG_DIR}/access.log combined
        PassengerEnabled on
</VirtualHost>

root@precise:~# cat /var/www/config.ru
# my_rack_app.rb
require 'rack'
require 'pp'
app = proc do |env|
  [ 200, {'Content-Type' => 'text/plain'}, [PP.pp(Rack::Request.new(env), "")] ]
end
run app

(Old Version)
apt-get install libapache2-mod-passenger=2.2.11debian-2
Result:
#<Rack::Request:0x7f1ab158c2a8
 @env=
  {"rack.run_once"=>false,
   "_"=>"_",
   "HTTP_ACCEPT_ENCODING"=>"gzip, deflate, sdch",
   "PATH_INFO"=>"/",
   "HTTP_CACHE_CONTROL"=>"max-age=0",
   "SCRIPT_NAME"=>"",
   "DOCUMENT_ROOT"=>"/var/www/public",
   "rack.version"=>[1, 0],
   "SERVER_SOFTWARE"=>"Apache/2.2.22 (Ubuntu)",
   "rack.multiprocess"=>true,
   "HTTP_ACCEPT_LANGUAGE"=>"en-US,en;q=0.8",
   "SERVER_PROTOCOL"=>"HTTP/1.1",
   "HTTP_ACCEPT"=>
    "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
   "HTTP_UPGRADE_INSECURE_REQUESTS"=>"1",
   "QUERY_STRING"=>"asff",
   "SERVER_ADMIN"=>"webmaster@localhost",
   "rack.url_scheme"=>"http",
   "rack.input"=>
    #<PhusionPassenger::Utils::RewindableInput:0x7f1ab158cc80
     @io=#<UNIXSocket:0x7f1ab1594548>,
     @rewindable_io=nil,
     @unlinked=false>,
   "REQUEST_URI"=>"/?asff",
   "rack.multithread"=>false,
   "REQUEST_METHOD"=>"GET",
   "REMOTE_ADDR"=>"10.48.134.22",
   "REMOTE_PORT"=>"36638",
   "SERVER_ADDR"=>"10.48.134.78",
   "SERVER_NAME"=>"10.48.134.78",
   "rack.errors"=>#<IO:0x7f1ab15f7aa8>,
   "HTTP_USER_AGENT"=>
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36",
   "HTTP_CONNECTION"=>"keep-alive",
   "HTTP_HOST"=>"10.48.134.78",
   "SERVER_PORT"=>"80"}>

(New Version)
apt-get upgrade
# upgrade only 'reloads' apache and does not restart it which doesn't load the new module, so restart it
/etc/init.d/apache2 restart

Result:
#<Rack::Request:0x7fea869062d0
 @env=
  {"_"=>"_",
   "rack.url_scheme"=>"http",
   "rack.version"=>[1, 0],
   "rack.multithread"=>false,
   "rack.run_once"=>false,
   "rack.input"=>
    #<PhusionPassenger::Utils::RewindableInput:0x7fea86906438
     @io=#<UNIXSocket:0x7fea86906848>,
     @rewindable_io=nil,
     @unlinked=false>,
   "rack.multiprocess"=>true,
   "rack.errors"=>#<IO:0x7fea8838bab0>}>

[Trent Lloyd (lathiat)]

I tracked down the source of the issue

The current patch modifies the addHeader() function itself to perform the check, this is invalid because this function is used internally to setup many headers from the environment such as the standard CGI HTTP_HOST, REQUEST_URI, etc.

The correct patch should only abort adding headers from the HTTP request.

The upstream patch/source for Passenger 5 was quite different to v2 here, however the upstream patch for Passenger 4 (https://github.com/phusion/passenger/commit/c04590871ca0878d4d3ac1220c5a554b049056b4) was very similar and I have backported this fix to precise in the attached debdiff. I have not backported the nginx part, it was not done originally.

Patch Testing:
 ** No Patch **
lathiat@ubuntu:~/src/lp1575220$ curl -s -H "X-Test-Dash-Header: Yes" -H "X_TEST_UNDERSCORE_HEADER: Yes" http://10.48.134.78/|grep -i test
   "HTTP_X_TEST_UNDERSCORE_HEADER"=>"Yes",
   "HTTP_X_TEST_DASH_HEADER"=>"Yes",

 ** Broken Patch **
lathiat@ubuntu:~/src/lp1575220$ curl -s -H "X-Test-Dash-Header: Yes" -H "X_Test_Underscore_header: Yes" http://10.48.134.78/|grep -i test

 ** New Proposed Patch **
lathiat@ubuntu:~/src/lp1575220$ curl -s -H "X-Test-Dash-Header: Yes" -H "X_Test_Underscore_header: Yes" http://10.48.134.78/|grep -i test
   "HTTP_X_TEST_DASH_HEADER"=>"Yes",

[Trent Lloyd (lathiat)]

Uploading a copy of the same debdiff, but against the original version (2.2.11debian-2). It's more clear exactly what is now changed and forms the fix, for review purposes.

[Steve Beattie (sbeattie)]

Trent, thanks for digging up a solution for this. I'll sponsor your upload to security today.

[Launchpad Janitor (janitor)]

This bug was fixed in the package passenger - 2.2.11debian-2+deb6u1ubuntu12.04.2

---------------
passenger (2.2.11debian-2+deb6u1ubuntu12.04.2) precise-security; urgency=medium

  * REGRESSION UPDATE: Fix for regression introduced in previous
    CVE-2015-7519 fix. All HTTP headers were dropped from the
    request which broke all applications. Backport the upstream
    fix from commit c04590871ca0878d4d3ac1220c5a554b049056b4 for
    Apache2 only (LP: #1575220)

 -- Trent Lloyd <email address hidden> Tue, 05 Jul 2016 00:42:47 +0800