libwww perl in ubuntu always enforces HTTPS server certificate

[janl]

Given this simple code:

    $ua = LWP::UserAgent->new;
    $ua->agent("netview");
    $ua->protocols_allowed( [ 'https' ] );
    $ua->ssl_opts( verify_hostname => 0 );
    push @{ $ua->requests_redirectable }, 'POST', 'GET';

    my $req = HTTP::Request->new( GET =>
                                  "https://$server/blc/api/routers/type/pe" );

    $req->content_type('application/json');
    $req->authorization_basic($apipw->{APIUSER}, $apipw->{APIPW});

    my $res = $ua->request($req);
    LOGDIE "Error getting PE routers via REST to $server: ".$res->status_line.
      "(".$res->content.")"
        if ! $res->is_success;

I get this message:
Error getting PE routers via REST to blc.serv.as2116.net: 500 Can't connect to blc.serv.as2116.net:443 (certificate verify failed)(Can't connect to blc.serv.as2116.net:443 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/share/perl5/LWP/Protocol/http.pm line 41.

Strace shows that the code is looking for a CA file from the OpenSSL package. blc.serv.as2116.net's sertificate is signed by a uncommon CA so this fails.

BUT it should not be trying to verify this at all due to the verify_hostname setting.

In HTTP::Protocol::https one finds a _extra_sock_opts function that's different than the official LWP release. I replaced it with the this LWP 6.04 code:

sub _extra_sock_opts
{
    my $self = shift;
    my %ssl_opts = %{$self->{ua}{ssl_opts} || {}};
    if (delete $ssl_opts{verify_hostname}) {
 $ssl_opts{SSL_verify_mode} ||= 1;
 $ssl_opts{SSL_verifycn_scheme} = 'www';
    }
    else {
 $ssl_opts{SSL_verify_mode} = 0;
    }
    if ($ssl_opts{SSL_verify_mode}) {
 unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
     eval {
  require Mozilla::CA;
     };
     if ($@) {
  if ($@ =! /^Can't locate Mozilla\/CA\.pm/) {
      $@ = <<'EOT';
Can't verify SSL peers without knowing which Certificate Authorities to trust

This problem can be fixed by either setting the PERL_LWP_SSL_CA_FILE
envirionment variable or by installing the Mozilla::CA module.

To disable verification of SSL peers set the PERL_LWP_SSL_VERIFY_HOSTNAME
envirionment variable to 0. If you do this you can't be sure that you
communicate with the expected peer.
EOT
  }
  die $@;
     }
     $ssl_opts{SSL_ca_file} = Mozilla::CA::SSL_ca_file();
 }
    }
    $self->{ssl_opts} = \%ssl_opts;
    return (%ssl_opts, $self->SUPER::_extra_sock_opts);
}

Then I get this instead

Error getting PE routers via REST to blc.serv.as2116.net: 401 Unauthorized({"error":{"code":401,"message":"Unauthorized: Basic Authentication Required"}})

which means that the SSL handshake was completed.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: libwww-perl 6.05-2
ProcVersionSignature: Ubuntu 3.13.0-43.72-generic 3.13.11.11
Uname: Linux 3.13.0-43-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
Date: Wed Jan 7 16:05:09 2015
InstallationDate: Installed on 2014-12-19 (19 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
PackageArchitecture: all
SourcePackage: libwww-perl
UpgradeStatus: No upgrade log present (probably fresh install)

[sullr (coyote-frank)]

The option verify_hostname is like the name suggests only responsible for verifying the host name against the certificate. It does not control the verification of the certificate chain or any other certificate validations, even if it can be used like this in some versions of LWP::Protocol::https. But this is actually a bug, see https://github.com/libwww-perl/lwp-protocol-https/pull/14 (very long discussion).

The only documentation of the option verify_hostname in LWP::UserAgent says:

           "verify_hostname" => $bool
            When TRUE LWP will for secure protocol schemes ensure it connects to servers that have a valid certificate
            matching the expected hostname.

Which confirms that this option cares about verifying the host name only.

To disable any kind of certificate validation you have to use ssl_opts to set SSL_verify_mode to 0 (i.e. SSL_VERIFY_NONE).

[janl (janl)]

So I changed LWP::Porotocol::https back and changed my program thus:

# $ua->ssl_opts( verify_hostname => 0 );
    $ua->ssl_opts( SSL_verify_mode => 0 );

This causes the same error:

Error getting PE routers via REST to blc.serv.as2116.net: 500 Can't connect to blc.serv.as2116.net:443 (certificate verify failed)(Can't connect to blc.serv.as2116.net:443 (certificate verify failed)

So it's still not possible to disable the certificate check. Not to mention that the documentation in the package hasn't been updated to reflect this.

# $ua->ssl_opts( verify_hostname => 0 );
    $ua->ssl_opts( SSL_verify_mode => 0 );
    $ua->ssl_opts( SSL_verifycn_scheme => 'none' );

Additionally it seems to break the principle of least surprise, if someone has working code tested with CPAN LWP and then tries to run it on Ubuntu it will fail without explanation (or any documentation).

[janl (janl)]

I forgot to write that the SSL_vrifycn_scheme => none setting was tested on a whim and it also did not help. That's why the code fragment is there.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Erik Squires (erik-squires)]

The problem is fixed by LWP::Protocol::https version 6.06. You can either get it from there, or at least patch the bad method. To make it easier, here is the fix, below. The issue seems to be the first else:

sub _extra_sock_opts
{
    my $self = shift;
    my %ssl_opts = %{$self->{ua}{ssl_opts} || {}};
    if (delete $ssl_opts{verify_hostname}) {
 $ssl_opts{SSL_verify_mode} ||= 1;
 $ssl_opts{SSL_verifycn_scheme} = 'www';
    }
    else {
 $ssl_opts{SSL_verify_mode} = 0;
    }
    if ($ssl_opts{SSL_verify_mode}) {
 unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
     eval {
  require Mozilla::CA;
     };
     if ($@) {
  if ($@ =! /^Can't locate Mozilla\/CA\.pm/) {
      $@ = <<'EOT';
Can't verify SSL peers without knowing which Certificate Authorities to trust

This problem can be fixed by either setting the PERL_LWP_SSL_CA_FILE
environment variable or by installing the Mozilla::CA module.

To disable verification of SSL peers set the PERL_LWP_SSL_VERIFY_HOSTNAME
environment variable to 0. If you do this you can't be sure that you
communicate with the expected peer.
EOT
  }
  die $@;
     }
     $ssl_opts{SSL_ca_file} = Mozilla::CA::SSL_ca_file();
 }
    }
    $self->{ssl_opts} = \%ssl_opts;
    return (%ssl_opts, $self->SUPER::_extra_sock_opts);
}

[Erik Squires (erik-squires)]

er, by "there" I meant from CPAN.