Bug #825593 “cntlm does not work at reboot” : Bugs : cntlm package : Ubuntu

Revision history for this message

Stefano Rivera (stefanor) wrote on 2011-08-13:

#1

> Does the cntlm service started too early on reboot?

Yes, this sounds like LP: #358298, which I worked around by adding an ifup.d script for.

Unfortunately, cntlm requires connectivity to its parent proxy at startup. It would be preferable if it didn't and just returned 500s until it could contact its parent.

Revision history for this message

Edoardo Tirtarahardja (etirta) wrote on 2011-08-14:

#2

Download full text (3.8 KiB)

Hi Stefano,

Thank you so much for your quick response :).

I have just installed the cntlm 0.91rc6 a couple of days ago via aptitude, so it should have your fix in '/etc/network/if-up.d/' (I will verify this on Mon. when I get back to the office). But from the boot log, I do notice the 2nd round print about ACLs are being added. It kind of puzzled me before, but after I read your fix, I understand that this got to be the 2nd attempt to start cntlm when the network interface is finally up.

This is my situation. I have 2 network interface, eth00 (192.168.1.1) and eth10(192.168.241.1 and a bunch of other secondaries IPs). Based on the my cntlm config 1 (that I described in my original posting), I get error print on the boot log stating that the last 'Listen' interface is not available. So I guess it takes longer to bring up eth10 as it has several IPs bound to it. So when eth00 is up and invokes '/etc/network/if-up.d/', the eth10 is not ready when cntlm is started. And when finally eth10 is up, the 2nd invocation of '/etc/network/if-up.d/' won't do anything to cntlm as start-stop-daemon simply quits as cntlm has been started. The 'netstat -an | grep LISTEN' seems to confirm this situation.

So I got into thinking, may be it's better to do 'restart' rather than 'start' in '/etc/network/if-up.d/'. With this, and ifup event will restart cntlm, so when some body do ifdown & then ifup manually, the cntlm is restarted and if the cntlm listen to the newly up interface, it will now listen properly. Of course there is a possible drawback, that if the cntlm does not listen to this newly up interface, then it may be restarted for nothing.

Btw, I have just learn linux (ubuntu) a few months back, so there might be some stuff I mentioned above is completely stupid or non-sense :(.

Do you know what happen with cntlm (or any process that open a socket for listening) if the listened interface suffers link failure? Will the socket connection is closed? Hence when the interface recovers later on, it will not listen to it any longer?

Will '/etc/network/if-up.d/' executed when the link recovers? Or it will only be executed on manual ifup?

If my suggestion using 'restart' instead of 'start' is acceptable, will you be kind propose the change? Also please incorporate: http://sourceforge.net/tracker/?func=detail&aid=3206995&group_id=197861&atid=963162
changes that actually check the '/etc/rc<x>.d/', whether the cntlm is supposed to be started at the current runlevel or not.

OK, now about the Config 2 of my cntlm. This one I still get puzzled :(. I configured cntlm as gateway, hence it open a socket connection '0.0.0.0:8081'. So during the '/etc/network/if-up.d/' invocation when eth00 is up, managed to start cntlm correctly (at least based on 'netstat -an | grep LISTEN'). So when the eth10 is up, even though the cntlm is not restart, it should still work as the linux network driver will forward the incoming packet to cntlm as the socket was opened with '0.0.0.0' address. And the fact that I got HTTP 502 result from cntlm, this confirms that cntlm does receive the HTTP GET coming from eth10. Btw, I also configure cntlm as standalone proxy (NoProxy *), so this reque...

Hi Stefano,

Thank you so much for your quick response :).

I have just installed the cntlm 0.91rc6 a couple of days ago via aptitude, so it should have your fix in '/etc/network/if-up.d/' (I will verify this on Mon. when I get back to the office). But from the boot log, I do notice the 2nd round print about ACLs are being added. It kind of puzzled me before, but after I read your fix, I understand that this got to be the 2nd attempt to start cntlm when the network interface is finally up.

This is my situation. I have 2 network interface, eth00 (192.168.1.1) and eth10(192.168.241.1 and a bunch of other secondaries IPs). Based on the my cntlm config 1 (that I described in my original posting), I get error print on the boot log stating that the last 'Listen' interface is not available. So I guess it takes longer to bring up eth10 as it has several IPs bound to it. So when eth00 is up and invokes '/etc/network/if-up.d/', the eth10 is not ready when cntlm is started. And when finally eth10 is up, the 2nd invocation of '/etc/network/if-up.d/' won't do anything to cntlm as start-stop-daemon simply quits as cntlm has been started. The 'netstat -an | grep LISTEN' seems to confirm this situation.

So I got into thinking, may be it's better to do 'restart' rather than 'start' in '/etc/network/if-up.d/'. With this, and ifup event will restart cntlm, so when some body do ifdown & then ifup manually, the cntlm is restarted and if the cntlm listen to the newly up interface, it will now listen properly. Of course there is a possible drawback, that if the cntlm does not listen to this newly up interface, then it may be restarted for nothing.

Btw, I have just learn linux (ubuntu) a few months back, so there might be some stuff I mentioned above is completely stupid or non-sense :(.

Do you know what happen with cntlm (or any process that open a socket for listening) if the listened interface suffers link failure? Will the socket connection is closed? Hence when the interface recovers later on, it will not listen to it any longer?

Will '/etc/network/if-up.d/' executed when the link recovers? Or it will only be executed on manual ifup?

If my suggestion using 'restart' instead of 'start' is acceptable, will you be kind propose the change? Also please incorporate: http://sourceforge.net/tracker/?func=detail&aid=3206995&group_id=197861&atid=963162
changes that actually check the '/etc/rc<x>.d/', whether the cntlm is supposed to be started at the current runlevel or not.

OK, now about the Config 2 of my cntlm. This one I still get puzzled :(. I configured cntlm as gateway, hence it open a socket connection '0.0.0.0:8081'. So during the '/etc/network/if-up.d/' invocation when eth00 is up, managed to start cntlm correctly (at least based on 'netstat -an | grep LISTEN'). So when the eth10 is up, even though the cntlm is not restart, it should still work as the linux network driver will forward the incoming packet to cntlm as the socket was opened with '0.0.0.0' address. And the fact that I got HTTP 502 result from cntlm, this confirms that cntlm does receive the HTTP GET coming from eth10. Btw, I also configure cntlm as standalone proxy (NoProxy *), so this request will result direct_request() in direct.c is called. And the only place in direct.c that generates HTTP 502 is if the host_connect() call failed. And I simply can't figure it out why on earth at this point the host_connect() fails. My IP trace on eth10 shows that cntlm doesn't even try to open TCP connection (SYN) to the requested server at all. A simple manual restart of cntlm fix this. Do you have any thought on this matter?

Btw, I also can't find where cntlm try to check the connectivity to it's parent proxy at startup, do you mind point me to it?

Again, many thanks for your kind response and I hope you have time to response to my lengthy questions :).

Cheers //Edo

Revision history for this message

Stefano Rivera (stefanor) wrote on 2011-08-14:

#3

Yup, obviously my workaround isn't much help when you have two interfaces. I suggest locally editing the ifup.d script, I don't think restarting is an appropriate response for everyone.

Even better, work on improving cntlm :)

Sorry, I can't understand what your issue with "config 2" is. Was the parent proxy available at the time that cntlm started? It needs to be.

Revision history for this message

Edoardo Tirtarahardja (etirta) wrote on 2011-08-14:

#4

Oke, I will modify my '/etc/network/if-up.d/' script.

I have asked the David Kubicek to fork a process to monitor the ifup event and act accordingly. Since I'm a newbie in linux, I don't have experience in linux programing :(.

In regards with my issue in Config 2, I check the source code of cntlm 0.91rc6 and can't find the part that cntlm try to contact the parent proxy. I might missed it, do you mind point me to it?

My parent proxy is in 192.168.241.1:8081, i.e. on eth10, it's not available when the cntlm is started due to eth00 is up. But since I configure it as standalone proxy, this parent proxy as far as I can tell is never used nor contacted. The proxy_connect() is called inside forward_request() and socks5_thread(), since I configure cntlm as standalone proxy & no socks5 proxy, the proxy_connect() should never be called.

If I remember correctly, even when I restart cntlm manually (after all my interfaces are up) I don't see any attempt to contact the parent proxy from the eth10 IP trace. I will verify again on Mon. to make sure this is the case.

Revision history for this message

Edoardo Tirtarahardja (etirta) wrote on 2011-08-15:

#5

Hi Stefano,

I just want to give you an update on my trouble shooting effort today.

I found out that the execution of '/etc/network/if-up.d/' is not reliable all the time. From one boot to another boot I get different result.

Practically I have 3 interfaces, lo, eth00, and eth10.

On some boot, I get '/etc/network/if-up.d/' 3x on each those interfaces. On other boot, I only get 1x. Most of the boot, I get 2x on eth00 & eth10 (some times the order is the other way around). Obviously at boot ubuntu may combine the execution of '/etc/network/if-up.d/' if multiple network interface is up very close.

Now, when I have multiple execution of '/etc/network/if-up.d/', I can see that some times I have race condition (if the '/etc/network/if-up.d/cntlm' are executed too close with each other) that the start-stop-daemon fail to detect the start of the previous instance. Giving me 2 instances of cntlm trying to start at the same time.

Due to all this process happens pretty much in parallel, I even have the following sequence:
- eth00 up event executes '/etc/network/if-up.d/cntlm'
- eth10 up event executes '/etc/network/if-up.d/cntlm'
- '/etc/init.d/cntlm' is executed via '/etc/rc2.d/S20cntlm'
- '/etc/init.d/cntlm' is executed by '/etc/network/if-up.d/cntlm' due to eth00 is up
This some times cause the 2nd instance of cntlm to be started if it's executed too close
with the above.
- '/etc/init.d/cntlm' is executed by '/etc/network/if-up.d/cntlm' due to eth10 is up

So I ends up putting the following in my '/etc/network/if-up.d/cntlm':
# Check whether cntlm is enabled in this runlevel, if so restart it.
level="unknown"
while [ "$level" = "unknown" ]; do
    level=`runlevel | cut -d" " -f2`
done
if [ -e /etc/rc${level}.d/S??cntlm -a "$IFACE" != "lo" ]; then
    logger -f /var/log/syslog -t $IFACE Restarting cntlm in $(($$ % 10)) s.
    sleep $(($$ % 10))
# invoke-rc.d --quiet cntlm restart >/dev/null 2>&1 || true
fi

And by putting random sleep, I managed to avoid race condition at most of the time.

But, the cntlm STILL failed in forwarding the request from the client. Even I take away the '/etc/network/if-up.d/cntlm', I still get exact same result on both Config 1 & 2.

After putting '-v' when I start the cntlm in '/etc/init.d/cntlm', I get extra prints and get the following failure:
direct_request() -> host_connect()> so_resolv().
with error code that the destination is temporarily not available.

Now the strange thing is I always managed to ping the destination server by name when the cntlm error situation occurs. Since the so_resolve() is only a wrapper to gethostbyname(), then it should work.

Putting delay upto 10 sec. inside '/etc/init.d/cntlm' just before calling start-stop-daemon in start case, didn't help. I literally *always* have to restart the cntlm manually to make it work.

Any thought???

Cheers //Edo

Hi Stefano,

I just want to give you an update on my trouble shooting effort today.

I found out that the execution of '/etc/network/if-up.d/' is not reliable all the time. From one boot to another boot I get different result.

Practically I have 3 interfaces, lo, eth00, and eth10.

On some boot, I get '/etc/network/if-up.d/' 3x on each those interfaces. On other boot, I only get 1x. Most of the boot, I get 2x on eth00 & eth10 (some times the order is the other way around). Obviously at boot ubuntu may combine the execution of '/etc/network/if-up.d/' if multiple network interface is up very close.

Now, when I have multiple execution of '/etc/network/if-up.d/', I can see that some times I have race condition (if the '/etc/network/if-up.d/cntlm' are executed too close with each other) that the start-stop-daemon fail to detect the start of the previous instance. Giving me 2 instances of cntlm trying to start at the same time.

Due to all this process happens pretty much in parallel, I even have the following sequence:
- eth00 up event executes '/etc/network/if-up.d/cntlm'
- eth10 up event executes '/etc/network/if-up.d/cntlm'
- '/etc/init.d/cntlm' is executed via '/etc/rc2.d/S20cntlm'
- '/etc/init.d/cntlm' is executed by '/etc/network/if-up.d/cntlm' due to eth00 is up
  This some times cause the 2nd instance of cntlm to be started if it's executed too close
  with the above.
- '/etc/init.d/cntlm' is executed by '/etc/network/if-up.d/cntlm' due to eth10 is up

So I ends up putting the following in my '/etc/network/if-up.d/cntlm':
# Check whether cntlm is enabled in this runlevel, if so restart it.
level="unknown"
while [ "$level" = "unknown" ]; do
    level=`runlevel | cut -d" " -f2`
done
if [ -e /etc/rc${level}.d/S??cntlm -a "$IFACE" != "lo" ]; then
    logger -f /var/log/syslog -t $IFACE Restarting cntlm in $(($$ % 10)) s.
    sleep $(($$ % 10))
#   invoke-rc.d --quiet cntlm restart >/dev/null 2>&1 || true
fi

And by putting random sleep, I managed to avoid race condition at most of the time.

But, the cntlm STILL failed in forwarding the request from the client. Even I take away the '/etc/network/if-up.d/cntlm', I still get exact same result on both Config 1 & 2.

After putting '-v' when I start the cntlm in '/etc/init.d/cntlm', I get extra prints and get the following failure:
direct_request() ->  host_connect()>  so_resolv().
with error code that the destination is temporarily not available.

Now the strange thing is I always managed to ping the destination server by name when the cntlm error situation occurs. Since the so_resolve() is only a wrapper to gethostbyname(), then it should work.

Putting delay upto 10 sec. inside '/etc/init.d/cntlm' just before calling start-stop-daemon in start case, didn't help. I literally *always* have to restart the cntlm manually to make it work.

Any thought???

Cheers //Edo

Revision history for this message

Stefano Rivera (stefanor) wrote on 2012-08-01:

#6

I believe this is fixed in the version published in quantal. The fix was r280 upstream.

Changed in cntlm (Ubuntu):
status:	New → Fix Released

Revision history for this message

Graham Inggs (ginggs) wrote on 2012-08-02:

#7

debdiff for Precise attached to LP: #1009436

Graham Inggs (ginggs) on 2012-09-13

description:

updated

Revision history for this message

Chris Halse Rogers (raof) wrote on 2012-09-26: Please test proposed package

#8

Hello Edoardo, or anyone else affected,

Accepted cntlm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cntlm/0.92.3-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cntlm (Ubuntu Precise):
status:	New → Fix Committed
tags:	added: verification-needed

Revision history for this message

Graham Inggs (ginggs) wrote on 2012-10-01:

#9

Booted up with network cable unplugged.
Started Firefox and entered a URL and pressed Enter.
The message '502 connection timed out. Cntlm proxy failed to complete the request' appeared.
Plugged in network cable, waited for 'Connection Established' to appear.
Clicked Refresh in Firefox.
Web page loaded successfully.

Tested on both the i386 and amd64 versions of cntlm from precise-proposed.

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Colin Watson (cjwatson) wrote on 2012-10-03: Update Released

#10

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-10-03:

#11

This bug was fixed in the package cntlm - 0.92.3-0ubuntu0.1

---------------
cntlm (0.92.3-0ubuntu0.1) precise-proposed; urgency=low

  * Backport 0.92.3-1ubuntu1 to Precise as an SRU (LP: #1009436):
    - Properly handle non-HTTP/1.1 keep-alive (LP: #1009436, #257210)
    - Support SHOUTcast (ICY) internet radio protocol (LP: #659809)
    - Fix error when downloading files >2GB (LP: #1031670)
    - Resolve proxy hostname on demand, not at startup (LP: #825593)

cntlm (0.92.3-1ubuntu1) quantal; urgency=low

* Cherry-pick r306 from 0.93 to properly handle non-HTTP/1.1 keep-alive
(LP: #1009436)

cntlm (0.92.3-1) unstable; urgency=low

  * New upstream release. Closes: #652725, #588920.
  * Fix Init script error, thanks Martijn. Closes: #588683.
  * Correct spellings in man page.
  * Update Standards Version, no changes needed.
-- Graham Inggs <email address hidden> Wed, 29 Aug 2012 16:26:00 +0200

Changed in cntlm (Ubuntu Precise):
status:	Fix Committed → Fix Released

Affects		Status	Importance	Assigned to	Milestone
	cntlm (Ubuntu)	Fix Released	Undecided	Unassigned
	Precise	Fix Released	Undecided	Unassigned

Ubuntu
cntlm package

cntlm does not work at reboot

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntucntlm package

cntlm does not work at reboot

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntu
cntlm package