[SRU] infinite loop in /etc/init/mysql.conf if mysqld is not running.

[Gary B]

Please see bug no. 551097 (under Upstart) for more details. This bug occurred when moving from 8.04 to Lucid (10.0.0.4) beta.

The file /etc/init/mysql.conf will go into an infinite loop waiting for mysqladmin to succeed if the mysqld daemon is failing. This error is silent, so it was rather difficult to track down where the problem lay - figuring out where initctl config files were, then trying to modify the conf file.

In my case this was leaving mysql-server in a state between not-installed and installed, which caused the entire Software Update system to screech to a halt. Every time I tried to run any update process it would hang indefinitely on the 'stop mysql' and 'start mysql' processes. I had to kill each of those in turn, so that dpkg could finish. /usr/sbin/mysqld did not yet exist.

To fix it, I finally ran the individual lines of the pre-start section. When I ran the install line, /usr/sbin/mysqld appeared. When I ran /usr/sbin/mysqld that failed because of complaints about certain lines in the old my.cnf file. I backed that up and copied my.cnf.dpkg-dist to my.cnf and mysqld could then start. Once I had mysqld runnable I was able to run mysqladmin, start mysql, stop mysql and so forth without further problems and now dpkg succeeds.

So there are really two problems:
  1) the infinite loop should be handled differently, and the errors in attempting to install and run mysqld should be visible somewhere so people can figure out what's wrong;
  2) 'start mysql' should have some error output;
  3) there are evidently some incompatibilities between the my.cnf from 8.04 and from 10.0.0.4.

here is the error message when running /usr/sbin/mysqld from the command line. AFAIK the old configuration is the default from 8.04
# /usr/sbin/mysqld
100329 14:58:46 [Warning] The syntax '--log' is deprecated and will be removed in MySQL 7.0. Please use '--general_log'/'--general_log_file' instead.
100329 14:58:46 [Note] Plugin 'FEDERATED' is disabled.
100329 14:58:46 InnoDB: Started; log sequence number 0 3776469
100329 14:58:46 [ERROR] /usr/sbin/mysqld: unknown option '--skip-bdb'
100329 14:58:46 [ERROR] Aborting

100329 14:58:46 InnoDB: Starting shutdown...
100329 14:58:47 InnoDB: Shutdown completed; log sequence number 0 3776469
100329 14:58:47 [Note] /usr/sbin/mysqld: Shutdown complete

[Mathias Gug (mathiaz)]

The skip-bdb option is no longer supported in mysql 5.1.

Was the original hardy my.cnf file modified before the upgrade? If so dpkg should have prompted you to review the changes between between the locally modified hardy version and the new maintainer lucid version of my.cnf.

[Gary B (gar37bic)]

Comparing old and new I don't think there were any changes, although there is a 'log' entry that might not have been original. I do not recall seeing any prompt on this but it's possible that I missed it - the upgrade from 8.04 to 10.0.0.4 was a long process, and a lot of it ran overnight while I was elsewhere. But if the dialog would have required me to click 'OK' or whatever then I don't think that happened. I can attach a copy of my old my.cnf if you want it.

Perhaps this is a minor issue, but two things could be done differently:
  * since 'skip_bdb' is deprecated precisely because it's no longer needed, this should at most generate a warning when mysql starts up. It's odd to me that mysql should abort because of a configuration option that is agreeing with it. But I suppose that is really up to the mysql folks.
  * IMHO the initctl mysql.conf file (or any automated script) should not have a potential infinite loop like this, especially with no output to tell what is happening. Perhaps this could try to start a certain number of times, then report whatever error messages are coming from mysqladmin and/or mysqld to syslog or dmesg or whatever. Then at least there would be some error output. To get fancy, you could have a double retry loop, spitting out a warning every time the outer loop goes round.

In any case it's just a bad idea to sit and spin forever with no output. I think this is the real 'bug'.

[Mathias Gug (mathiaz)]

Could paste a diff between your old hardy my.cnf file and the lucid one?

[Gary B (gar37bic)]

Here 'tis. my.cnf is a copy of my.cnf.pkg-dist and my.cnf.old is my old one. I should say that it is possible that I was asked about whether to keep the old one or install the new one, and I might have said keep the old one. I just don't recall.

$ diff my.cnf my.cnf.old
29a30,40
> default-collation=latin1_general_ci
>
> #Enter a name for the query log file. Otherwise a default name will be used.
> #log=/var/log/mysql_query.log
>
>
> #Logs will be rotated after expire-log-days days
> expire_logs_days=30
>
>
>
42a54
> pid-file = /var/run/mysqld/mysqld.pid
47a60
> language = /usr/share/mysql/english
58,62c71,72
< thread_stack = 192K
< thread_cache_size = 8
< # This replaces the startup script and checks MyISAM tables if needed
< # the first time they are touched
< myisam-recover = BACKUP
---
> thread_stack = 128K
> thread_cache_size = 8
69c79
< query_cache_limit = 1M
---
> query_cache_limit = 1M
76,81c86,89
< # As of 5.1 you can enable the log at runtime!
< #general_log_file = /var/log/mysql/mysql.log
< #general_log = 1
<
< log_error = /var/log/mysql/error.log
<
---
> log = /var/log/mysql/mysql.log
> #
> # Error logging goes to syslog. This is a Debian improvement :)
> #
96a105,109
> # * BerkeleyDB
> #
> # Using BerkeleyDB is now discouraged as its support will cease in 5.1.12.
> skip-bdb
> #
100a114,115
> # You might want to disable InnoDB to shrink the mysqld process by circa 100MB.
> #skip-innodb
126a142,153
> # * NDB Cluster
> #
> # See /usr/share/doc/mysql-server-*/README.Debian for more information.
> #
> # The following configuration is read by the NDB Data Nodes (ndbd processes)
> # not from the NDB Management Nodes (ndb_mgmd processes).
> #
> # [MYSQL_CLUSTER]
> # ndb-connectstring=127.0.0.1
>
>
> #
128c155
< # The files must end with '.cnf', otherwise they'll be ignored.
---
> # The files must end with '.cnf', otherwise they'll be ignored.
130a158
>

[Nev Delap (nevdelap)]

I have the same problem in a clean install of 10.04 (iso downloaded today). I believe I had the same problem when I did an upgrade of 9.10 to 10.04 yesterday - I had to kill the install of mysql-server which had hung, but I've done the complete install today.

Synaptic hangs on start mysql and the install can't complete. Same thing if I uninstall the half installed version, and when I reinstall again.

[whitefawn (whitefawn)]

This affects me as well.
in /etc/mysql/conf.d there was a symlink to /home/user/my.cnf[mysql]
-----------------------------------------begin my.cnf------------------------
default-character-set = utf8

[mysqld]

default-character-set = utf8

character-set-server = utf8
character_set_database = utf8

-----------------------------------------begin my.cnf------------------------

After I removed the symlink I was able to start/restart mysql.

[Joe Holloway (jholloway7)]

In the old /etc/init.d/mysql script, there was a similar "check status" loop, but it only spun 14 times before exiting. If there were a problem with your server, you'd only block for max of ~14 seconds while trying to ping the server before breaking out.

    for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do
        sleep 1
        if mysqld_status check_alive nowarn ; then break; fi
        ...
    done
    ...

In the new `upstart` script /etc/init/mysql.conf, the analogous status check was changed to a "while ! ..." loop which means the script will spin forever when your server has a problem?

    while ! /usr/bin/mysqladmin --defaults-file=$HOME/debian.cnf ping
    do
        sleep 1
    done

It seems that there are a few common errors that are easily resolvable, for example "Can't connect to MySQL server through socket '/var/run/mysqld/mysqld.sock'" error or issues with the debian-sys-maint user that some of us have from time-to-time, but there appears to be a chicken/egg scenario here caused by this infinite loop.

As a side note, I've now completely lost my /etc/init/mysql.conf file and can't seem to get it back through the package manager regardless of purging/reinstalling packages. That concerns me a bit, but I can't pinpoint the behavior well enough to log a separate bug at this point.

[Mathias Gug (mathiaz)]

On Wed, May 05, 2010 at 03:17:12PM -0000, Joe Holloway wrote:
>
> As a side note, I've now completely lost my /etc/init/mysql.conf file
> and can't seem to get it back through the package manager regardless of
> purging/reinstalling packages. That concerns me a bit, but I can't
> pinpoint the behavior well enough to log a separate bug at this point.
>

Make sure you're *purging* the package that ships /etc/init/mysql.conf.

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

[pdf (pdffs)]

Infinite loop is bad - kills my HA clusters if there's an error, as the service cannot be migrated away from the node with the issue. Certainly propose re-instating the limit, as this is a regression.

[ServerAlex (serveralex)]

I'm experiencing the same hang. But it also hangs on stop/restart when there is an error in the my.cnf. This is really annoying..

[Sam Barker (sam-quadrocket)]

I updated to version 5.1.41-3ubuntu12.1 yesterday and my problems with mysql hanging on start up have gone away.

[pdf (pdffs)]

@Sam Barker - you must have inadvertently fixed whatever error was causing the "/usr/bin/mysqladmin --defaults-file=$HOME/debian.cnf ping" command to fail. Try breaking your my.cnf as others have suggested, and you'll see the infinite loop still exists.

[Sam Barker (sam-quadrocket)]

I'll take that back. Its failing again after a reboot.

After I get the hung instance stopped sudo dpkg-reconfigure mysql-server-5.1 fixes the problem until reboot.

[C de-Avillez (hggdh2)]

Nominating for Lucid per zul's request

[Chuck Short (zulcss)]

For those who are having problems with this can you try the version in my ppa when it builds:

https://edge.launchpad.net/~zulcss/+archive/server-bug-fixes

thanks
chuck

[StephanNies (nies-stephan)]

Please note that the cause of the problem is that /etc/init/mysql.conf is trying to start mysqld as user root. If you start as user mysql everything works fine.

The infinte loop described above in

 while ! /usr/bin/mysqladmin --defaults-file=$HOME/debian.cnf ping
    do
        sleep 1
    done

is caused by mysqld not running and therefore /usr/bin/mysqladmin --defaults-file=$HOME/debian.cnf ping
failing.
With no proper timeout this causes an infinte loop.

So you have to apply the following 2 fixes to /etc/init/mysql.conf:

1) to make msqld work:

exec sudo -u mysql /usr/sbin/mysqld # <- important execute as use mysql

2) to make sure the loop times out:

   count=0
   while [ $count -lt 10 ] && ! /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping 2>/dev/null
   do
       echo "waiting for mysqld: ${count}s ..."
       sleep 1
       count=`expr $count + 1`
   done

   if [ $count -eq 10 ]
   then
       echo "timeout."
   else
       exec $HOME/debian-start
   fi

[Artis Caune (artis.caune)]

When using EBS snashots on EC2, debian-sys-maint is different on new servers, so it stuck forever.

Also init/mysql.conf use:
    exec /usr/sbin/mysqld

and I see error when running it like that:

100528 11:00:25 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
100528 11:00:25 [ERROR] Aborting
100528 11:00:25 [Note] /usr/sbin/mysqld: Shutdown complete

should'n it use mysqld_safe instead ?

This works for me:

....
exec /usr/bin/mysqld_safe

post-start script
    for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do
        /usr/bin/mysqladmin --defaults-file=$HOME/debian.cnf ping && exec $HOME/debian-start
        sleep 1
    done
end script

[pdf (pdffs)]

@Chuck - preliminary testing seems to indicate that the upstart job in your package will allow mysql to enter started status even when it does not start successfully. I'll try look at creating something a little cleaner in the next couple of days.

[pdf (pdffs)]

How about the attached upstart job?

[pdf (pdffs)]

Sorry, slight modification

[Mr. Bug (bug-trash-mail)]

just my 2 cents: The problem was (in my case) that at the time mysql upstart job is fired /var/run isn't mounted. There should be a tempfs in /var/run. Additional apparmor prevents mysql to put its sock under /var/run instead of /var/run/mysqld.

How can I tell upstart to start after mounted-varrun.conf is run?!

[pdf (pdffs)]

@Mr. Bug - I think you've got other problems if /var/run is not mounted by the time mysql starts. And mysql is already set to start after all local filesystems.

[m2nis (m-ubuntu)]

I will try to describe all what I have find on this problem which start with the upgrade from 8.04 to 10.04.

First (I forgot how the problem began...), I remove mysql-server :

$ sudo apt-get remove mysql-server --purge
$ sudo apt-get autoremove
$ sudo apt-get autoclean
and I also remove the directories /etc/mysql, /var/lib/mysqld, /var/run/mysqld, /var/log/mysql

reboot

then reinstall of the package mysql-server. This install gives me the error (it's a translation) "Unable to change the password for user 'root' of mysql".

Then :

$ sudo service mysql start
start: Job failed to start

If I copy the my.cnf which is missing, the "service start" gives me an infinite loop. But the line :

$ sudo su - mysql service mysql start

works. But if I try to connect :

$ mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

That's the main informations I have.

[m2nis (m-ubuntu)]

Not sure how I did it, but I succeeded in removing the password for root. I can now connect to the database. But if I try to change the root password of mysql using "sudo dpkg-reconfigure mysql-server-5.1", I get :

stop: Unknown instance:

and another loop I can break with Ctrl+C. And the password is not changed.

[Tom Ellis (tellis)]

Hit the same issue, when there was a typo in my.cnf, mysql-server is a stuck in an infinite post-start loop.

[Zack Evans (zevans23)]

In my case it must have been webmin that's put something in the my.cnf file that it doesn't like.

I'll have to find out what it doesn't like and file a separate bug against webmin, as the cause - but meanwhile I can confirm there is a bug with the startup script. It doesn't fail gracefully, just hangs.

Is this bug in the equivalent Debian upstream?

[Zack Evans (zevans23)]

Oh - the startup problem is disguised as a permissions problem, but actually it's because the db file is on an NTFS partition - so it's not webmin.

But - a startup error would be nice...!

[pdf (pdffs)]

The bug doesn't exist in Debian since they don't use Upstart - OT, but IMHO it was way too early to introduce Upstart since it doesn't even provide the same level of functionality as SysV, but that's another story... The important thing when migrating though is not to make assumptions when changing the behaviour, as is what happened here.

[stininkey skunkenheimer (jdieter-twonails)]

Does ANYONE have a solution for: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
Can I install an old version or something? I MUST get this server up. Nothing will get it to run.

[Artis Caune (artis.caune)]

On 7 July 2010 05:34, stininkey skunkenheimer <email address hidden> wrote:
> Does ANYONE have a solution for: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
> Can I install an old version or something? I MUST get this server up. Nothing will get it to run.
>

Try to edit /etc/init/mysql.conf and change:
"exec /usr/sbin/mysqld"
to
"exec /usr/bin/mysqld_safe"

--
Artis Caune

    Everything should be made as simple as possible, but not simpler.

[Robbie Williamson (robbiew)]

Targeting to 10.04.1 as mysql is a very popular server application. If a solution cannot be found in time, then it needs to be targeted to an SRU or 10.04.2.

[chag (chagam)]

Hi,

is there any solution to this major issue (mysql serever not working in ubuntu 10.04) ?

[Chuck Short (zulcss)]

Statement of Impact: In some cases (ie broken mysql configs). the mysql upstart script will fail to start. This is due to a bug in the mysql upstart task that will wait forever for mysql to start.

How this has been addressed: This has been fixed and tested in maverick.

How to reproduce:

1. Break mysql
2. Try to start mysql

There should be no regressions for this fix.

If you have any questions please let me know.

chuck

[Martin Pitt (pitti)]

Accepted mysql-dfsg-5.1 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Jürgen Kreileder (jk)]

The new post-start script is broken, see bug #608423

[Neil Wallace (rowinggolfer)]

it seems that the inifinte loop is fixed by this patch, but mysql is not starting at boot after installing the packages from proposed, so boot time is significantly increased.

jurgen's new bug may be relevant??

[Richard Hansen (rhansen)]

Setting verification-failed due to bug #608423.

[Martin Pitt (pitti)]

Accepted mysql-dfsg-5.1 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Fixed in maverick's mysql-5.1 5.1.48-1ubuntu4.

[Richard Hansen (rhansen)]

Setting verification-failed due to bug #611101 (upstart script does not sleep between mysqladmin pings).

[Richard Hansen (rhansen)]

Verification also failed due to bug #611102 (missing space preventing startup at boot).

[Martin Pitt (pitti)]

Accepted mysql-dfsg-5.1 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Richard Hansen (rhansen)]

5.1.41-3ubuntu12.6 works for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-dfsg-5.1 - 5.1.41-3ubuntu12.6

---------------
mysql-dfsg-5.1 (5.1.41-3ubuntu12.6) lucid-proposed; urgency=low

  * debian/mysql-server-5.1.mysql.upstart:
    - Insert missing space so that mysqld starts at boot again
    - Sleep between pings to mysqld
    - Immediately return failure if 'exec' fails
    - This fixes (LP: #611101) and (LP: #611102)

mysql-dfsg-5.1 (5.1.41-3ubuntu12.5) lucid-proposed; urgency=low

  * debian/mysql-server-5.1.mysql.upstart: Fix typo. (LP: #608423)

mysql-dfsg-5.1 (5.1.41-3ubuntu12.4) lucid-proposed; urgency=low

  * debian/mysql-server-5.1.mysql.upstart: Don't wait forever to mysql to come up.
    (LP: #551130)
 -- a7x <email address hidden> Wed, 28 Jul 2010 11:37:25 -0400

[IAmDaveMyers (davemyers)]

I don't know if this bug is still open, (or if my problem is related to another bug), but the 'released fix' hasn't been in the lucid repositories.

when I perform the following

sudo apt-get install mysql-server-5.1

this is the output...
___
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  tinyca mailx
The following NEW packages will be installed
  mysql-server-5.1
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 0B/7,008kB of archives.
After this operation, 15.0MB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously deselected package mysql-server-5.1.
(Reading database ... 271591 files and directories currently installed.)
Unpacking mysql-server-5.1 (from .../mysql-server-5.1_5.1.41-3ubuntu12.6_i386.deb) ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
Setting up mysql-server-5.1 (5.1.41-3ubuntu12.6) ...
101027 15:46:57 [Note] Plugin 'FEDERATED' is disabled.
101027 15:46:57 InnoDB: Started; log sequence number 0 44233
101027 15:46:57 InnoDB: Starting shutdown...
101027 15:46:58 InnoDB: Shutdown completed; log sequence number 0 44233
start: Job failed to start

Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

___

My concern is that this is the LTS version. As always I am upgrading after the version has been out for a while, in the expectation that problems like these would not occur.

My initial upgrade from 8.04 broke MySQL, and stopped my local test versions for a number of test web sites (drupal) and a test implementation of Sugar CRM I had going.

I assume that there is a similar problem in the 10.04 LTS server edition also ? if so that will be bad new for someone!

I hope someone out there is able to regress the 'fix released' notification, or at least confirm that my problems are not specific to my setup.

I followed m2nis' details on post 23 for removing my old mysql instalation (or rather I renamed the directories so as to retain my data tables), then I did the following

___
su mysql service mysql start
start: Rejected send message, 1 matched rules; type="method_call", sender=":1.50" (uid=112 pid=20679 comm="start) interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply=0 destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init"))
___

then so as to confirm that it is indeed not running...

___
ps aux|grep mysql
root 20762 0.0 0.1 3328 784 pts/0 S+ 22:40 0:00 grep mysql
___

During my upgrade (which I performed stepwise through the versions, as I wasn't being given the option to go directly from 8.04 to 10.04?) I was prompted to file a bug report about mysql, here is the link to it
https://bugs.launchpad.net/ubuntu/+source/mysql-dfsg-5.1/+bug/666383

The reason I think this is in fact somehow all related is because my var/log/syslog ouptut looks remarkable similar to the original posters

I am now going to try implementing the changes as suggested above, to see if that solves my problem.

David

[Richard Hansen (rhansen)]

Please file a new bug report -- there may have been a regression.

[IAmDaveMyers (davemyers)]

@a7x

rather than file a new one, I'll add in further details to the one that I mentioned in my above post.

Also I'm trying to initiate the path from Chuck Short (post 15 above), but how do I do this?

This is to confirm if his patch is OK an hence show that a regression has occured

I've added the ppa to synaptic, but mysql doesn't come up in the list of available packages from that source?

so I decided to download the file

mysql-server-5.1_5.1.41-3ubuntu12.2_i386.deb

then installed using gdebi, - which kindly tells me that there is a newer version in the repositories.

then I try to start the service, and get the following...

[code]
sudo service mysql start
start: Job failed to start
[/code]

I'm obviously doing something wrong.

OK back to trying the other fixes....

All further correspondence will now be on the open bug...
https://bugs.launchpad.net/ubuntu/+source/mysql-dfsg-5.1/+bug/666383

David

[MarkieB (ubunt-u-markbenjamin)]

I'll add comment here as it looks as though there is a regression, I tried to install mysql-server-5.1 today in Maverick, dpkg got stuck, then after some struggling I adjusted /etc/init/mysql.conf to start mysqld_save as -u mysql, good up to the stage of trying to connect; then needed to 'downgrade' the dpkg edit of /etc/apparmor.d/usr.sbin.mysqld; files attached, working file is now called usr.sbin.mysqld, dpkg had saved it as usr.sbin.mysqld.dpkg-old;

broken file is now called usr.sbin.mysqld.dpkg-new

[MarkieB (ubunt-u-markbenjamin)]

In fact I've confirmed that the changes to /etc/init/mysql.conf are unnecessary, it's somehow in the magic of /etc/apparmor.d/usr.sbin.mysqld were I to delve into its intricacies

[MarkieB (ubunt-u-markbenjamin)]

incidentally that's mysql-server-5.1_5.1.49-1ubuntu8.1_amd64.deb ; the file that I've uploaded as usr.sbin.mysqld.dpkg-new is the "package maintainer's version" so should be unedited

[Richard Hansen (rhansen)]

@MarkieB: Please file a new bug report for the regression; this one is closed.

[amichair (amichai2)]

>>> www.xviapills.ru

- There is no reason to feel depressed if you, like so many other men today, have hit the rough spot of your sexual life when you cannot seem to be performing as well as you used to do. Your agitation and frustration are easily understood, and still you should know that with the modern development of worldwide pharmacological industries it is but natural that there is bound to be a solution for your own needs when it comes down to solving your erectile dysfunction problems.

- We suggest you getting erectile dysfunction drugs from our online drugstore cheaply, easily and without much to-do. You will always be taken care of and attended to with extreme care, and you will have the possibility of choosing from a wide range of erectile dysfunction products to find a solution that will best suit your particular needs.

- The whole world puts great trust in those famous diamond shaped pills, and if you are not an exception we would like to revise some basic points that speak in favor of putting your male health in the hands of those magical tablets.
Firstly, there is no need for regular therapy � those blue pills have to be taken directly before the planned sexual contact, and extra convenience shines through when you come to realize that the pills will not start working unless sufficient sexual stimulation is provided.

- Secondly, this erectile dysfunction solution is known for its extremely low adverse side effects profile, which makes it available for all age groups of patients. Besides, it really does not matter how old you are when it goes about taking erectile dysfunction pills � just make sure that you do not overdose, and they will work for you at their best!

- Finally, those fabulous pills do not only provide you with stable erection that can be maintained for as long as you need it � they also help you to achieve unheard sensitivity during the sexual act as well as make your genitals look much more sizey!

- Think of all those benefits of taking erectile dysfunction pills and the effect you can produce on your loved one � and do not hesitate to make your order and / or refills right away!

--- > www.xviapills.ru < ---

 _V______C________L
 ___I_______I_______E
 _____A_______A_______V
 _______G_______L_______I
 _________R_______I_______T
 ___________A_______S_______R
 _____________________________A

   Classic male pilules for boosting pilules on much better prices than usually! Get them to your house confidentially, full money-back if no super-result achieved, no questions asked! When you need wild bed action, clicking here will be the first step to fantastic results!
-
 - - -
--------- www.xviapills.ru
 - - -
-