radeon_info_ioctl: [340700.513619] BUG: unable to handle kernel paging request at fffffffff0ea1e74

[Moisa Doru]

$uname -a
Linux doru-laptop 2.6.35-7-generic #11-Ubuntu SMP Wed Jul 7 04:25:00 UTC 2010 x86_64 GNU/Linux
$cat /proc/version_signature
Ubuntu 2.6.35-7.11-generic 2.6.35-rc4
$dmesg
[...]
[340443.929733] i2c i2c-2: sendbytes: NAK bailout.
[340448.259815] linuxdcpp[21889]: segfault at c0 ip 00007f094cb19644 sp 00007f093ce7a2d0 error 4 in libpthread-2.12.so[7f094cb10000+18000]
[340664.824023] i2c i2c-2: sendbytes: NAK bailout.
[340700.513619] BUG: unable to handle kernel paging request at fffffffff0ea1e74
[340700.513626] IP: [<ffffffffa0384c60>] radeon_info_ioctl+0x20/0x150 [radeon]
[340700.513655] PGD 1a2c067 PUD 1a2d067 PMD 0
[340700.513660] Oops: 0000 [#1] SMP
[340700.513663] last sysfs file: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/C1E9/current_now
[340700.513667] CPU 0
[340700.513668] Modules linked in: michael_mic arc4 nls_iso8859_1 nls_cp437 vfat fat binfmt_misc parport_pc ppdev rfcomm sco bridge stp bnep l2cap vboxnetadp vboxnetflt vboxdrv kvm_intel kvm vga16fb vgastate joydev snd_hda_codec_analog usbhid snd_hda_intel snd_hda_codec radeon snd_hwdep snd_pcm hid snd_seq_midi snd_rawmidi snd_seq_midi_event ttm lib80211_crypt_tkip snd_seq btusb snd_timer snd_seq_device video drm_kms_helper wl(P) hp_wmi output bluetooth snd psmouse serio_raw drm lib80211 lp intel_agp soundcore snd_page_alloc i2c_algo_bit parport usb_storage ahci libahci e1000e
[340700.513710]
[340700.513713] Pid: 26271, comm: googleearth-bin Tainted: P 2.6.35-7-generic #11-Ubuntu 30D7/HP Compaq 6820s
[340700.513716] RIP: 0010:[<ffffffffa0384c60>] [<ffffffffa0384c60>] radeon_info_ioctl+0x20/0x150 [radeon]
[340700.513730] RSP: 0018:ffff880063b8fdd8 EFLAGS: 00010296
[340700.513732] RAX: ffff880137b4e000 RBX: fffffffff0ea1e74 RCX: ffffffffa0423471
[340700.513735] RDX: ffff88001d4ac540 RSI: ffff880063b8fe58 RDI: ffff880128062800
[340700.513737] RBP: ffff880063b8fdf8 R08: ffffffffa042de70 R09: ffffffffa0384c40
[340700.513740] R10: ffff880063b8fe58 R11: 0000000000000001 R12: ffff880128062800
[340700.513742] R13: 0000000000000067 R14: 00000000c0106467 R15: 0000000000000000
[340700.513745] FS: 0000000000000000(0000) GS:ffff880001e00000(0063) knlGS:00000000f4bce720
[340700.513747] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
[340700.513750] CR2: fffffffff0ea1e74 CR3: 0000000134cbb000 CR4: 00000000000026f0
[340700.513752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[340700.513754] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[340700.513757] Process googleearth-bin (pid: 26271, threadinfo ffff880063b8e000, task ffff88000d7316e0)
[340700.513760] Stack:
[340700.513762] ffff880063b8fdf8 ffffffff81100537 000000000000669f ffff88001d4ac540
[340700.513765] <0> ffff880063b8ff18 ffffffffa008435a 0000000000000067 000000000000e200
[340700.513770] <0> 0000020000000001 ffff88004fafd998 ffffea000116e768 ffffffffa042de70
[340700.513774] Call Trace:
[340700.513783] [<ffffffff81100537>] ? unlock_page+0x27/0x30
[340700.513798] [<ffffffffa008435a>] drm_ioctl+0x34a/0x4c0 [drm]
[340700.513811] [<ffffffffa0384c40>] ? radeon_info_ioctl+0x0/0x150 [radeon]
[340700.513816] [<ffffffff8111f7a9>] ? handle_mm_fault+0x1b9/0x440
[340700.513832] [<ffffffffa0406c84>] radeon_kms_compat_ioctl+0x14/0x30 [radeon]
[340700.513837] [<ffffffff811993f3>] compat_sys_ioctl+0xb3/0x270
[340700.513842] [<ffffffff810450ef>] sysenter_dispatch+0x7/0x2e
[340700.513844] Code: 1f 44 00 00 e8 a2 fe fd e0 c9 c3 55 48 89 e5 53 48 83 ec 18 0f 1f 44 00 00 48 8b 5e 08 48 8b 87 98 03 00 00 48 c7 c1 71 34 42 a0 <44> 8b 0b 44 89 4d ec 44 8b 06 41 83 f8 05 76 30 31 c0 48 c7 c2
[340700.513878] RIP [<ffffffffa0384c60>] radeon_info_ioctl+0x20/0x150 [radeon]
[340700.513892] RSP <ffff880063b8fdd8>
[340700.513893] CR2: fffffffff0ea1e74
[340700.513897] ---[ end trace 13c86cc5b7c36a0e ]---
[340715.018139] i2c i2c-2: sendbytes: NAK bailout.
[340725.048712] i2c i2c-2: sendbytes: NAK bailout.
[340735.068815] i2c i2c-2: sendbytes: NAK bailout.
[...]
$sudo lspci
00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c)
00:01.0 PCI bridge: Intel Corporation Mobile PM965/GM965/GL960 PCI Express Root Port (rev 0c)
00:19.0 Ethernet controller: Intel Corporation 82562GT 10/100 Network Connection (rev 03)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 03)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03)
00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3)
00:1f.0 ISA bridge: Intel Corporation 82801HEM (ICH8M) LPC Interface Controller (rev 03)
00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 03)
00:1f.2 SATA controller: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA AHCI Controller (rev 03)
01:00.0 VGA compatible controller: ATI Technologies Inc RV516 [Mobility Radeon X1350]
10:00.0 Network controller: Broadcom Corporation BCM4312 802.11a/b/g (rev 02)

ProblemType: KernelOops
DistroRelease: Ubuntu 10.10
Package: linux-image-2.6.35-7-generic 2.6.35-7.11
Regression: Yes
Reproducible: No
ProcVersionSignature: Ubuntu 2.6.35-7.11-generic 2.6.35-rc4
Uname: Linux 2.6.35-7-generic x86_64
NonfreeKernelModules: wl
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
Annotation: Your system might become unstable now and might need to be restarted.
AplayDevices:
 **** List of PLAYBACK Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
   Subdevices: 0/1
   Subdevice #0: subdevice #0
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: doru 2003 F.... pulseaudio
 /dev/snd/pcmC0D0p: doru 2003 F...m pulseaudio
 /dev/snd/seq: timidity 1767 F.... timidity
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xdc524000 irq 47'
   Mixer name : 'Analog Devices AD1981'
   Components : 'HDA:11d41981,103c30d7,00100200 HDA:11c11040,103c1378,00100200'
   Controls : 12
   Simple ctrls : 10
Date: Thu Jul 15 22:55:58 2010
Failure: oops
Frequency: This has only happened once.
HibernationDevice: RESUME=UUID=fdfec0ac-31df-4c7b-be67-be4816d0d9f2
MachineType: Hewlett-Packard HP Compaq 6820s
ProcCmdLine: root=UUID=ccd09fa2-cd27-4467-9d25-bba4ed58777c ro quiet splash radeon.new_pll=0
RelatedPackageVersions: linux-firmware 1.37
SourcePackage: linux
Title: [340700.513619] BUG: unable to handle kernel paging request at fffffffff0ea1e74
WpaSupplicantLog:

dmi.bios.date: 01/11/2008
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: 68MDD Ver. F.09
dmi.board.name: 30D7
dmi.board.vendor: Hewlett-Packard
dmi.board.version: KBC Version 83.12
dmi.chassis.asset.tag: CNU81002RB
dmi.chassis.type: 10
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvr68MDDVer.F.09:bd01/11/2008:svnHewlett-Packard:pnHPCompaq6820s:pvrF.09:rvnHewlett-Packard:rn30D7:rvrKBCVersion83.12:cvnHewlett-Packard:ct10:cvr:
dmi.product.name: HP Compaq 6820s
dmi.product.version: F.09
dmi.sys.vendor: Hewlett-Packard

[Dave Gilbert (ubuntu-treblig)]

I've got the same backtrace on (a slightly modified) 2.6.35-8

[ 2119.731044] BUG: unable to handle kernel paging request at ffffffffed4f9294
[ 2119.731049] IP: [<ffffffffa0134e50>] radeon_info_ioctl+0x20/0x150 [radeon]
[ 2119.731064] PGD 1a2c067 PUD 1a2d067 PMD 0
[ 2119.731067] Oops: 0000 [#1] SMP
[ 2119.731069] last sysfs file: /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq
[ 2119.731071] CPU 0
[ 2119.731072] Modules linked in: snd_seq_dummy nls_iso8859_1 nls_cp437 vfat fat ip6table_filter ip6_tables binfmt_misc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm dm_crypt snd_hda_codec_atihdmi snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_seq_midi snd_pcm snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device snd soundcore snd_page_alloc ppdev parport_pc lp parport vga16fb vgastate radeon ttm firewire_ohci usbhid firewire_core r8169 drm_kms_helper hid usb_storage crc_itu_t mii drm pata_via i2c_algo_bit configfs
[ 2119.731098]
[ 2119.731100] Pid: 3703, comm: npviewer.bin Not tainted 2.6.35-8-generic #13daveg P55M Pro/To Be Filled By O.E.M.
[ 2119.731102] RIP: 0010:[<ffffffffa0134e50>] [<ffffffffa0134e50>] radeon_info_ioctl+0x20/0x150 [radeon]
[ 2119.731112] RSP: 0018:ffff88017c099dd8 EFLAGS: 00010296
[ 2119.731113] RAX: ffff880228278000 RBX: ffffffffed4f9294 RCX: ffffffffa01d3811
[ 2119.731115] RDX: ffff88017c3e86c0 RSI: ffff88017c099e58 RDI: ffff880229193800
[ 2119.731116] RBP: ffff88017c099df8 R08: ffffffffa01de210 R09: ffffffffa0134e30
[ 2119.731118] R10: ffff88017c099e58 R11: 0000000000000001 R12: ffff880229193800
[ 2119.731119] R13: 0000000000000067 R14: 00000000c0106467 R15: 0000000000000000
[ 2119.731121] FS: 0000000000000000(0000) GS:ffff880001e00000(0063) knlGS:00000000f69b4760
[ 2119.731123] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
[ 2119.731124] CR2: ffffffffed4f9294 CR3: 000000015e7ec000 CR4: 00000000000006f0
[ 2119.731126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2119.731128] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2119.731129] Process npviewer.bin (pid: 3703, threadinfo ffff88017c098000, task ffff880226e0db80)
[ 2119.731131] Stack:
[ 2119.731132] ffff88017c099df8 ffffffff81100557 0000000000000e77 ffff88017c3e86c0
[ 2119.731134] <0> ffff88017c099f18 ffffffffa002335a 0000000100000067 000000000000e200
[ 2119.731137] <0> 0000020000000001 ffff88016c935a88 ffffea0004fc03a8 ffffffffa01de210
[ 2119.731139] Call Trace:
[ 2119.731145] [<ffffffff81100557>] ? unlock_page+0x27/0x30
[ 2119.731154] [<ffffffffa002335a>] drm_ioctl+0x34a/0x4c0 [drm]
[ 2119.731163] [<ffffffffa0134e30>] ? radeon_info_ioctl+0x0/0x150 [radeon]
[ 2119.731167] [<ffffffff8111f7b9>] ? handle_mm_fault+0x1b9/0x440
[ 2119.731179] [<ffffffffa01b6e84>] radeon_kms_compat_ioctl+0x14/0x30 [radeon]
[ 2119.731182] [<ffffffff81199363>] compat_sys_ioctl+0xb3/0x270
[ 2119.731185] [<ffffffff8104510f>] sysenter_dispatch+0x7/0x2e
[ 2119.731186] Code: 1f 44 00 00 e8 72 fc 22 e1 c9 c3 55 48 89 e5 53 48 83 ec 18 0f 1f 44 00 00 48 8...

[Dave Gilbert (ubuntu-treblig)]

Still occurs on 2.6.35-12-generic (64bit) - fully repeatable with google-earth.
Running with drm.debug=0x07 I see:

Aug 1 01:22:11 major kernel: [ 68.691776] [drm:drm_stub_open],
Aug 1 01:22:11 major kernel: [ 68.691779] [drm:drm_open_helper], pid = 2227, minor = 0
Aug 1 01:22:11 major kernel: [ 68.691791] [drm:drm_ioctl], pid=2227, cmd=0x80046402, nr=0x02, dev 0xe200, auth=0
Aug 1 01:22:11 major kernel: [ 68.691795] [drm:drm_add_magic], 1
Aug 1 01:22:11 major kernel: [ 68.691796] [drm:drm_getmagic], 1
Aug 1 01:22:11 major kernel: [ 68.691826] [drm:drm_ioctl], pid=1312, cmd=0x40046411, nr=0x11, dev 0xe200, auth=1
Aug 1 01:22:11 major kernel: [ 68.691830] [drm:drm_authmagic], 1
Aug 1 01:22:11 major kernel: [ 68.691831] [drm:drm_remove_magic], 1
Aug 1 01:22:11 major kernel: [ 68.692502] [drm:drm_ioctl], pid=2227, cmd=0xc0406400, nr=0x00, dev 0xe200, auth=1
Aug 1 01:22:11 major kernel: [ 68.692507] [drm:drm_ioctl], pid=2227, cmd=0xc0406400, nr=0x00, dev 0xe200, auth=1
Aug 1 01:22:11 major kernel: [ 68.694756] [drm:drm_ioctl], pid=2227, cmd=0xc0106467, nr=0x67, dev 0xe200, auth=1
Aug 1 01:22:11 major kernel: [ 68.844096] [drm:drm_ioctl], pid=2227, cmd=0xc0106467, nr=0x67, dev 0xe200, auth=1
Aug 1 01:22:11 major kernel: [ 68.844109] BUG: unable to handle kernel paging request at ffffffffec8f9294
Aug 1 01:22:11 major kernel: [ 68.844112] IP: [<ffffffffa0141e50>] radeon_info_ioctl+0x20/0x150 [radeon]
Aug 1 01:22:11 major kernel: [ 68.844126] PGD 1a2c067 PUD 1a2d067 PMD 0
Aug 1 01:22:11 major kernel: [ 68.844128] Oops: 0000 [#1] SMP
Aug 1 01:22:11 major kernel: [ 68.844130] last sysfs file: /sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq
Aug 1 01:22:11 major kernel: [ 68.844132] CPU 7
Aug 1 01:22:11 major kernel: [ 68.844133] Modules linked in: ip6table_filter ip6_tables dm_crypt binfmt_misc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm snd_hda_codec_atihdmi snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device ppdev snd parport_pc soundcore snd_page_alloc lp parport radeon usbhid ttm r8169 firewire_ohci hid usb_storage mii firewire_core drm_kms_helper pata_via drm crc_itu_t i2c_algo_bit configfs
Aug 1 01:22:11 major kernel: [ 68.844156]
Aug 1 01:22:11 major kernel: [ 68.844159] Pid: 2227, comm: googleearth-bin Not tainted 2.6.35-12-generic #17-Ubuntu P55M Pro/To Be Filled By O.E.M.
Aug 1 01:22:11 major kernel: [ 68.844161] RIP: 0010:[<ffffffffa0141e50>] [<ffffffffa0141e50>] radeon_info_ioctl+0x20/0x150 [radeon]
Aug 1 01:22:11 major kernel: [ 68.844170] RSP: 0018:ffff88021a3a5dd8 EFLAGS: 00210296
Aug 1 01:22:11 major kernel: [ 68.844171] RAX: ffff88022abfa000 RBX: ffffffffec8f9294 RCX: ffffffffa01e0811
Aug 1 01:22:11 major kernel: [ 68.844173] RDX: ffff88020b1c4d80 RSI: ffff88021a3a5e58 RDI: ffff8802292f0000
Aug 1 01:22:11 major kernel: [ 68.844174] RBP: ffff88021a3a5df8 R08: ffffffffa01eb210 R09: ffffffffa0141e30
Aug 1 01:22:11 major kernel: [ 68.844176] R...

[Freedesktop-treblig (freedesktop-treblig)]

2.6.35 kernel as of git/ubuntu 2.6.35-13.18

In radeon_kms.c there is :

int radeon_info_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
{
        struct radeon_device *rdev = dev->dev_private;
        struct drm_radeon_info *info;
        struct radeon_mode_info *minfo = &rdev->mode_info;
        uint32_t *value_ptr;
        uint32_t value;
        struct drm_crtc *crtc;
        int i, found;

        info = data;
        value_ptr = (uint32_t *)((unsigned long)info->value);
        value = *value_ptr;

I think that *value_ptr should be done with a DRM_COPY_FROM_USER since I'm managing to trigger an oops from it.
(see ubuntu bug 606081)

Dave

[Freedesktop-treblig (freedesktop-treblig)]

Created an attachment (id=37502)
Kernel patch to add copy from user

Seems to make Google Earth 5.1.3535.3218 work for me.

[Dave Gilbert (ubuntu-treblig)]

I think there are two separate problems here; the 1st is that radeon_info_ioctl is missing a copy_from_user
(which I've reported as freedesktop.org bug 29340 - and linked it). That's why it oops's

The 2nd problem I'm less sure of but it looks like a question of what a 32bit app is supposed to put into the ioctl data; I added
some printf's to radeon_info_ioctl and saw this:

[ 127.105635] radeon_info_ioctl: Dev=ffff88022afa5800 data=ffff88020e1d3e58 filp=ffff88020f5a8540
[ 127.105639] radeon_info_ioctl: rdev=ffff88022b164000
[ 127.105640] radeon_info_ioctl: value_ptr=00000000ffc6ac2c
[ 127.105641] radeon_info_ioctl: value=0
[ 127.105642] radeon_info_ioctl: info->request=0

[ 127.172280] radeon_info_ioctl: Dev=ffff88022afa5800 data=ffff88020e1d3e58 filp=ffff88020f5a8540
[ 127.172284] radeon_info_ioctl: rdev=ffff88022b164000
[ 127.172286] radeon_info_ioctl: value_ptr=ffffffffecc17ab4
[ 127.172298] BUG: unable to handle kernel paging request at ffffffffecc17ab4

and the data structure passed to the ioctl apparently is:

struct drm_radeon_info {
        uint32_t request;
        uint32_t pad;
        uint64_t value;
};

Is that correct for both 32 and 64bit processes? If so then it looks like googleearth is sign extending it's pointer when
it stuffs it into value which would be a googleearth bug; however if we're expecting a 32bit app only to know about
the bottom 32bits then we need a compat routine to clean value up.

Dave

[Dave Gilbert (ubuntu-treblig)]

The attached patch clears this Oops for me, and gets google earth 5.1.3535.3218 working; however it doesn't
actually address any problems that may be in any 32 bit compat code so maybe there are others needed.
(also posted to the FDO bugzilla).

[Airlied-freedesktop (airlied-freedesktop)]

I've queued this fix to Linus, thanks for figuring what was broken.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.35-15.21

---------------
linux (2.6.35-15.21) maverick; urgency=low

  [ Luke Yelavich ]

  * [Config] CONFIG_SND_USB_UA101=m for all architectures

  [ Upstream Kernel Changes ]

  * Input: introduce MT event slots
  * Input: document the MT event slot protocol
  * (pre-stable) sched: Revert nohz_ratelimit() for now
  * (pre-stable) drm/radeon/kms: add missing copy from user
    - LP: #606081

  [ Leann Ogasawara ]

  * rebase to v2.6.35.1
 -- Leann Ogasawara <email address hidden> Mon, 09 Aug 2010 09:24:04 -0700

[Michael (michael-luetz)]

>>> www.cheap-itemsmed.ru

- There is no reason to feel depressed if you, like so many other men today, have hit the rough spot of your sexual life when you cannot seem to be performing as well as you used to do. Your agitation and frustration are easily understood, and still you should know that with the modern development of worldwide pharmacological industries it is but natural that there is bound to be a solution for your own needs when it comes down to solving your erectile dysfunction problems.

- We suggest you getting erectile dysfunction drugs from our online drugstore cheaply, easily and without much to-do. You will always be taken care of and attended to with extreme care, and you will have the possibility of choosing from a wide range of erectile dysfunction products to find a solution that will best suit your particular needs.

- The whole world puts great trust in those famous diamond shaped pills, and if you are not an exception we would like to revise some basic points that speak in favor of putting your male health in the hands of those magical tablets.
Firstly, there is no need for regular therapy – those blue pills have to be taken directly before the planned sexual contact, and extra convenience shines through when you come to realize that the pills will not start working unless sufficient sexual stimulation is provided.

- Secondly, this erectile dysfunction solution is known for its extremely low adverse side effects profile, which makes it available for all age groups of patients. Besides, it really does not matter how old you are when it goes about taking erectile dysfunction pills – just make sure that you do not overdose, and they will work for you at their best!

- Finally, those fabulous pills do not only provide you with stable erection that can be maintained for as long as you need it – they also help you to achieve unheard sensitivity during the sexual act as well as make your genitals look much more sizey!

- Think of all those benefits of taking erectile dysfunction pills and the effect you can produce on your loved one – and do not hesitate to make your order and / or refills right away!

--- > www.cheap-itemsmed.ru < ---

 _V______C________L
 ___I_______I_______E
 _____A_______A_______V
 _______G_______L_______I
 _________R_______I_______T
 ___________A_______S_______R
 _____________________________A

   Willing to make sweet love to her more, but having troubles with stamina and libido? Our products will instantly make your body as energetic as your soul is! Find more about our pilules and fantastic prices on them! You can get your package delivered confidentially!
-
 - - -
--------- www.cheap-itemsmed.ru
 - - -
-

[Oxy (alessio-boidi)]

Be in good health Customer
Thank you for subm zca itting your information for potential empl icq oyment opport bbt unities.
We look forward to reviewing your appl lzh ication,
but can not do so until you comp cvs lete our internal application.

The pay range for available posi lgz tions range from $35.77 per hour to $57.62 per hour.
Prior to begin able to be consi ow dered, you will first need you to formally apply.
Please go here to be kma gin the process:

www.shrinkurl.ru/ca6ed4

Also, the following perks are pote ywq ntially ava cv ilable:

 - Pa pp id Time Off
 - Hea ur lth Benefits Package
 - Hig fq her than average salaries
 - Tuition Reim rb bursement
 - Exte zkr nsive 401(k)program

Please take the time to follow the dire nj ctions and complete the entire application pro msb cess.

---------------------------------------------------------------

Best Reg mya ards,

Rock Force Mana fc gement