Cannot start vncserver in PPC Ubuntu 14.04 LE

[bugproxy]

== Comment: #0 - Praneeth R. Pulusani <email address hidden> - 2014-07-14 17:27:30 ==
---Problem Description---
Cannot start a vncserver session
This bug is a clone of bug 112779

Contact Information = Praneeth <email address hidden> Backup: Doug <email address hidden>

---uname output---
Linux schwartz 3.13.0-30-generic #54-Ubuntu SMP Mon Jun 9 22:46:02 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = Server-8284-22A-SN10334BV

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 Install tightvncserver using sudo apt-get install
Launch vncserver using "vncserver :1"
It fails to launch

Userspace tool common name: tightvncserver

The userspace tool has the following bit modes: 64bit

Userspace rpm: tightvncserver_1.3.9-6.4ubuntu1_ppc64el.deb

Userspace tool obtained from project website: na

*Additional Instructions for Praneeth <email address hidden> Backup: Doug <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach ltrace and strace of userspace application.

Here is the console output:
root@schwartz:/home/guest#
root@schwartz:/home/guest#
root@schwartz:/home/guest#
root@schwartz:/home/guest# vncserver -kill :1

Can't find file /root/.vnc/schwartz:1.pid
You'll have to kill the Xtightvnc process manually

root@schwartz:/home/guest# ps -a | grep vnc
root@schwartz:/home/guest# ps -aux | grep vnc
root 29524 0.0 0.0 6656 2944 pts/1 S+ 10:24 0:00 grep --color=auto vnc
root@schwartz:/home/guest# cd ~/.vnc/
root@schwartz:~/.vnc# ls -a
. .. passwd schwartz:1.log
root@schwartz:~/.vnc# rm schwartz\:1.log
root@schwartz:~/.vnc# cd ..
root@schwartz:~# vncserver :1

Warning: schwartz:1 is taken because of /tmp/.X1-lock
Remove this file if there is no X server schwartz:1
A VNC server is already running as :1
root@schwartz:~# rm /tmp/.X1-lock
root@schwartz:~# vncserver :1

Warning: schwartz:1 is taken because of /tmp/.X11-unix/X1
Remove this file if there is no X server schwartz:1
A VNC server is already running as :1
root@schwartz:~# rm /tmp/.X11-unix/X1
root@schwartz:~# ps -aux | grep vnc
root 29538 0.0 0.0 6656 2944 pts/1 S+ 10:35 0:00 grep --color=auto vnc
root@schwartz:~# vncserver :1
*** Error in `Xtightvnc': free(): invalid next size (fast): 0x0000010033cd6760 ***
Couldn't start Xtightvnc; trying default font path.
Please set correct fontPath in the vncserver script.
*** Error in `Xtightvnc': free(): invalid next size (fast): 0x00000100377b8280 ***
Couldn't start Xtightvnc process.

30/06/14 10:35:39 Xvnc version TightVNC-1.3.9
30/06/14 10:35:39 Copyright (C) 2000-2007 TightVNC Group
30/06/14 10:35:39 Copyright (C) 1999 AT&T Laboratories Cambridge
30/06/14 10:35:39 All Rights Reserved.
30/06/14 10:35:39 See http://www.tightvnc.com/ for information on TightVNC
30/06/14 10:35:39 Desktop name 'X' (schwartz:1)
30/06/14 10:35:39 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
30/06/14 10:35:39 Listening for VNC connections on TCP port 5901
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring
30/06/14 10:35:40 Xvnc version TightVNC-1.3.9
30/06/14 10:35:40 Copyright (C) 2000-2007 TightVNC Group
30/06/14 10:35:40 Copyright (C) 1999 AT&T Laboratories Cambridge
30/06/14 10:35:40 All Rights Reserved.
30/06/14 10:35:40 See http://www.tightvnc.com/ for information on TightVNC
30/06/14 10:35:40 Desktop name 'X' (schwartz:1)
30/06/14 10:35:40 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
30/06/14 10:35:40 Listening for VNC connections on TCP port 5901
Font directory '/usr/share/fonts/X11/Speedo/' not found - ignoring
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring

root@schwartz:~#

----------------------------------------------
~/.vnc/schwartz:1.log contents
30/06/14 10:35:39 Xvnc version TightVNC-1.3.9
30/06/14 10:35:39 Copyright (C) 2000-2007 TightVNC Group
30/06/14 10:35:39 Copyright (C) 1999 AT&T Laboratories Cambridge
30/06/14 10:35:39 All Rights Reserved.
30/06/14 10:35:39 See http://www.tightvnc.com/ for information on TightVNC
30/06/14 10:35:39 Desktop name 'X' (schwartz:1)
30/06/14 10:35:39 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
30/06/14 10:35:39 Listening for VNC connections on TCP port 5901
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring
30/06/14 10:35:40 Xvnc version TightVNC-1.3.9
30/06/14 10:35:40 Copyright (C) 2000-2007 TightVNC Group
30/06/14 10:35:40 Copyright (C) 1999 AT&T Laboratories Cambridge
30/06/14 10:35:40 All Rights Reserved.
30/06/14 10:35:40 See http://www.tightvnc.com/ for information on TightVNC
30/06/14 10:35:40 Desktop name 'X' (schwartz:1)
30/06/14 10:35:40 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
30/06/14 10:35:40 Listening for VNC connections on TCP port 5901
Font directory '/usr/share/fonts/X11/Speedo/' not found - ignoring
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring

== Comment: #2 - David Heller <email address hidden> - 2014-07-14 18:11:15 ==
(In reply to comment #0)
> ---Problem Description---
> Cannot start a vncserver session
> This bug is a clone of bug 112779
>
I asked the submitter to do this due to multiple mirroring issues with the previous bug.

== Comment: #3 - Praneeth R. Pulusani <email address hidden> - 2014-07-14 18:30:10 ==
Also, different display number or other command line options appear to make no difference. Same error from vncserver.

root@schwartz:/home/guest# vncserver :2 -geometry 1280x1024 -depth 16 -alwaysshared
*** Error in `Xtightvnc': free(): invalid next size (fast): 0x0000010031fd0aa0 ***
Couldn't start Xtightvnc; trying default font path.
Please set correct fontPath in the vncserver script.
*** Error in `Xtightvnc': free(): invalid next size (fast): 0x0000010025c681f0 ***
Couldn't start Xtightvnc process.

14/07/14 17:20:31 Xvnc version TightVNC-1.3.9
14/07/14 17:20:31 Copyright (C) 2000-2007 TightVNC Group
14/07/14 17:20:31 Copyright (C) 1999 AT&T Laboratories Cambridge
14/07/14 17:20:31 All Rights Reserved.
14/07/14 17:20:31 See http://www.tightvnc.com/ for information on TightVNC
14/07/14 17:20:31 Desktop name 'X' (schwartz:2)
14/07/14 17:20:31 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
14/07/14 17:20:31 Listening for VNC connections on TCP port 5902
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring
14/07/14 17:20:32 Xvnc version TightVNC-1.3.9
14/07/14 17:20:32 Copyright (C) 2000-2007 TightVNC Group
14/07/14 17:20:32 Copyright (C) 1999 AT&T Laboratories Cambridge
14/07/14 17:20:32 All Rights Reserved.
14/07/14 17:20:32 See http://www.tightvnc.com/ for information on TightVNC
14/07/14 17:20:32 Desktop name 'X' (schwartz:2)
14/07/14 17:20:32 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
14/07/14 17:20:32 Listening for VNC connections on TCP port 5902
Font directory '/usr/share/fonts/X11/Speedo/' not found - ignoring
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-07-22 06:03 EDT-------
*** Bug 113610 has been marked as a duplicate of this bug. ***

[Dave Heller (hellerda)]

Please give this a higher priority and provide some target for resolution. Thx.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-07-29 13:54 EDT-------
There are 3 major problems with VNC on Ubuntu LE.

First, when compiling with gcc you have to use a compiler that contains a fix for gcc bug 61300

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61300

Second, there is a storage allocation error in Xvnc/programs/Xserver/dix/pixmap.c when it does not allocate enough storage.

129c129
< pPixmap = (PixmapPtr)xalloc(pScreen->totalPixmapSize + pixDataSize);
---
> pPixmap = (PixmapPtr)xalloc(pScreen->totalPixmapSize + pixDataSize + sizeof(DevUnion));

The above two things will get vncserver to run if it is compiled as a 32 bit program. There is one additional change if you need a 64 bit vncserver. The header file Xvnc/include/Xmd.h does not properly detect a 64 bit compilation and uses the wrong typedefs/defines

55c55
<
---
> #include <limits.h>
62c62
< #if defined(__alpha) || defined(__alpha__) || defined(__x86_64__)
---
> #if defined(__alpha) || defined(__alpha__) || defined(__x86_64__) || ((defined(__WORDSIZE)) && (__WORDSIZE == 64))

After getting the server to run I ran into problems getting any window managers to start properly (I tried several). I did not determine if that was from further problems in vncserver or in the window managers.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-17 18:24 EDT-------
*** Bug 116146 has been marked as a duplicate of this bug. ***

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-17 19:12 EDT-------
*** Bug 116154 has been marked as a duplicate of this bug. ***

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-10-17 08:43 EDT-------
Installed linuxvnc server on ppc64el,
$ sudo /usr/bin/linuxvnc
[sudo] password for ubuntu:
17/10/2014 04:37:04 Using device /dev/tty2.
17/10/2014 04:37:04 Listening for VNC connections on TCP port 5900
17/10/2014 04:37:04 Listening for VNC connections on TCP6 port 5900
# then started VNC client
[fauck@poc1 ~]$ vncviewer fauck4

TigerVNC Viewer 64-bit v1.2.80 (20130608)
Built on Jun 8 2013 at 09:16:58
Copyright (C) 1999-2011 TigerVNC Team and many others (see README.txt)
See http://www.tigervnc.org for information on TigerVNC.

Fri Oct 17 05:38:02 2014
CConn: connected to host fauck4 port 5900
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8

Fri Oct 17 05:38:25 2014
PlatformPixelBuffer: Using default colormap and visual, TrueColor, depth 24.

Fri Oct 17 05:38:27 2014
CConn: Using pixel format depth 24 (32bpp) little-endian rgb888
CConn: Using Tight encoding

==> Server responding correctly
17/10/2014 04:38:02 Got connection from client 172.16.0.1
17/10/2014 04:38:02 other clients:
17/10/2014 04:38:02 Normal socket connection
17/10/2014 04:38:02 Client Protocol Version 3.8
17/10/2014 04:38:02 Protocol version sent 3.8, using 3.8
17/10/2014 04:38:02 rfbProcessClientSecurityType: executing handler for type 1
17/10/2014 04:38:02 rfbProcessClientSecurityType: returning securityResult for client rfb version >= 3.8
17/10/2014 04:38:27 Pixel format for client 172.16.0.1:
17/10/2014 04:38:27 32 bpp, depth 24, little endian
17/10/2014 04:38:27 true colour: max r 255 g 255 b 255, shift r 16 g 8 b 0
17/10/2014 04:38:27 Enabling full-color cursor updates for client 172.16.0.1
17/10/2014 04:38:27 Enabling NewFBSize protocol extension for client 172.16.0.1
17/10/2014 04:38:27 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFECC)
17/10/2014 04:38:27 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFECD)
17/10/2014 04:38:27 Enabling LastRect protocol extension for client 172.16.0.1
17/10/2014 04:38:27 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEC7)
17/10/2014 04:38:27 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEC8)
17/10/2014 04:38:27 Using image quality level 8 for client 172.16.0.1
17/10/2014 04:38:27 Using JPEG subsampling 0, Q92 for client 172.16.0.1
17/10/2014 04:38:27 Using tight encoding for client 172.16.0.1

and a window with a prompt appeared
Ubuntu Utopic Unicorn (development branch) fauck4 tty2
fauck4 login:

------- Comment From <email address hidden> 2014-10-17 08:44 EDT-------
Problem is then isolated to thightvncserver which is know to be non functional.
Rejecting bug as the Ubuntu VNC server to be supported is linuxvnc

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Please attach patches in unified diff format where possible. I have extracted the changes described in comment #3 on this bug into a diff locally, and applied them to the version of tighvnc source in Ubuntu 15.04 (version 1.3.10-0ubuntu1). After applying these changes, Xtightvnc still fails with the same error as before:

$ tightvncserver :1
*** Error in `Xtightvnc': free(): invalid next size (fast): 0x00000100231b5170 ***
Couldn't start Xtightvnc; trying default font path.
Please set correct fontPath in the tightvncserver script.
*** Error in `Xtightvnc': free(): invalid next size (fast): 0x00000100041e8240 ***
Couldn't start Xtightvnc process.
[...]

So it doesn't appear that this is a complete fix for a 64-bit build.

The changes to Xvnc/programs/Xserver/dix/pixmap.c don't look correct to me, either. Relevant type declarations:

typedef struct _Pixmap {
    DrawableRec drawable;
    int refcnt;
    int devKind;
    DevUnion devPrivate;
#ifdef PIXPRIV
    DevUnion *devPrivates; /* real devPrivates like gcs & windows */
#endif
} PixmapRec;

(Xvnc/programs/Xserver/include/pixmapstr.h)

typedef struct _Pixmap *PixmapPtr;

(Xvnc/programs/Xserver/include/pixmap.h)

Relevant block of code (after patching):

    PixmapPtr pPixmap;
#ifdef PIXPRIV
[...]
    DevUnion *ppriv;
[...]

    pPixmap = (PixmapPtr)xalloc(pScreen->totalPixmapSize + pixDataSize + sizeof(DevUnion));
    if (!pPixmap)
        return NullPixmap;
    ppriv = (DevUnion *)(pPixmap + 1);
    pPixmap->devPrivates = ppriv;

So pPixmap is supposed to be a pointer to an object of type struct _Pixmap; but it doesn't allocate it as a multiple of sizeof(struct _Pixmap), so what ensures that the further references aren't walking off the end of the allocated memory? In practice, it looks like the answer may be "nothing".

Has this patch been submitted to upstream? As it doesn't fix the bug when applied against 1.3.10, it would be good to see a fix accepted upstream before we try to patch this in Ubuntu.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-05-04 16:50 EDT-------
Steve,

This is a closed bug from IBM side. The developer working on this bug doesn't seem to be working at IBM anymore and I understand that there is no one else working to port VNC to ppc64el.

[Breno Leitão (breno-leitao)]

Reopening this bug, since we would like to have tightvnc on ppc64el arch.

[Lynn (lynnw)]

Does this mean that there is a fix? If so, I would like to try it out.

[Breno Leitão (breno-leitao)]

Not yet, but, we should pursue looking for a fix.

[Curaga (lauri-kasanen)]

Please try the patches at
http://sourceforge.net/p/vnc-tight/bugs/1353/#1406

[Breno Leitão (breno-leitao)]

Also sent Lauri's patch to Debian BTS.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788107

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-06-23 15:30 EDT-------
Hmm, I see the following in the launchpad bug. Not sure how to interpret this but I think we need this for both Wily and Trusty LTS..

bugproxy (bugproxy) on 2015-06-08
tags:
added: targetmilestone-inin1510
removed: targetmilestone-inin1404

This bug has been open for a while and it looks like we finally have a patch upstream so we'd like to get it integrated ASAP. To my understanding there is still no working VNC server for the LE distributions (other than roll-your-own).

I'm trying to figure out how we can tell if the patch has been integrated, My usual trick of grepping thru the Packages.bz2 files on ports.ubuntu.com isn't showing me anything definitive here. I see entries for libvncserver but no "tightvncserver" here. I"m not sure if the update would be reflected in the meta-package or in the libvncserver patckages..?

$ wget -qO- http://ports.ubuntu.com/dists/trusty-updates/main/binary-ppc64el/Packages.bz2 | bunzip2 | grep vnc | grep Filename
Filename: pool/main/libv/libvncserver/libvncserver-config_0.9.9+dfsg-1ubuntu1.1_ppc64el.deb
Filename: pool/main/libv/libvncserver/libvncserver-dev_0.9.9+dfsg-1ubuntu1.1_ppc64el.deb
Filename: pool/main/libv/libvncserver/libvncserver0_0.9.9+dfsg-1ubuntu1.1_ppc64el.deb
Filename: pool/main/libv/libvncserver/libvncserver0-dbg_0.9.9+dfsg-1ubuntu1.1_ppc64el.deb

In any event: Ubuntu, please address this with high priority and let us know if a separate bug is needed for Wily. Thx.

[Steve Langasek (vorlon)]

Why is this a priority for inclusion in trusty? The bug log includes a comment from Thierry that the linuxvnc package works. Is there a reason this is not sufficient for trusty?

[Launchpad Janitor (janitor)]

This bug was fixed in the package tightvnc - 1.3.10-0ubuntu2

---------------
tightvnc (1.3.10-0ubuntu2) wily; urgency=medium

  * debian/patches/0001-Clean-up-the-malloc-conflicts.patch: [PATCH 1/2]
    Clean up the malloc conflicts. Thanks to Lauri Kasanen
    <email address hidden>.
  * debian/patches/0002-Add-ppc64el-support.patch: [PATCH 2/2] Add
    ppc64el support. Thanks to Lauri Kasanen <email address hidden>. Closes
    LP: #1341829.

 -- Steve Langasek <email address hidden> Tue, 30 Jun 2015 14:14:17 -0700

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-07-02 21:46 EDT-------
(In reply to comment #36)
> Why is this a priority for inclusion in trusty? The bug log includes a
> comment from Thierry that the linuxvnc package works. Is there a reason
> this is not sufficient for trusty?

"linuxvnc" is another name for tigervnc? I'm starting to lose track..

In any event, I think folks on this thread who got a working VNC server on Trusty for ppc64el did it by building some package from source. Is there any working VNC server for Trusty that can be installed using apt-get, either from the Server ISO or the upstream repository?

Sorry if I'm missing something, but I remember looking several times before and not finding anything for Trusty except the broken tightvncserver.

[Steve Langasek (vorlon)]

The package is named linuxvnc. It is included in the Ubuntu archive for ppc64el in 14.04.

[Steve Langasek (vorlon)]

it should be noted that the linuxvnc package advertises itself as exporting a Linux virtual terminal over VNC, not an X session. So it's possible that linuxvnc is not fit for purpose, despite having been mentioned in this bug thread.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-07-03 13:28 EDT-------
> it should be noted that the linuxvnc package advertises itself as exporting
> a Linux virtual terminal over VNC, not an X session. So it's possible that
> linuxvnc is not fit for purpose, despite having been mentioned in this bug
> thread.
Right. Linuxvnc works fine, but we need a Graphical interface.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-23 08:59 EDT-------
Closing at our side. Since this is fixed a long time ago.

[Theresa Xu (theresax)]

I see the same problem on my P8 LE with Ubuntu 14.04.

[Breno Leitão (breno-leitao)]

What vnc package version you have?

[Theresa Xu (theresax)]

I got "TightVNC Server version 1.3.9"

[Breno Leitão (breno-leitao)]

The problem was fixed on version 1.3.10, as showed by the changelog:

tightvnc (1.3.10-0ubuntu2) wily; urgency=medium

  * debian/patches/0001-Clean-up-the-malloc-conflicts.patch: [PATCH 1/2]
    Clean up the malloc conflicts. Thanks to Lauri Kasanen
    <email address hidden>.
  * debian/patches/0002-Add-ppc64el-support.patch: [PATCH 2/2] Add
    ppc64el support. Thanks to Lauri Kasanen <email address hidden>. Closes
    LP: #1341829.

I understand that this version never made 14.04 archive, so, you need a newer Ubuntu release to take benefit of this fix.

[Theresa Xu (theresax)]

Which Ubuntu version do I need? I am not sure what 1.3.10 map to. Many thanks!

[Breno Leitão (breno-leitao)]

You want version 'wily' or later. I would recommend 16.04, which is a very stable and LTS release.