headless virtualbox crashing - possible problem with interaction with libvncserver (including patch)

[Gary Palmer]

On Sat, May 12, 2012 at 03:31:38PM -0400, Gary Palmer wrote:
> Hi,
> 
> I have virtualbox-ose-4.1.14 and libvncserver-0.9.9 installed on FreeBSD 9
> and while I could start one VM using VBoxHeadless fine, the 2nd one would
> start but if I tried to attach to the VNC server it would crash with a bus
> error.  The only thing I noticed in the startup messages was
> 
> 12/05/2012 19:01:24 Listening for VNC connections on TCP port 5901
> 12/05/2012 19:01:24 rfbListenOnTCP6Port: error in bind IPv6 socket: Address already in use
> 
> I tracked this down to libvncserver having IPv6 support but virtualbox
> not setting the IPv6 port, just the IPv4 port.  I recompiled with debugging
> symbols and the attached patch and the crashes have stopped.  I'm
> inclined to suspect that the patch stops the crash.  I'm half pondering
> altering the patch a little so that if a port isn't specified it uses the
> autoprobe feature of libvncserver to find a free port rather than blindly
> assuming 5900 all the time.

The patch got stripped, probably for having the wrong MIME type.  Let me
try again.

Gary
-------------- next part --------------
--- src/VBox/Frontends/VBoxHeadless/FramebufferVNC.cpp.orig	2012-05-12 19:50:26.807639833 +0100
+++ src/VBox/Frontends/VBoxHeadless/FramebufferVNC.cpp	2012-05-12 19:50:38.568636058 +0100
@@ -89,7 +89,10 @@
     vncServer = rfbGetScreen(0, NULL, mWidth, mHeight, 8, 3, 1);
     vncServer->screenData = (void*)this;
     if (mVncPort)
+    {
         vncServer->port = mVncPort;
+        vncServer->ipv6port = mVncPort;
+    }
     char *pszDesktopName;
     rc = RTStrAPrintf(&pszDesktopName, "%s - VirtualBox", pszName);
     if (rc >= 0)