[svlug] ssh tunnels and Virtual Machines

Daevid Vincent daevid at daevid.com
Mon Jan 25 17:57:46 PST 2010


I am a huge fan of virtual machines for development work (LAMP). However,
our mysql DB is about 100GB and growing rapidly daily. I use a VM for the
benefits of samba/sshfs/ssh/cvs/etc. and it has it's own local DB skeleton,
but sometimes I have to point it at the big momma to make sure the data is
right.

Anyways, due to security "issues" (which are debatable), my VM can't get to
the mysql DBs as I'm on a 10.139.x.x network and they only allow a
connection from 10.10.10.x network. This can't/isn't going to change much
to my chagrin.

Soooo, enter the beauty of ssh tunnels. I setup one with a very simple
script:

developer at mypse ~ $ cat pse_mount
---------------------------------------------------------------------------
-----------
#!/bin/bash

#usually this is $USER, but in this VM, everyone is 'developer' so it won't
work.
SSHACCOUNT=vincentd

#echo "mounting pse02 and pse01 sshfs drives..."
#sshfs $SSHACCOUNT at pse01:/home/$SSHACCOUNT /home/$SSHACCOUNT/pse01 -o
cache=no
#sshfs $SSHACCOUNT at pse02:/var/www/$SSHACCOUNT /home/$SSHACCOUNT/pse02 -o
cache=no

echo "Setting up mysql tsocks and ssh tunnels to 10.10.10.0/24 network..."

ssh -CfND 1080 $SSHACCOUNT at pse01
# to test it, use this:
#   tsocks mysql -uUSERNAME -pPASSWORD -h10.10.10.41 -P3306 agis_core

ssh $SSHACCOUNT at pse01 -L5541:10.10.10.41:3306 -fN
ssh $SSHACCOUNT at pse02 -L5542:10.10.10.42:3306 -fN
# to test it, use this:
#   mysql -h127.0.0.1 -P5541 -uUSERNAME -pPASSWORD agis_core
#   mysql -h127.0.0.1 -P5542 -uUSERNAME -pPASSWORD agis_core

exit 0
---------------------------------------------------------------------------
-----------

Note the -fN option which is supposed to keep the tunnel persistent (i.e.
back grounded) as otherwise if you close the shell, the tunnel collapses
with it.

This works supercalifragilisticexpialidocious until I "pause" the
Virtualbox VM.
(note that 'pause' is NOT the same as "suspend" or "hibernate" -- as Ubuntu
has no idea it just went 'away'. Virtualbox is just freezing the state to
disk, therefore Ubuntu never gets any S4 signals or whatever they are.)

So what happens is, the next time I un-pause, the VM is exactly where I
left it, EXCEPT that the tunnel has gone away. This also leaves all these
ghosts around...

developer at mypse ~ $ netstat | grep 554
tcp        1      0 localhost:50633         localhost:5542
CLOSE_WAIT
tcp        1      0 localhost:33829         localhost:5541
CLOSE_WAIT
tcp        1      0 localhost:36756         localhost:5542
CLOSE_WAIT
tcp        1      0 localhost:59284         localhost:5541
CLOSE_WAIT

developer at mypse ~ $ lsof -i tcp:5541
developer at mypse ~ $ lsof -i tcp:5542

Note how it appears that nothing is using these ports.

I have to manually start it again.

developer at mypse ~ $ ssh vincentd at pse02 -L5542:10.10.10.42:3306 -fN
developer at mypse ~ $ netstat | grep 554
tcp        1      0 localhost:50633         localhost:5542
CLOSE_WAIT
tcp        1      0 localhost:33829         localhost:5541
CLOSE_WAIT
tcp        0      0 localhost:43945         localhost:5541
ESTABLISHED
tcp        0      0 localhost:33604         localhost:5542
ESTABLISHED
tcp        1      0 localhost:36756         localhost:5542
CLOSE_WAIT
tcp        1      0 localhost:59284         localhost:5541
CLOSE_WAIT

developer at mypse ~ $ lsof -i tcp:5542
COMMAND  PID      USER   FD   TYPE DEVICE SIZE NODE NAME
ssh     4793 developer    4u  IPv4 141026       TCP localhost:5542 (LISTEN)
ssh     4793 developer    5u  IPv6 141027       TCP ip6-localhost:5542
(LISTEN)
ssh     4793 developer    6u  IPv4 141054       TCP
localhost:5542->localhost:33611 (ESTABLISHED)

What I fear is that as I pause this more and more (like on the weekends
when I shut down my PC), this will just keep growing. And actually
'rebooting' the ubuntu VM is also not convenient and sort of cripples one
of the most useful features of a VM -- the speed to start.

Is there a way to
[a] force the CLOSE_WAIT guys to go away. They don't seem to ever go away.
I've checked throughout the day today and they're still there always.
[b] force ssh to just re-use the tunnel I had previously set up? Like
re-start it?
[c] I'd prefer not to /etc/init.d/networking restart as that will kill my
putty shell and any other connections I might have going on.





More information about the svlug mailing list