When you say the "connection drops", do you mean that the NIC loses link and then has it comes back? Are you using DHCP to get the main address? It sounds like perhaps dhclient is running and clearing the addresses it doesn't know about.