misc/159918: inter-jail communication failure

[markham breitbach]

>Number:         159918
>Category:       misc
>Synopsis:       inter-jail communication failure
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Aug 19 16:10:10 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     markham breitbach
>Release:        FreeBSD 8.1-RELEASE
>Organization:
>Environment:
FreeBSD kimmirut-pd1.qiniq.com 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:55:53 UTC 2010     root at almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

>Description:
We run a system with two jails operating different services in very remote locations.  Each jail is assigned a unique routable IP address.

Intermittently, we lose power unexpectedly and backup batteries drain before we can get the servers shut down cleanly.

Ocassionally after the servers come back up, we cannot connect to the services provided by jail2 from jail1.


The jails are quite minimal, containing the files necessary to run the services and very little else.  Any suggestions to further diagnose this problem would greatly be appreciated.
>How-To-Repeat:
We cannot repeat this problem on the bench, but his happens in the field every few months.
>Fix:
The solution to the problem is not always consistent.

Sometimes the solution is as simple as flushing the arp cache.

Sometimes the dance is a bit more complicated and involves unaliasing the addresses from the interface, flushing the route and arp caches and re-aliasing the interface IP. (and restarting the services)



>Release-Note:
>Audit-Trail:
>Unformatted: