[vnet] [epair] epair interface stops working after some time

[Reshad Patuck]

Hi,

I attempted to unload the pf module, but this did not cause any changes.

I am not creating/destroying any VNET jails at the time epais stop to function.
Multiple VNET jails are started when I start the box, but no further activity (starts or stops of vnet jails, creation deletion of epair interfaces, pf start, stop or reload)

I have been monitoring output from the following:
- netstat -ss
- netstat -m
- vmstat -z
- vmstat -m

I will add 'netstat -i' to my battery of monitoring commands.

So far I the only pattern I can see out of the ordinary is the 'vmstat -m' output for epairs.
Where the size seems to keep growing, and at some point, the memory-use and high-use grow too.
The epair interface seems to stop working when the memory-use and high-use grow.
I have also noticed that these parameters stay almost constant on other boxes.

Here is a link (http://dpaste.com/3WB6AD4.txt) to the csv file containing the 'vmstat -m' output for 'epair' over time.
I noticed the epair being to fail at timestamp 2018-01-09T07:56Z, but this test ran every 5 minutes so it could be upto 5 minutes before this timestamp.
NOTE: I have used --libxo on the vmstat to get json output, it seems to have lost the trailing 'K' in the memory-use column.

I will update things here if I find anything else in the logs.

Please let me know if there is anything else I should look at, or if there is any other output you would like.

Best regards,

Reshad

On Thursday 11 January 2018 2:20:06 AM IST Kristof Provost wrote:
> On 5 Jan 2018, at 20:54, Reshad Patuck wrote:
> > I have done the following on both servers to test what happens:
> > - Created a new epair interface epair3a and epair3b
> > - upped both interfaces
> > - given epair3a IP address 10.20.30.40/24 (I don't have this subnet
> > anywhere in my network)
> > - attempted to ping 10.20.30.50
> > - checked for any packets on epair3b
> > On the server where epairs are working, I can see APR packets for
> > 10.20.30.50, but on the server where epairs are not working I cant see 
> > any
> > packets on epair3b.
> > I can however see the arp packets on epair3a on both servers.
> >
> So epair3a was not added to the bridge and epair3b was not added to a 
> jail?
> That’s interesting, because it should mean the problem is not with the 
> bridge or jail.
> As it affects ARP packets it also shouldn’t be a pf problem.
> It might be worth unloading the pf module, just to re-confirm, but I 
> wouldn’t expect it to make a difference.
> 
> > Please let me know if there is anything I can do the debug this issue 
> > or if
> > you need any other information.
> >
> Are you creating/destroying vnet jails at any point? Is there a 
> correlation with that and the start of the epair issues?
> 
> Are there any errors in `netstat -s` or `netstat -i epair3a` ?
> 
> Regards,
> Kristof