possible disruption of dovecot traffic on 1 of 2 freebsd hosts

[Julian H. Stacey]

Hi freebsd-net at freebsd.org
I'm seeing strange net behaviour with POP3 on 1 of 2 servers,
I would appreciate advice please, perhaps name of net tools to test with ? 

I have 1 local client POP3 fed from 2 remote servers:

land.berklix.org 
 a jail under another FreeBSD.
 FreeBSD land.berklix.org 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4
 #0: Sat May 28 12:23:44 UTC 2016
 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

slim.berklix.org 
 under vmware, dont know whats outside
 FreeBSD slim.berklix.org 10.3-STABLE FreeBSD 10.3-STABLE #0: Tue
 Aug 16 18:09:22 CEST 2016
 jhs at slim.berklix.org:/usr/obj/usr/src/sys/GENERIC  amd64

slim also has /etc/rc.conf
  vmware_guest_vmblock_enable="YES"
  vmware_guest_vmhgfs_enable="YES"
  vmware_guest_vmmemctl_enable="YES"      # ports/emulators/open-vm-tools-nox11
  vmware_guest_vmxnet_enable="YES"        # ports/emulators/open-vm-tools-nox11
  vmware_guestd_enable="YES"              # ports/emulators/open-vm-tools-nox11

Both land & slim run POP3 servers:
  /usr/ports/mail/dovecot # pkg info | grep dovecot # dovecot-1.2.17_6
  All src/ & ports/ self compiled (though not kernels of prison & vmware).

host land is reliable I can always succeed with
	fetchmail land.berklix.org

host slim I can fetchmail for a while,
	then it locks up & periodically
	& I must manualy mv & sftp /var/mail/jhs

The receiving local client shows this:
	fetchmail -v -v slim.berklix.org
	........
	fetchmail: SMTP> MAIL FROM:<MAILER-DAEMON at slim.berklix.org> SIZE=14380
	fetchmail: SMTP< 250 2.1.0 <MAILER-DAEMON at slim.berklix.org>... Sender ok
	fetchmail: SMTP> RCPT TO:<jhs at localhost>
	fetchmail: SMTP< 250 2.1.5 <jhs at localhost>... Recipient ok
	fetchmail: SMTP> DATA
	fetchmail: SMTP< 354 Enter mail, end with "." on a line by itself
	#**************************.**********************.************************.**************.*************.**************.**************.**************.**************.**************.*************.**************.**********fetchmail: socket error while fetching from jhs at slim.berklix.org
	fetchmail: 6.3.8 querying slim.berklix.org (protocol POP3) at Fri Sep  9 22:49:31 2016: poll completed
	fetchmail: discarding new UID list
	fetchmail: Query status=2 (SOCKET)
	fetchmail: Deleting fetchids file.
	fetchmail: normal termination, status 2
	fetchmail: Deleting fetchids file.

/var/log/maillog:
 Land:
  dovecot: POP3(jhs): Disconnected: Logged out top=0/0, retr=0/0,
  del=0/0, size=0
 Slim:
  dovecot: POP3(jhs): Connection closed: Connection reset by peer
  top=1/14385, retr=0/0, del=0/9, size=112436

Nothing suspicious in /var/log/messages on slim

Strangely, I can sftp this /var/mail/jhs data to host=land, then I
can fetchmail it from host=land no problem. Makes me suspect a
packet problem ?  I have kept 2 samples of that data, so can repeat this.

I've noticed some stickiness on ssh session to host=slim in last weeks
(& slim is a newish (some weeks) re-installation of FreeBSD on a newish
vmwaee host (previous hardware problem) so something might have changed.)

I'm wondering what tools to use to analyse & compare connections
between home to both servers ? Reccomendations from /usr/ports/net/ I guess ?

I'm rusty, but I guess it can't be eg something like long ago DSL
MTU length issue, cos the same DSL link when up, works fine from
host=land, just not from host=slim

Can't be something weird in my dovecot POP3 server config, because
both have identical symbolic links
	/usr/local/etc/dovecot.conf -> ../../../site/usr/local/etc/dovecot.conf
& /site is frequently updated on both to keep identical

my .fetchmailrc entries to fetch from both server are symmetric too.

It's not some differential packet filtering in kernel issue, as with 
ipfw show
 land (jailed, work ok)
  ipfw: socket: Operation not permitted
 slim (vmware, locks up occasionaly, Ive not compiled ipfw in my kernel)
  ipfw: getsockopt(IP_FW_GET): Protocol not available

I do have an ipfw rule set on my home gate, but neither those nor the 2
IPs of remote server have changed in a long time, I wrote them symmetricly
way back, used to work on both, can't be that.

Top shows both hosts 96 to 99% idle most of the time.

Any suggestion of net / packet performance tools I might run please ?

PS I am subscribed to net at freebsd.org, but a CC to me also nice, Thanks !

Cheers,
Julian
-- 
Julian Stacey, BSD Linux Unix Sys Eng Consultant Munich
 Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable.
 http://berklix.eu/brexit/#stolen_votes