ntpd dies nightly on a server with jails
O. Hartmann
o.hartmann at walstatt.org
Thu Mar 23 14:38:49 UTC 2017
On Fri, 17 Mar 2017 12:20:15 -0600
Ian Lepore <ian at freebsd.org> wrote:
> On Fri, 2017-03-17 at 18:05 +0100, O. Hartmann wrote:
> > Am Wed, 15 Mar 2017 13:12:37 -0700
> > Cy Schubert <Cy.Schubert at komquats.com> schrieb:
> >
> > >
> > > Hi O.Hartmann,
> > >
> > > I'll try to answer as much as I can in the noon hour I have left.
> > >
> > > In message <20170315071724.78bb0bdc at freyja.zeit4.iv.bundesimmobilie
> > > n.de>,
> > > "O. H
> > > artmann" writes:
> > > >
> > > > Running a host with several jails on recent CURRENT (12.0-CURRENT
> > > > #8 r315187:
> > > > Sun Mar 12 11:22:38 CET 2017 amd64) makes me trouble on a daily
> > > > basis.
> > > >
> > > > The box is an older two-socket Fujitsu server equipted with two
> > > > four-core
> > > > Intel(R) Xeon(R) CPU L5420 @ 2.50GHz.
> > > >
> > > > The box has several jails, each jail does NOT run service ntpd.
> > > > Each jail has
> > > > its dedicated loopback, lo1 throughout lo5 (for the moment) with
> > > > dedicated IP
> > > > :
> > > > 127.0.1.1 - 127.0.5.1 (if this matter, I believe not).
> > > >
> > > > The host itself has two main NICs, broadcom based. bcm0 is
> > > > dedicated to the
> > > > host, bcm1 is shared amongst the jails: each jail has an IP bound
> > > > to bcm1 via
> > > > whihc the jails communicate with the network.
> > > >
> > > > I try to capture log informations via syslog, but FreeBSD's ntpd
> > > > seems to be
> > > > very, very sparse with such informations, coverging to null - I
> > > > can't see
> > > > anything suiatble in the logs why NTPD dies almost every night
> > > > leaving the
> > > > system with a wild reset of time. Sometimes it is a gain of 6
> > > > hours, sometime
> > > > s
> > > > it is only half an hour. I leave the box at 16:00 local time
> > > > usually and take
> > > > care again at ~ 7 o'clock in the morning local time.
> > > We will need to turn on debugging. Unfortunately debug code is not
> > > compiled
> > > into the binary. We have two options. You can either update
> > > src/usr.sbin/ntp/config.h to enable DEBUG or build the port (it's
> > > the exact
> > > same ntp) with the DEBUG option -- this is probably simpler. Then
> > > enable
> > > debug with -d and -D. -D increases verbosity. I just committed a
> > > debug
> > > option to both ntp ports to assist here.
> > >
> > > Next question: Do you see any indication of a core dump? I'd be
> > > interested
> > > in looking at it if possible.
> > >
> > > >
> > > >
> > > > When the clock is floating that wild, in all cases ntpd isn't
> > > > running any mor
> > > > e.
> > > > I try to restart with options -g and -G to adjust the time
> > > > quickly at the
> > > > beginning, which works fine.
> > > This is disconcerting. If your clock is floating wildly without
> > > ntpd
> > > running there are other issues that might be at play here. At most
> > > the
> > > clock might drift a little, maybe a minute or two a day but not by
> > > a lot.
> > > Does the drift cause your clocks to run fast or slow?
> > >
> > > >
> > > >
> > > > Apart from possible misconfigurations of the jails (I'm quite new
> > > > to jails an
> > > > d
> > > > their pitfalls), I was wondering what causes ntpd to die. i can't
> > > > determine
> > > > exactly the time of its death, so it might be related to
> > > > diurnal/periodic
> > > > processes (I use only the most vanilla configurations on
> > > > periodic, except for
> > > > checking ZFS's scrubbing enabled).
> > > As I'm a little rushed for time, I didn't catch whether the jails
> > > themselves were also running ntpd... just thought I'd ask. I don't
> > > see how
> > > zfs scrubbing or any other periodic scripts could cause this.
> > >
> > > >
> > > >
> > > > I'ven't had the chance to check whether the hardware is
> > > > completely all right,
> > > > but from a superficial point of view there is no issue with high
> > > > gain of the
> > > > internal clock or other hardware issues.
> > > It's probably a good idea to check. I don't think that would cause
> > > ntpd any
> > > gas. I've seen RTC battery messages on my gear which haven't caused
> > > ntpd
> > > any problem. I have two machines which complain about RTC battery
> > > being
> > > dead, where in fact I have replaced the batteries and the messages
> > > still
> > > are displayed at boot. I'm not sure if it's possible for a kernel
> > > to damage
> > > the RTC. In my case that doesn't cause ntpd any problems. It's
> > > probably
> > > good to check anyway.
> > >
> > > >
> > > >
> > > > If there are known issues with jails (the problem occurs since I
> > > > use those),
> > > > advice is appreciated.
> > > Not that I know of.
> > >
> > >
> > Just some strange news:
> >
> > I left the server the whole day with ntpd disabled and I didn't watch
> > a gain of the RTC
> > by one second, even stressing the machine.
> >
> > But soon after restarting ntpd, I realised immediately a 30 minutes
> > off! This morning,
> > the discrapancy was almost 5 hours - it looked more like a weird
> > ajustment to another
> > time base than UTC.
> >
> > Over the weekend I'll leave the server with ntpd disabled and only
> > RTC running. I've the
> > strange feeling that something is intentionally readjusting the ntpd
> > time due to a
> > misconfiguration or a rogue ntp server in the X.CC.pool.ntp.org
> >
>
> The rogue server theory is a bad one, unless you have configured just a
> single server in your ntp.conf and it is the rogue. Ntpd requires
> agreement among the set of configured servers, it will ignore outliers.
Past weekend, I had switched off ntpd and ran the server completely with the
onboard RTC. On Monday morning when I entered the office, the clock was in
synchronisation with the official time.
As usual, I update sources and buildworld. After a couple of builds over the
week and letting ntpd restart via rc.conf as usual after rebooting, I checked
over the past two days and i found the server always in a state of dissonant
clock.
The more curious part is that the clock is almost 6 hours behind UTC. I can not
tell whether the ntpd is still trying to adjust time to a foreign clock which
has another time reference. I checked the TZ and everything seems all right.
>
> It would help to have some actual data. What does ntpq -p show right
> after starting ntpd? Then a few minutes later, then again 10 minutes
[RESTART]
remote refid st t when poll reach delay offset jitter
==============================================================================
0.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
1.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
2.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
3.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
ptbtime1.ptb.de .INIT. 16 u - 64 0 0.000 0.000 0.000
ptbtime2.ptb.de .INIT. 16 u - 64 0 0.000 0.000 0.000
[after 1 Minute]
remote refid st t when poll reach delay offset jitter
==============================================================================
0.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
1.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
2.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
3.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
ptbtime1.ptb.de .PTB. 1 u 34 64 1 16.931 -4.841 0.000
ptbtime2.ptb.de .PTB. 1 u 34 64 1 18.273 -5.518 0.000
fks.dan.net.uk 117.161.90.132 3 u 31 64 1 24.217 -3.904 0.000
213.95.200.109 213.95.151.123 2 u 33 64 1 25.464 -2.449 0.000
ns3.customer-re 192.53.103.108 2 u 35 64 1 23.905 -1.187 0.000
ns1.blazing.de 213.172.96.14 2 u 36 64 1 17.045 -3.017 0.000
ntp2.m-online.n 212.18.1.106 2 u 36 64 1 20.758 -2.693 0.000
stratum2-3.NTP. 129.70.130.71 2 u 35 64 1 22.000 -3.800 0.000
estoma.de 144.76.96.7 3 u 33 64 1 7.919 -3.182 0.000
clint.blazing.d 213.172.96.14 2 u 34 64 1 17.642 -2.932 0.000
news01.nierle.c 192.53.103.103 2 u 34 64 1 19.880 -3.750 0.000
q.fu110.de 131.234.137.64 2 u 35 64 1 16.649 -6.037 0.000
[after ~10 Minutes]
remote refid st t when poll reach delay offset jitter
==============================================================================
0.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
1.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
2.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
3.de.pool.ntp.o .POOL. 16 p - 64 0 0.000 0.000 0.000
#ptbtime1.ptb.de .PTB. 1 u 45 64 177 15.740 0.289 1.147
#ptbtime2.ptb.de .PTB. 1 u 38 64 177 17.489 -0.651 1.632
#fks.dan.net.uk 117.161.90.132 3 u 46 64 177 21.736 -0.634 9.040
-213.95.200.109 213.95.151.123 2 u 41 64 177 23.400 1.216 1.353
+ns1.blazing.de 213.172.96.14 2 u 48 64 177 16.848 1.912 0.570
*ntp2.m-online.n 212.18.1.106 2 u 48 64 177 20.681 2.409 0.927
-stratum2-3.NTP. 129.70.130.71 2 u 44 64 177 20.868 1.482 0.719
+clint.blazing.d 213.172.96.14 2 u 42 64 177 16.612 2.374 12.795
-news01.nierle.c 192.53.103.103 2 u 40 64 177 20.127 1.504 12.851
#q.fu110.de 131.234.137.64 2 u 103 64 176 16.070 -0.769 0.663
> after that, etc. What is in the /var/db/ntpd.drift file? Are you
> using the standard freebsd ntp.conf file as delivered, or have you
> customized it? Any non-default settings in your rc.conf related to
> ntp?
The line in /etc/rc.conf is:
ntpd_flags="-4 -g -G -I 192.168.0.1 -p /var/run/ntpd.pid -f /var/db/ntpd.drift"
The IP at -I is the IP of the primary NIC of the machine, which has two NICs.
I use a customized /etc/ntp.conf and I did a lot of variations during the
approach to figure out the problem. I did the same on host onto the same
network, but being of "modern date" (regarding hardware, the server in question
is an 2008 two-socket Core2Duo XEON box with 2x 4 cores) and which does not host
jails. The reference host seems not to show the weird clock gain.
the recent /etc/ntp.conf looks this now:
tos minclock 3 maxclock 6
server ptbtime1.ptb.de
server ptbtime2.ptb.de
pool 0.de.pool.ntp.org
pool 1.de.pool.ntp.org
pool 2.de.pool.ntp.org
pool 3.de.pool.ntp.org
restrict 192.168.0.0 mask 255.255.255.0 noquery kod nomodify notrap \
nopeer
restrict default limited kod nomodify notrap noquery nopeer
restrict -6 default limited kod nomodify notrap noquery nopeer
restrict source limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict 127.127.1.0
restrict -6 ::1
leapfile "/var/db/ntpd.leap-seconds.list"
>
> -- Ian
More information about the freebsd-current
mailing list