postgres, initdb, FreeBSD bug?
Konstantin Belousov
kostikbel at gmail.com
Sat Dec 15 09:53:34 UTC 2012
On Fri, Dec 14, 2012 at 10:43:00PM -0600, David Noel wrote:
> I've been fighting with a bug I can't quite seem to figure out and was
> told that this might be the place to come. I'm running
> postgresql-9.2.2 on FreeBSD 8.3-RELEASE-p5 and am having things break
> down when I try to run initdb. I got in contact with the pgsql-general
> mailing list and we debugged the issue to the point where it seemed
> that this might be a FreeBSD-related error. Relevant excerpts from
> several email are below that piece together the error:
>
> I'm running into the following error message when running initdb (FreeBSD host):
>
> ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug
> The files belonging to this database system will be owned by user "pgsql".
> This user must also own the server process.
>
> The database cluster will be initialized with locales
> COLLATE: C
> CTYPE: en_US.UTF-8
> MESSAGES: en_US.UTF-8
> MONETARY: en_US.UTF-8
> NUMERIC: en_US.UTF-8
> TIME: en_US.UTF-8
> The default text search configuration will be set to "english".
>
> creating directory /zdb/pgsql/data ... ok
> creating subdirectories ... ok
> selecting default max_connections ... 100
> selecting default shared_buffers ... 32MB
> creating configuration files ... ok
> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL:
> could not open file "pg_xlog/000000010000000000000001" (log file 0,
> segment 1): No such file or directory
> child process exited with exit code 1
> initdb: removing data directory "/zdb/pgsql/data"
>
> ...
>
> Interestingly, I have a second--virtually identical--server that I
> just tried initdb on. FreeBSD 8.3-RELEASE-p5, postgresql-server-9.2.2.
> Exact same "FATAL: could not open file pg_xlog" error. So it is
> reproducible.
>
> ...
>
> The relevant part of the ktrace output is
>
> 71502 postgres CALL unlink(0x7fffffffc130)
> 71502 postgres NAMI "pg_xlog/xlogtemp.71502"
> 71502 postgres RET unlink -1 errno 2 No such file or directory
> 71502 postgres CALL
> open(0x7fffffffc130,O_RDWR|O_CREAT|O_EXCL,S_IRUSR|S_IWUSR)
> 71502 postgres NAMI "pg_xlog/xlogtemp.71502"
> 71502 postgres RET open 3
> 71502 postgres CALL write(0x3,0x801a56030,0x2000)
> 71502 postgres GIO fd 3 wrote 4096 bytes
> .... a lot of uninteresting write() calls snipped ...
> 71502 postgres RET write 8192/0x2000
> 71502 postgres CALL close(0x3)
> 71502 postgres RET close 0
> 71502 postgres CALL unlink(0x7fffffffbc60)
> 71502 postgres NAMI "pg_xlog/000000010000000000000001"
> 71502 postgres RET unlink -1 errno 2 No such file or directory
> 71502 postgres CALL link(0x7fffffffc130,0x7fffffffbc60)
> 71502 postgres NAMI "pg_xlog/xlogtemp.71502"
> 71502 postgres NAMI "pg_xlog/000000010000000000000001"
> 71502 postgres RET link -1 errno 1 Operation not permitted
> 71502 postgres CALL unlink(0x7fffffffc130)
> 71502 postgres NAMI "pg_xlog/xlogtemp.71502"
> 71502 postgres RET unlink 0
> 71502 postgres CALL open(0x7fffffffc530,O_RDWR,<unused>0x180)
> 71502 postgres NAMI "pg_xlog/000000010000000000000001"
> 71502 postgres RET open -1 errno 2 No such file or directory
>
> This corresponds to the execution of XLogFileInit(), and what's
> evidently happening is that we successfully create and zero-fill
> the first xlog segment file under a temporary name, but then
> the attempt to rename it into place with link() fails with EPERM.
>
> This is really a WTF kind of failure, I think. The directory is
> certainly writable --- it was made under our own UID, and what's
> more we just managed to create the file there under its temp name.
> So how can we get an EPERM failure from link()?
>
> I think this is a kernel bug.
>
> regards, tom lane
>
> PS: one odd thing here is that the ereport(LOG) in
> InstallXLogFileSegment isn't doing anything; otherwise we'd have gotten
> a much more helpful error report about "could not link file". I don't
> think we run the bootstrap mode with log_min_messages set high enough to
> disable LOG messages, so why isn't it printing? Nonetheless, this error
> shouldn't have occurred.
>
> ...
>
> Where to from here? The freebsd-database at freebsd.org mailing list? The
> postgresql port maintainer? Who should I be in touch with?
>
> ...
>
> You need to talk to some FreeBSD kernel hackers about why link()
> might be failing here. Since you see it on UFS too, we can probably
> exonerate the ZFS filesystem-specific code.
>
> I did some googling and found that EPERM can be issued if the filesystem
> doesn't support hard links (which shouldn't apply to ZFS I trust).
> Also, Linux has a "protected_hardlinks" option that causes certain
> attempts at creating hard links to fail --- but our use-case here
> doesn't fall foul of any of those restrictions AFAICS, and of course
> FreeBSD isn't Linux. Still, I wonder if you're running into some
> misdesigned or misimplemented security restriction. You might want
> to look at your kernel parameters and see if any of them look like
> they might have to do with restricting hard-link operations.
>
> Also, since Amitabh failed to duplicate the failure on both earlier
> and later FreeBSD kernels, and we've not heard reports of this from
> anybody else either, it seems more than possible that it's a plain
> old bug in the specific kernel version you're using.
>
> As a short-term workaround, I'd suggest rebuilding with
> HAVE_WORKING_LINK disabled. (Just remove that #define from
> src/include/pg_config_manual.h and rebuild.)
>
> regards, tom lane
>
> ...
>
> Does this make any sense to anyone?
Show the ktrace from the same error on UFS.
Show the security.bsd sysctl settings, in particular, harlink_check_{u,g}id.
Show the ls -la output for the pg_xlog directory.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20121215/0575eb8f/attachment.sig>
More information about the freebsd-hackers
mailing list