Storable byteorder incompatibilities in different FreeBSD installations

[Fernan Aguero]

Hi,

I've been bitten by binary incompatibilitiess (bytorder, and/or byte
sizes of ints, longs, and whatnot) between two perl installations
(both on FreeBSD, using latest perl5.8 from ports).

Upon investigation, it does not seem clear to me why the two
installations should be different. And it prompted me to ask a few
questions :)

The context:
I have a perl web application running under mod_perl in FreeBSD that
dumps Perl data structures using Storable (binary format) and stores
them, after encoding them with Base64, into a MySQL database. This
application is under production and we have just deployed a test
server for development in another FreeBSD box.

Binary incompatibilities when reading the Storable binary data have
been observed in the past when running the web application from Linux
boxes (Ubuntu, i386 or amd64).

Now, they have been also observed under FreeBSD in the new test
server. This prompted us to do an investigation of the issue, because
up to now we were assuming we would have no problems if we kept our
production servers running FreeBSD.

Below is the description of each box and the output of:
perl -MConfig -le "print @Config{byteorder}"
perl -MConfig -le "print @Config{shortsize}"
perl -MConfig -le "print @Config{intsize}"
perl -MConfig -le "print @Config{longsize}"
perl -MConfig -le "print @Config{longlongsize}"
perl -V (I'm not including the full output, due to its size, only
diffs between selected cases)

1. Production server: FreeBSD-6.3, i386, perl5.8 from ports
byteorder: 12345678
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8

2. Development/test server: FreeBSD-7.1-RC1, amd64, perl5.8 from ports
byteorder: 12345678
shortsize: 2
intsize: 4
longsize:8
lonlongsize: 8

3. Yet another server where we did some tests: FreeBSD-6.3, i386,
perl5.8 from ports, perl5.8 standalone build

perl5.8 from ports
byteorder: 12345678
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8

perl5.8, standalone build
cd /usr/ports/lang/perl5.8 && make extract
cd work/perl-5.8.8 && ./Configure (answering all questions with the
default values except for the prefix installation)
make && make test && make install (PREFIX is /home/fernan/perl)
byteorder: 1234
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8

As you can see, a Perl built outside of the ports system, has a
different binary configuration (and should produce incompatible
Storable images) as the Perl built from ports. As a sidenote, the
binary configuration is similar to the one found in ubuntu
pre-compiled binary perl packages (Linux, see below).

Attached to this message is a diff of the output produced by 'perl -V'
both for the perl built from the lang/perl5.8 port, and from the perl
built from sources, as explained above.

4. Other test boxes: Ubuntu-8.10, i386, perl5.8 and perl5.10
byteorder: 1234
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8

Up to now the questions that come to mind are:

i) why is it that the perl built from ports produces binary dumps
(using Storable) with incompatible byteorder with respect to the one
built from Perl sources? (both perls were built on the same platform,
same gcc)

ii) Why does the Perl built from ports have an unusual byteorder?

iii) Why does Perl built from ports in different platforms produce
incompatible binary dumps (this time because of a difference in
longsize)? Is it because it was compiled with a different gcc (3.4.6
in FBSD-6.3 vs 4.2.1 in FBSD-7.1)? Or because it was compiled in a
different platform (amd64 vs i386)?

Reading the Configure script that comes with perl-5.8.8, and regarding
the byteorder, it says that:
"A big-endian machine like a Pyramid or a Motorola 680?0 chip will
come out to 4321. A little-endian machine like a Vax or an Intel 80?86
chip would be 1234. Other machines may have weird orders like 3412.  A
Cray will report 87654321,
an Alpha will report 12345678. If the test program works the default is
probably right. I'm now running the test program..."

So why is it that building Perl outside of the ports system makes the
test program report '1234' which is what I would expect from an i386
box, while it gives '12345678' when running from within the ports
system? Why this unusual byteorder?


And now for a pragmatic and urgent question:
iv) how can I use the ports system to build a Perl in different FBSD
boxes that would result in the same binary configuration (byteorder,
shortsize, intsize, etc.) to guarantee it would produce compatible
(i.e. readable from other FBSD boxes) dumps of Perl data structures?


Sorry for the long post, and have yourselves a happy and peaceful 2009!

-- 
fernan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: perl.gama.diff
Type: text/x-patch
Size: 4574 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-perl/attachments/20090102/2e58d6e2/perl.gama.bin