FreeBSD on Amazon AWS EC2 long standing performance problems

[Gunther Schadow]

Hi Sean and Brendan

On 2/5/2021 6:14 PM, Sean Chittenden wrote:

> To be clear, this is a known issue that needs attention: it is not a
> benchmarking setup problem.  Network throughput has a similar problem and
> needs similar attention.  Cloud is not a fringe server workload.  -sc

I am so glad you're saying that, because I was afraid I'd have to argue
again and again to make people see there is a problem.

But I come here anyway with some numbers:

This is the Amazon Linux system which I fell back to in desperation, launched,
added ZFS, compiled and set up PostgreSQL-13.2 (whatever newest) and am now

pg_dumpall -h db-old |dd bs=10M status=progress |psql -d postgres

we had sucked the data tables at 32 MB/s that is about the same speed as I
got on the FreeBSD. I assume that might be network bound.

Now it's regenerating the indexes and I see the single EBS g3 volume on fire:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     7.00  781.00  824.00 99508.50 76675.00   219.54     1.98    1.66    1.75    1.57   0.41  66.40
nvme1n1           0.00     3.00 1740.00  456.00 222425.00 33909.50   233.46     4.43    2.42    2.29    2.91   0.46 100.80
nvme1n1           0.00     0.00 1876.00  159.00 239867.00 16580.00   252.04     3.56    2.20    2.09    3.47   0.49 100.00
nvme1n1           0.00     0.00 1883.00  151.00 240728.00 15668.00   252.11     3.49    2.15    2.10    2.83   0.49 100.00
nvme1n1           0.00     0.00 1884.00  152.00 240593.50 15688.00   251.75     3.54    2.19    2.13    3.00   0.49 100.00
nvme1n1           0.00     1.00 1617.00  431.00 206680.00 50047.50   250.71     4.50    2.63    2.49    3.13   0.48  98.40
nvme1n1           0.00     1.00 1631.00  583.00 208331.50 47909.00   231.47     4.75    2.54    2.49    2.66   0.45 100.00
nvme1n1           0.00     0.00 1892.00  148.00 241128.50 15440.00   251.54     3.20    2.01    1.96    2.73   0.49 100.00

On FreeBSD I haven't got the numbers saved now, but with all the simultaneous
reading and writing activity going on, the system got down to just about 40 MB/s
read and 40 MB/s write, if lucky.

There is heavy read of base tables, then sorting in temporary space (resad/write),
then write to the WAL and to the index. This stuff would bring FreeBSD to its
knees.

Here on Linux I'm not even trying to be smart. With the FreeBSD attempt I had
already taken the different read and write streams to different EBS disks each
having a different ZFS pool. That helped a little bit. But not much. On this
Linux thing here I didn't even bother and it's going decently fast. There is
still some 20% iowait, which could probably be optimized by doing what I did for
FreeBSD, separate devices, or, I might try to just make a single ZFS volume
striped from 5 smaller 100 GB drives rather than a single 500 GB drive.

This was just to provide some numbers that I have here.

But I absolutely maintain that this has got to be a well known problem and that
it's not about the finer subtleties of how to make a valid benchmark
comparison.

regards,
-Gunther