Re: Very slow scp performance comparing to Linux

From: Mark Millard <marklmi_at_yahoo.com>
Date: Wed, 30 Aug 2023 08:49:54 UTC
On Aug 30, 2023, at 01:22, Mark Millard <marklmi@yahoo.com> wrote:

> On Aug 30, 2023, at 01:17, Mark Millard <marklmi@yahoo.com> wrote:
> 
>> On Aug 29, 2023, at 12:52, Mark Millard <marklmi@yahoo.com> wrote:
>> 
>>> Wei Hu <weh_at_microsoft.com> wrote on
>>> Date: Tue, 29 Aug 2023 12:55:35 UTC :
>>> 
>>>> Thanks for the update. Seems the numbers are the same on zfs and ufs. That's 
>>>> good to know. 
>>>> 
>>>> Yes, your numbers on ARM64 are better than mine on Intel. However, my original
>>>> intention was to find out why scp on Linux is performing much better than FreeBSD
>>>> under the same hardware env. 
>>>> 
>>>> Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4 
>>>> file system.
>>> 
>>> 
>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
>>> Hyper-V UEFI reports for the disk created:
>>> 
>>> 1. SCSI Disk 0,0
>>>   The boot loader did not load an operating system.
>>> 
>>> (It then reports the network adapter attempt found no
>>> boot image, but that is expected.)
>>> 
>>> That leaves me wondering if Hyper-V Quick Create
>>> established a VM file holding Intel/AMD material
>>> despite the aarch64 context.
>>> 
>>> Establishing a Ubuntu more directly is not familiar and
>>> will have to be a background activity and, so, likely
>>> will not be timely. If I did any experiments outside
>>> Hyper-V (native booting), they would be with slower
>>> USB3 SSD media than I use for FreeBSD.
>>> 
>>> I did notice that Hyper-V Quick Create did not create
>>> a fixed sized disk but a dynamic sized one. That is
>>> different than what I did for FreeBSD.
>>> 
>>> Also, it was not obvious if you were after aarch64
>>> Hyper-V testing vs. native-boot testing vs. both. So
>>> I may have gone the wrong direction from the start.
>>> It is possible that I'd find establishing a native-boot
>>> easier and then be able to have a VM file created from
>>> the media, more like what I did with FreeBSD.
>>> 
>>> The Ubuntu activity likely would not be analogous to
>>> the FreeBSD builds having -mcpu= optimization used.
>>> 
>>> Back to $work.
>>> 
>> 
>> I found a sequence of UI operations that worked for
>> installing Ubuntu server 22.04.3 into Hyper-V in
>> Windows 11 Pro on the Windows Dev Kit 2023 via
>> use of a downloaded *.iso .
>> 
>> The kernel that results predates 6.0:
>> 
>> $ uname -ap
>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
>> 
>> Using my usual rule of rebooting before the first scp:
>> 
>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img markmi@localhost:FreeBSD-14-TEST.img
>> . . .
>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              100% 5120MB 431.3MB/s   00:11 
>> 
>> $ rm FreeBSD-14-TEST.img
>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img markmi@localhost:FreeBSD-14-TEST.img
>> . . .
>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              100% 5120MB 482.2MB/s   00:10
>> 
>> Definitely faster than the FreeBSD results that I reported
>> earlier, including faster than the ThreadRipper 1950X with
>> Optane in a PCIe slot (more like 300 MiBytes/sec).
>> 
>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
>> disk under Hyper-V.
>> 
>> For reference:
>> 
>> $ lsblk -f
>> NAME   FSTYPE   FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
>> loop0  squashfs 4.0                                                    0   100% /snap/core20/1977
>> loop1  squashfs 4.0                                                    0   100% /snap/lxd/24326
>> loop2  squashfs 4.0                                                    0   100% /snap/snapd/19459
>> sda                                                                              ├─sda1 vfat     FAT32       F7E9-1344                                 1G     1% /boot/efi
>> └─sda2 ext4     1.0         48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe   99.3G    14% /
>> 
>> 
>> 
>> An experiment would be to have a small amount if RAM relative
>> the file size. That would force it to actually write to media
>> for some part of the file copy.
> 
> The wording was poor: "force it" here is just from the
> Ubuntu viewpoint. I make no claim to know if Hyper-V
> is actually writing the material out to media at the
> time vs. later.
> 
>> So using 1024 MiByte of RAM assigned in Hyper-V:
>> 
>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img markmi@localhost:FreeBSD-14-TEST.img
>> . . .
>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              100% 5120MB 407.5MB/s   00:12
>> 
>> $ rm FreeBSD-14-TEST.img
>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img markmi@localhost:FreeBSD-14-TEST.img
>> . . .
>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              100% 5120MB 404.7MB/s   00:12
>> 
>> Still definitely faster than the FreeBSD results that I
>> reported earlier, including faster than the ThreadRipper
>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).

One more variation in ubuntu under Hyper-V, still with 1024 MiBytes
of assigned RAM: use of localhost:/dev/null

$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img markmi@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              

$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img markmi@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              100% 5120MB 492.9MB/s   00:10


The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS context):

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img root@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img root@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img                                                                                              100% 5120MB 198.7MB/s   00:25


Note: At most one VM running at a time, never both in overlapping times.

===
Mark Millard
marklmi at yahoo.com