dd performance [was Re: Call for Foundation-supported Project Ideas]

This is very surprising to me.  I never see dd take significant CPU
consumption until the speed gets up into the GB/s range.  What are you
using for the bs= option?  If you set that too low, or use the
default, it will needlessly consume extra CPU and IOPs.  I usually set
it to 1m for this kind of usage.  And what kind of HDDs are these,
connected to what kind of controller?

On Thu, Dec 2, 2021 at 9:54 AM Stefan Blachmann <sblachmann@gmail.com> wrote:
>
> Regarding the suggestions to either improve or replace the ULE
> scheduler, I would like to share another observation.
>
> Usually when I need to zero out HDDs using dd, I use a live Linux.
> This time I did that on FreeBSD (13).
> My observations:
> - On the same hardware, the data transfer rate is a small fraction
> (about 1/4th) of which is achieved by Linux.
> - The first dd process, which erases the first HDD, gets almost all
> CPU and I/O time. The second process which does the second HDD is
> getting starved. It actually really starts only after the first one
> finished.
>
> To me it was *very* surprising to find out that, while erasing two
> similar HDDs concurrently takes about one day on Linux, on FreeBSD,
> the first HDD was finished after three days, and only after that the
> remaining second dd process got the same CPU time, making it proceed
> fast instead of creepingly slowly.
>
> So I guess this might be a scheduler issue.
> I certainly will do some tests using the old scheduler when I got time.
> And, I ask myself:
> Could it be a good idea to sponsor porting the Dragonfly scheduler to FreeBSD?
>
> On 12/2/21, Johannes Totz <jo@bruelltuete.com> wrote:
> > On 29/11/2021 03:17, Ed Maste wrote:
> >> On Sun, 28 Nov 2021 at 19:37, Steve Kargl
> >> <sgk@troutmask.apl.washington.edu> wrote:
> >>>
> >>> It's certainly not the latest and greatest,
> >>> CPU: Intel(R) Core(TM)2 Duo CPU     T7250  @ 2.00GHz (1995.04-MHz
> >>> K8-class CPU)
> >>
> >> If you're content to use a compiler from a package you can save a lot
> >> of time by building with `CROSS_TOOLCHAIN=llvm13` and
> >> `WITHOUT_TOOLCHAIN=yes`. Or, instead of WITHOUT_TOOLCHAIN perhaps
> >> `WITHOUT_CLANG=yes`, `WITHOUT_LLD=yes` and `WITHOUT_LLDB=yes`.
> >
> > (re-send to list, sorry)
> > Can we disconnect the compiler optimisation flag for base and clang? I
> > don't need the compiler to be build with -O2 but I want the resulting
> > base system to have optimisations enabled.
> > Right now, looks like both get -O2 and a lot of time is spent on
> > optimising the compiler (for no good reason).
> >
> >
>