Question about bottle neck in storage

[Pete Wright]

On 9/24/19 8:45 AM, John Fleming wrote:
> Is there anyway to see how busy a SAS/Sata controller is vs disks? I
> have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid
> 10 ZFS.
>
> When firing off a dd I (bs=1G count=10) seems like the disks never go
> above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW
> or if its something else (like terrible dd options).
>
> my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a
> dedicated 8 drive board inside the front of the R720. Basically i'm
> just saying its not a single SAS cable to 14 drives.
>
> Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)
>
> Both controllers are in 8x slots running PCIe gen 3.

might want to take a look at sysutils/intel-pcm 
(https://github.com/opcm/pcm).  I *think* this should give you metrics 
on PCIe bus utilization among other useful status.

Also, lookup the bandwidth for the PCIe bus and see if your aggregate 
disk throughput on one of the PCIe lanes is saturating the bus (pcm 
should also help here).  You can also run "zpool iostat -v 2" to see per 
disk i/o metrics to help determine if this is an issue.


> BTW i'm sure this has been asked a million times but what would be
> some decent benchmark tests while i'm at it?

I generally run several tests and then compare results, for example 
bonnie++, iozone, iperf (writing over the wire and to disk) as well as 
some more realistic scripts based on the use-case i'm building a 
solution for.  hope that helps.

-pete

-- 
Pete Wright
pete at nomadlogic.org
@nomadlogicLA