User cpu time VS system cpu time

[Ivan Voras]

On 05/06/10 13:33, cronfy wrote:
> Hello,
>
>>> I want to understand difference between user CPU time and system CPU
>>> time in system accounting.
>> But keep in mind that "kernel time" is a broad category - while IO time in
>> itself does not count as CPU time, file system operations for example do,
>> because they really can be CPU intensive.
>
> Ivan, thanks for the great explanation.
>
> I think that I can measure user filesystem usage with sa - it reports
> number of IO operations per user/command. In which other cases kernel
> time is used instead of user time for a process? I do not mean all of
> them - just that usually occur in practice.

Everything the kernel does when requested by the user is counted as 
kernel time - file system access, network access, getpid(), 
gettimeofday(), process scheduling, memory management, etc.

> I've noticed that there are moments when system load in top for system
> time is very high (60-80% while user load is 15-25%, this produces
> very high LA also). All processes that were run at this time show high
> kernel time usage, although they usually do not. System is getting
> back to normal after Apache restart (I think this is related to Apache
> shared memory somehow, but not sure).

As I told you before - monitor the "top" line for the processes you 
suspect and you will get a fairly good idea what they are doing. Look at 
the "STATE" column.

When you are looking at per-process statistics, the system time is also 
accounted. For example, if a process takes 50% of a CPU, it is possible 
that it takes 25% in userspace and 25% in kernel (the reverse is not 
true - kernel can take system CPU time without it being accounted on 
behalf of processes).

> This makes me suspect that system time in sa can not be relied on
> while measuring user system usage, because it notably varies under
> some circumstances for same operations. Am I wrong?

Everything can be accounted for by enough statistics :)