network bitrate of a poll of processes

[Marc Spitzer]

On Mon, May 26, 2008 at 4:27 PM, Mathieu Prevot
<mathieu.prevot at gmail.com> wrote:
> 2008/5/26 Marc Spitzer <mspitzer at gmail.com>:
>> On Mon, May 26, 2008 at 12:14 PM, Mathieu Prevot
>> <mathieu.prevot at gmail.com> wrote:
>>> 2008/5/26 Dan Nelson <dnelson at allantgroup.com>:
>>>> In the last episode (May 26), Mathieu Prevot said:
>>>>> Hi,
>>>>>
>>>>> I would like to know the bitrate of a pool of child processes that use
>>>>> a network connection, how can I have something like netstat -w1
>>>>> provide but at the process level ?
>>>>
>>>> If you can segregate them to their own UID, you can use an ipfw "count
>>>> ip from any to any uid 6666" rule to count the packets.  Another option
>>>> would be to jail them to a dedicated IP address and count traffic on
>>>> that IP.
>>>
>>> Both are nice... thank you. And don't you have something more portable
>>> ie. that could run on "any" posix os - like a sh script that use
>>> standard tools -  and that doesn't need root user intervention ?
>>>
>>> Mathieu
>>
>> I do not think so, firewall for example is controlled by root.  And
>> users do not generally have detailed access to other users
>> information.  You can look into the BASM framework to track what your
>> apps are doing and if you know the behavior to expect you can use
>> netflow/argus/tcpdump to track that and post process the logs in each
>> case into something useful.
>>
>> What are you trying to do, details do help?
>>
>> marc
>
> I have a script that generates many child processes that download
> videos (~10MB) in this case, or do heavy data processing/analysing. I
> can tune the number of workers as I want, but It's a shared cluster so
> I would like to
>
> 1) measure 2) optimize/control (by number of workers ? at the script
> level ! so I can update thing very dynamically)
> a) CPU load b) IO on drive(s) c) network load
>
> and I prefer solutions that don't need root access/intervention. I can
> measure indirectly network load using `date +%s` and `du -b |sed
> 's/\t.*//' ` regularly... but I may have to deal with several 10GB
> later - and actually the number of files is more critical than their
> size.
> Maybe writing a (very) quick and dirty python app that listen/attach
> to a group of processes (for the network case) ?
>
> Mathieu
>

for the CPU stuff take a look at ps:
bob% ps aO%mem,%cpu,ppid
  PID %MEM %CPU  PPID  TT  STAT      TIME COMMAND
  768  0.2  0.0     1  v0  Is+    0:00.01 /usr/libexec/getty Pc ttyv0
  769  0.2  0.0     1  v1  Is+    0:00.01 /usr/libexec/getty Pc ttyv1
  770  0.2  0.0     1  v2  Is+    0:00.01 /usr/libexec/getty Pc ttyv2
  771  0.2  0.0     1  v3  Is+    0:00.01 /usr/libexec/getty Pc ttyv3
  772  0.2  0.0     1  v4  Is+    0:00.01 /usr/libexec/getty Pc ttyv4
  773  0.2  0.0     1  v5  Is+    0:00.01 /usr/libexec/getty Pc ttyv5
  774  0.2  0.0     1  v6  Is+    0:00.01 /usr/libexec/getty Pc ttyv6
  775  0.2  0.0     1  v7  Is+    0:00.01 /usr/libexec/getty Pc ttyv7
  272  0.2  0.0     1 con- I      0:00.04 dhclient: le0 [priv] (dhclient)
  951  0.6  0.0   949  p0  Rs     0:00.28 -zsh (zsh)
 1056  0.2  0.0   951  p0  R+     0:00.01 ps aO%mem,%cpu,ppid

add what you need and you have your cpu and such stats, it might do
your IO as well also look for thing in (s)bin that end in stat for
things that might be a better fit.

Network I do not know how to get you the level of access you want
without root/trusted access at least on collection.  You will not need
it on processing.

as far as managing thing after you have the information I have nothing
for you there.

good luck,

marc
-- 
Freedom is nothing but a chance to be better.
Albert Camus