Giving more CPU time to a swapping process?

[Erik Trulsson]

On Thu, Apr 06, 2006 at 04:43:42PM +0800, Karl Ma wrote:
> Hi,
> 
> I have a python program in freebsd, doing a heavey indexing job involving a
> mega size array.
> 
> The process is so memory-hungry that it starts swap after the physical RAM
> max out. (To be exact, I've lowered the per-process limitation to make this
> possible).
> 
> However, when I use top to monitor the status, the STATE of the process
> started to stay as "swread" for most of the time (instead of RUN before
> using swap) and its priority has dropped to -20; and the corresponding WCPU
> drops to around 1% only. And the CPU consumption time in total (for the
> whole job) would only increase a minute or two even the process has been
> running for more than a few hours.

"swread" means that the process is busy waiting for information to be read
from (or possibly written to) the swap space.  The process cannot use any
actual CPU time until it has gotten that information from swap.

It sounds like the process has entered that condition generally known as
"thrashing" where it spends most of the time swapping instead of doing
useful work.



> In Windows XP, which has less per-task resource restriction (I guess?), I
> did successfully complete the task on the same hardware machine; although it
> takes more than 30 mins.

Apparently the algorithms Windows XP uses to manage swap are a better fit
for this particular programs memory usage pattern than those of FreeBSD (or
you might run fewer other programs in parallell of XP, leaving more free
memory for this particular program.)  It is quite likely that for some
other programs you would see the reverse situation with XP being much
slower.

> 
> How can I push up the priority of the whole paging task? How can I
> allocate more CPU attention to this process? I've tried using "nice"
> but it does not help.

That won't help.  You need to add more RAM (or decrease the memory usage of
the program, or remove any other memory-hungry programs running at the same
time, or change the memory access pattern of the program so that it has
greater reference locality so it does not need to swap as often.)

> 
> Please refer to the below top's snapshots at different times. (A) is earlier
> than (B), and so on.
> 
> Thanks for your help in advance.
> 
> 
> = (A) =
> last pid:   766;  load averages:  0.66,  0.21,  0.11    up 0+00:19:54
> 23:23:04
> 63 processes:  2 running, 61 sleeping
> CPU states: 96.6% user,  0.0% nice,  3.4% system,  0.0% interrupt,  0.0%idle
> Mem: 341M Active, 35M Inact, 98M Wired, 704K Cache, 91M Buf, 522M Free
> Swap: 998M Total, 998M Free
> 
> PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> 751 root        1 118    0   245M   244M RUN      0:59 96.57% python
> 
> = (B) =
> last pid:   792;  load averages:  1.02,  0.70,  0.36    up 0+00:24:20
> 23:27:30
> 62 processes:  2 running, 60 sleeping
> CPU states: 92.5% user,  0.0% nice,  6.0% system,  1.5% interrupt,  0.0%idle
> Mem: 766M Active, 67M Inact, 115M Wired, 45M Cache, 109M Buf, 3636K Free
> Swap: 998M Total, 37M Used, 962M Free, 3% Inuse, 17M Out
> 
> PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> 751 root        1 128    0   744M   743M RUN      5:14 94.63% python
> 
> = (C) =
> last pid:   792;  load averages:  1.06,  0.81,  0.43    up 0+00:25:54
> 23:29:04
> 62 processes:  2 running, 60 sleeping
> CPU states: 95.5% user,  0.0% nice,  4.1% system,  0.4% interrupt,  0.0%idle
> Mem: 849M Active, 2868K Inact, 115M Wired, 28M Cache, 109M Buf, 1656K Free
> Swap: 998M Total, 95M Used, 903M Free, 9% Inuse, 236K Out
> 
> PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> 751 root        1 128    0   848M   841M RUN      6:42 91.46% python
> 
> = (D) =
> last pid:   792;  load averages:  1.15,  0.87,  0.47    up 0+00:26:36
> 23:29:46
> 62 processes:  1 running, 61 sleeping
> CPU states: 28.6% user,  0.0% nice, 18.8% system,  1.5% interrupt, 51.1%idle
> Mem: 843M Active, 5380K Inact, 116M Wired, 31M Cache, 109M Buf, 1656K Free
> Swap: 998M Total, 142M Used, 856M Free, 14% Inuse, 932K In, 56M Out
> 
> PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> 751 root        1 -20    0   878M   841M swread   7:14 79.05% python
> 
> = (E) =
> last pid:   817;  load averages:  0.06,  0.36,  0.36    up 0+00:32:31
> 23:35:41
> 62 processes:  1 running, 61 sleeping
> CPU states:  4.9% user,  0.0% nice,  7.1% system,  0.0% interrupt, 88.0%idle
> Mem: 803M Active, 76M Inact, 88M Wired, 28M Cache, 109M Buf, 1656K Free
> Swap: 998M Total, 234M Used, 765M Free, 23% Inuse, 3148K In, 2284K Out
> 
> PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> 751 root        1 -20    0   915M   756M swread   8:01  1.03% python




-- 
<Insert your favourite quote here.>
Erik Trulsson
ertr1013 at student.uu.se