Performance Problems.. Server hardware smoked by $500 box?

[John Straiton]

> The first step would be to try to quantify the performance
> difference in serving the actual web pages.  Find a single 
> page that you think is slow on the production system and that 
> can be accessed without having to be part of a session, and 
> quantify the performance difference for that page.  Remember 
> you don't care about high load, just a single user request.  
> You could use apachebench ("ab", comes with apache... 
> something like "ab -c 1 -n 20
> http://server/path/to/page") or any simple command line tool 
> that you can time (eg. repeatedly run "time GET 
http://server/path/to/page"). Do this from as near as possible to the
box you are running the web browser that sees the slowness loading.

I ran a couple of tests, and put the one you requested along with one of
the more amusing ones to me, below. These request a page that uses PHP
sessions (the sessions are kept over NFS) and a couple of database
queries. All 3 servers use the same database, tho obviously the database
server handles them locally along with handling the files locally vs
over NFS like the other two would.

I think to sidestep any more testing that may/may not give us more
clues, what we're going to press forward with is trying to get another
>133Mhz FSB setup in here since they're so cheap and just testing to see
if we see similar results if we set it's harddrive up as a mirror of the
production web machine. If by using the same OS installation, and only
change the underlying hardware, I'd say that would more-than-likely tell
us if this is strictly a hardware issue, assuming it performs equally
well as the development machine while using the production machine's
setup.

John Straiton
jks@ clickcom.com
Clickcom, Inc
704-365-9970x101 


Running "ab -c 1 -n 10" on these machines:
>Production Web over NFS
Benchmarking 209.198.22.161 (be patient).....done
Server Software:        Apache/2.0

Server Hostname:        209.198.22.161
Server Port:            80

Document Length:        67769 bytes

Concurrency Level:      1
Time taken for tests:   4.918 seconds
Complete requests:      10
Failed requests:        0
Broken pipe errors:     0
Total transferred:      681350 bytes
HTML transferred:       677690 bytes
Requests per second:    2.03 [#/sec] (mean)
Time per request:       491.80 [ms] (mean)
Time per request:       491.80 [ms] (mean, across all concurrent
requests)
Transfer rate:          138.54 [Kbytes/sec] received

>Production DB serving pages locally
Benchmarking 209.198.22.35 (be patient).....done
Server Software:        Apache/1.3.27

Server Hostname:        209.198.22.35
Server Port:            80

Document Length:        68585 bytes

Concurrency Level:      1
Time taken for tests:   2.232 seconds
Complete requests:      10
Failed requests:        0
Broken pipe errors:     0
Total transferred:      689870 bytes
HTML transferred:       685850 bytes
Requests per second:    4.48 [#/sec] (mean)
Time per request:       223.20 [ms] (mean)
Time per request:       223.20 [ms] (mean, across all concurrent
requests)
Transfer rate:          309.08 [Kbytes/sec] received

> Development Server over NFS
Benchmarking dev.clickcomworks.net (be patient).....done
Server Software:        Apache/1.3.27

Server Hostname:        dev.clickcomworks.net
Server Port:            8000

Document Length:        68237 bytes

Concurrency Level:      1
Time taken for tests:   1.799 seconds
Complete requests:      10
Failed requests:        0
Broken pipe errors:     0
Total transferred:      686380 bytes
HTML transferred:       682370 bytes
Requests per second:    5.56 [#/sec] (mean)
Time per request:       179.90 [ms] (mean)
Time per request:       179.90 [ms] (mean, across all concurrent
requests)
Transfer rate:          381.53 [Kbytes/sec] received


Now with "ab -c 10 -n 10", which is just beautiful

>Production Web over NFS
Benchmarking 209.198.22.161 (be patient).....done
Server Software:        Apache/2.0

Server Hostname:        209.198.22.161
Server Port:            80

Document Length:        67769 bytes

Concurrency Level:      100
Time taken for tests:   6.424 seconds
Complete requests:      10
Failed requests:        0
Broken pipe errors:     0
Total transferred:      817620 bytes
HTML transferred:       813228 bytes
Requests per second:    1.56 [#/sec] (mean)
Time per request:       64240.00 [ms] (mean)
Time per request:       642.40 [ms] (mean, across all concurrent
requests)
Transfer rate:          127.28 [Kbytes/sec] received


>Production DB over NFS
Benchmarking 209.198.22.35 (be patient).....done
Server Software:        Apache/1.3.27

Server Hostname:        209.198.22.35
Server Port:            80

Document Length:        68585 bytes

Concurrency Level:      10
Time taken for tests:   34.430 seconds
Complete requests:      100
Failed requests:        0
Broken pipe errors:     0
Total transferred:      6898700 bytes
HTML transferred:       6858500 bytes
Requests per second:    2.90 [#/sec] (mean)
Time per request:       3443.00 [ms] (mean)
Time per request:       344.30 [ms] (mean, across all concurrent
requests)
Transfer rate:          200.37 [Kbytes/sec] received

>Development over NFS
Benchmarking dev.clickcomworks.net (be patient).....done
Server Software:        Apache/1.3.27

Server Hostname:        dev.clickcomworks.net
Server Port:            8000

Document Length:        68237 bytes

Concurrency Level:      100
Time taken for tests:   2.093 seconds
Complete requests:      10
Failed requests:        0
Broken pipe errors:     0
Total transferred:      1046656 bytes
HTML transferred:       1040240 bytes
Requests per second:    4.78 [#/sec] (mean)
Time per request:       20930.00 [ms] (mean)
Time per request:       209.30 [ms] (mean, across all concurrent
requests)
Transfer rate:          500.07 [Kbytes/sec] receive