Benchmark 1 - Dynamic pages vs. static pages vs. Memcached

Posted on 14.12.2010 by Kim N. Lesmer.
This is a benchmark test of the loading speed of dynamic pages vs. static pages vs. pages stored in Memchached using PHP. The test was performed on an Intel P4 1.7 Ghz with 1036028 kB of memory over a privat LAN. The server runs MySQL 5.0.51a and PHP 5.3.3 on a standard Debian installation. Between each change of settings all related processes was restarted.

The first test is run using time curl. The first time Memcached is run isn't displayed in any of the tests below (that's when the data get stored in the cache).

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row

Loading dynamic data 3 times.

real    0m0.030s
real    0m0.033s
real    0m0.031s

Loading static data 3 times (134 KB).

real    0m0.019s
real    0m0.019s
real    0m0.020s

Loading dynamic data from Memcached 3 times.

real    0m0.031s
real    0m0.038s
real    0m0.048s

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 10 times

Loading dynamic data 3 times.

real    0m0.247s
real    0m0.249s
real    0m0.247s

Loading static data 3 times (1.3 MB).

real    0m0.124s
real    0m0.124s
real    0m0.123s

Loading dynamic data from Memcached 3 times.

real    0m0.213s
real    0m0.214s
real    0m0.208s

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 20 times

Loading dynamic data 3 times.

real    0m0.557s
real    0m0.577s
real    0m0.562s

Loading static data 3 times (2.6 MB).

real    0m0.325s
real    0m0.323s
real    0m0.325s

Loading dynamic data from Memcached 3 times.

real    0m0.398s
real    0m0.410s
real    0m0.401s

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 30 times

Loading dynamic data 3 times.

real    0m0.830s
real    0m0.831s
real    0m0.836s

Loading static data 3 times (3.9 MB).

real    0m0.471s
real    0m0.467s
real    0m0.475s

Loading dynamic data from Memcached 3 times.

real    0m0.578s
real    0m0.569s
real    0m0.585s

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 40 times

Loading dynamic data 3 times.

real    0m1.099s
real    0m1.100s
real    0m1.106s

Loading static data 3 times (5.2 MB).

real    0m0.615s
real    0m0.638s
real    0m0.621s

Loading dynamic data from Memcached 3 times.

real    0m0.766s
real    0m0.764s
real    0m0.773s

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 50 times

Loading dynamic data 3 times.

real    0m1.374s
real    0m1.362s
real    0m1.372s

Loading static data 3 times (6.5 MB).

real    0m0.772s
real    0m0.769s
real    0m0.763s

Loading dynamic data from Memcached 3 times.

real    0m0.933s
real    0m0.953s
real    0m0.945s

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 100 times

Loading dynamic data 3 times.

real    0m2.734s
real    0m2.728s
real    0m2.732s

Loading static data 3 times (12.9 MB).

real    0m1.502s
real    0m1.508s
real    0m1.507s

Loading dynamic data from Memcached 3 times.

real    0m1.870s
real    0m1.868s
real    0m1.867s

Using the Apache AB tool

In the next test I will try re-run some of the above tests, but this time I will test using the Apache HTTP server benchmarking tool:

ab -kc 10 -t 30 http://webserver/test.php

This will open 10 connections, using Keep-Alive on them and hammering the webserver for 30 seconds through those connections.

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row

Loading dynamic data.

Finished 1517 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        135916 bytes
Concurrency Level:      10
Time taken for tests:   30.007 seconds
Complete requests:      1517
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      206795814 bytes
HTML transferred:       206440256 bytes
Requests per second:    50.55 [#/sec] (mean)
Time per request:       197.808 [ms] (mean)
Time per request:       19.781 [ms] (mean, across all concurrent requests)
Transfer rate:          6729.95 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6   6.2      5      34
Processing:    52  191  56.1    185     524
Waiting:       12  102  46.3    100     479
Total:         52  197  56.5    191     535
Percentage of the requests served within a certain time (ms)
  50%    190
  66%    213
  75%    227
  80%    237
  90%    261
  95%    288
  98%    347
  99%    375
 100%    535 (longest request)

Loading static data (134 KB).

Finished 2580 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /static1.html
Document Length:        136438 bytes
Concurrency Level:      10
Time taken for tests:   30.011 seconds
Complete requests:      2580
Failed requests:        0
Write errors:           0
Keep-Alive requests:    2560
Total transferred:      352966330 bytes
HTML transferred:       352010040 bytes
Requests per second:    85.97 [#/sec] (mean)
Time per request:       116.323 [ms] (mean)
Time per request:       11.632 [ms] (mean, across all concurrent requests)
Transfer rate:          11485.41 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   5.5      0     105
Processing:    37  116  10.7    116     267
Waiting:        2   99  16.9    105     126
Total:         38  116  13.3    116     290
Percentage of the requests served within a certain time (ms)
  50%    116
  66%    116
  75%    116
  80%    116
  90%    116
  95%    118
  98%    129
  99%    175
 100%    290 (longest request)

Loading dynamic data from Memcached.

Finished 2295 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        135898 bytes
Concurrency Level:      10
Time taken for tests:   30.010 seconds
Complete requests:      2295
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      312996949 bytes
HTML transferred:       312460583 bytes
Requests per second:    76.48 [#/sec] (mean)
Time per request:       130.762 [ms] (mean)
Time per request:       13.076 [ms] (mean, across all concurrent requests)
Transfer rate:          10185.38 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   11   6.5     10      35
Processing:    25  119  46.9    112     435
Waiting:        7   33  23.9     27     305
Total:         29  130  47.8    121     436
Percentage of the requests served within a certain time (ms)
  50%    121
  66%    138
  75%    151
  80%    161
  90%    191
  95%    223
  98%    265
  99%    296
 100%    436 (longest request)

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 10 times

Loading dynamic data.

Finished 180 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        1356685 bytes
Concurrency Level:      10
Time taken for tests:   30.257 seconds
Complete requests:      180
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      244327776 bytes
HTML transferred:       244285370 bytes
Requests per second:    5.95 [#/sec] (mean)
Time per request:       1680.941 [ms] (mean)
Time per request:       168.094 [ms] (mean, across all concurrent requests)
Transfer rate:          7885.84 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   12  17.1      3      88
Processing:  1035 1623 187.2   1617    2255
Waiting:      738 1349 165.9   1367    1955
Total:       1084 1634 191.2   1628    2309
Percentage of the requests served within a certain time (ms)
  50%   1628
  66%   1696
  75%   1730
  80%   1761
  90%   1854
  95%   1942
  98%   2112
  99%   2189
 100%   2309 (longest request)

Loading static data (1.3 MB).

Finished 255 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /static2.html
Document Length:        1361896 bytes
Concurrency Level:      10
Time taken for tests:   30.018 seconds
Complete requests:      255
Failed requests:        0
Write errors:           0
Keep-Alive requests:    255
Total transferred:      353160469 bytes
HTML transferred:       353061987 bytes
Requests per second:    8.49 [#/sec] (mean)
Time per request:       1177.181 [ms] (mean)
Time per request:       117.718 [ms] (mean, across all concurrent requests)
Transfer rate:          11489.17 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:   395 1153 463.6   1041    3199
Waiting:        2  110  22.7    115     142
Total:        395 1153 463.6   1041    3199
Percentage of the requests served within a certain time (ms)
  50%   1039
  66%   1224
  75%   1307
  80%   1400
  90%   1753
  95%   2093
  98%   2384
  99%   2956
 100%   3199 (longest request)

Loading dynamic data from Memcached.

Finished 248 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        1356505 bytes
Concurrency Level:      10
Time taken for tests:   30.085 seconds
Complete requests:      248
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      342312256 bytes
HTML transferred:       342252608 bytes
Requests per second:    8.24 [#/sec] (mean)
Time per request:       1213.088 [ms] (mean)
Time per request:       121.309 [ms] (mean, across all concurrent requests)
Transfer rate:          11111.65 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   35  24.8     30     112
Processing:   444 1144 480.5   1076    2737
Waiting:       83  246 126.2    216     874
Total:        484 1180 480.4   1113    2825
Percentage of the requests served within a certain time (ms)
  50%   1113
  66%   1366
  75%   1484
  80%   1571
  90%   1824
  95%   2013
  98%   2492
  99%   2672
 100%   2825 (longest request)

From the above it is very clear that static pages significantly improve page loading speed. Memcached improves page loading speed significantly too, but it comes with a price, mainly memory.

Before I end the testing I would like to try to use the PHP include() function to include static content and see how much such an approach slows down the page loading speed.

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row

Loading static data using include().

Finished 2543 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        136438 bytes
Concurrency Level:      10
Time taken for tests:   30.003 seconds
Complete requests:      2543
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      347874361 bytes
HTML transferred:       347280444 bytes
Requests per second:    84.76 [#/sec] (mean)
Time per request:       117.984 [ms] (mean)
Time per request:       11.798 [ms] (mean, across all concurrent requests)
Transfer rate:          11322.76 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13   5.3     13      34
Processing:    24  105  24.5    103     211
Waiting:        4   23   9.4     22      74
Total:         34  118  25.1    115     220
Percentage of the requests served within a certain time (ms)
  50%    115
  66%    123
  75%    130
  80%    135
  90%    151
  95%    162
  98%    178
  99%    186
 100%    220 (longest request)

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 10 times

Loading static data using include().

Finished 256 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        1361896 bytes
Concurrency Level:      10
Time taken for tests:   30.021 seconds
Complete requests:      256
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      351585096 bytes
HTML transferred:       351523817 bytes
Requests per second:    8.53 [#/sec] (mean)
Time per request:       1172.713 [ms] (mean)
Time per request:       117.271 [ms] (mean, across all concurrent requests)
Transfer rate:          11436.65 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   57  22.4     57     118
Processing:   236 1097 143.2   1093    1470
Waiting:       55  124  39.3    116     265
Total:        236 1154 139.6   1141    1522
Percentage of the requests served within a certain time (ms)
  50%   1141
  66%   1180
  75%   1206
  80%   1236
  90%   1335
  95%   1409
  98%   1427
  99%   1473
 100%   1522 (longest request)

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 20 times

Loading static data using include().

Finished 126 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /test.php
Document Length:        2723516 bytes
Concurrency Level:      10
Time taken for tests:   30.471 seconds
Complete requests:      126
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      356600854 bytes
HTML transferred:       356569399 bytes
Requests per second:    4.14 [#/sec] (mean)
Time per request:       2418.332 [ms] (mean)
Time per request:       241.833 [ms] (mean, across all concurrent requests)
Transfer rate:          11428.68 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   72  29.8     78     119
Processing:   848 2227 397.5   2144    3576
Waiting:       50  220  89.8    198     539
Total:        849 2299 398.5   2235    3689
Percentage of the requests served within a certain time (ms)
  50%   2235
  66%   2333
  75%   2424
  80%   2466
  90%   2623
  95%   3384
  98%   3566
  99%   3589
 100%   3689 (longest request)

I'm going to try loading the same page directly as HTML.

Displaying a table from MySQL containing 1.080 posts with only 1 column in each row - looped 20 times

Loading static data (2.6 MB).

Finished 127 requests
Server Software:        Apache/2.2.9
Server Hostname:        webserver
Server Port:            83
Document Path:          /static3.html
Document Length:        2723516 bytes
Concurrency Level:      10
Time taken for tests:   30.160 seconds
Complete requests:      127
Failed requests:        0
Write errors:           0
Keep-Alive requests:    127
Total transferred:      354821737 bytes
HTML transferred:       354770999 bytes
Requests per second:    4.21 [#/sec] (mean)
Time per request:       2374.788 [ms] (mean)
Time per request:       237.479 [ms] (mean, across all concurrent requests)
Transfer rate:          11488.98 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:  1039 2301 638.1   2156    5480
Waiting:        2  109  29.7    120     133
Total:       1039 2301 638.1   2156    5480
Percentage of the requests served within a certain time (ms)
  50%   2151
  66%   2313
  75%   2597
  80%   2727
  90%   3128
  95%   3384
  98%   4037
  99%   4503
 100%   5480 (longest request)

Conclusion

Using the include().function to display static pages doesn't slow down the loading time, but there is a major difference in the loading time between serving pages dynamically, serving static pages, and using Memcached as a memory cache.

Caching can be performed at many levels, but the most important level is HTTP. A static HTML document that uses HTTP caching Last-Modified header tag will give a much better performance than a dynamically generated document.

The only bottleneck with static pages will be the maximum number of child processes that the webserver can create, but this is another issue all together.

If you have any comments or corrections feel free to email me.