Benchmark 4 - Notes About Caching and Benchmarking

Posted on 17.12.2010 by Kim N. Lesmer.

When benchmarking web related cache tools you really should benchmark the application that you want to improve performance upon.

Keep in mind the golden rule of benchmarking: All benchmarks are flawed benchmarks!

If you are running a benchmark on MySQL Cache using a simple SELECT query and your real life application is using prepared statements, your benchmark results will be useless because MySQL Cache doesn't work on prepared statements.

Benchmarking is always a bit difficult. The results may vary depending on both hardware, network, and software. If either one changes the results will change as well.

The individual settings also plays a major role. In case you are running a test on a memory based cache, the size of the memory matters and fragmentation matters, but you must also remember to evaluate the difference between the development environment and the production environment.

If your application is an Open Source application that needs deployment by several people (perhaps even thousands of people) you should always avoid cache features in development. This way you are sure to get a good feeling for how your application performs if the cache features isn't available.

Every time any part of your application runs more than once and produces the exact same result you have failed a little bit. Every time your entire application runs and produces the exact same result as the last time it ran you have failed a lot.

Caching is not an optimization to be introduced some time later in development. It is an integrate part of good coding style and development principles from the unset.

By this I don't mean that your application should depend on a specific cache feature that may or may not be available, but you should always consider how you can make your application perform optimally and avoid duplication both in code and running results.

Static pages should always be an integrated part of any web application even if the underlying content is dynamically stored.

Sometimes this isn't possible for parts of the application and in such cases a memory based cache such as Memcached is extremely useful.

I hope you have found my benchmarking articles useful even though I know that real life applications benchmarking and good programing design really is the key issue when it comes to performance.

If you have any comments or corrections feel free to email me.