We encourage you to start looking at your site’s speed (the tools above provide a great starting point) — not only to improve your ranking in search engines, but also to improve everyone’s experience on the Internet.

The post lists a number of tools everyone should be using already, such as YSlow, google’s own PageSpeed, online speed waterfall diagram tool and webmaster tools. Webmaster tools recently added a beta feature which provides data about your sites speed relative to other sites on the internet.

Here is a sample report:

webmaster tools page speed chart

Performance overview
On average, pages in your site take 6.3 seconds to load (updated on Apr 11, 2010). This is slower than 83% of sites. These estimates are of medium accuracy (between 100 and 1000 data points). The chart below shows how your site’s average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times.

Linus System admin blog will have a series on page speed improvement.  Stay tuned!

I’ll assume that you have installed apc already, if this is not the case this will probably be something as simple as running
pecl install apc
or downloading the archive from pecl and running:
phpize; ./configure; make; make install

The APC extension needs to be enabled either in php.ini or in one included file with a line like this:
extension=apc.so
there are many other parameters that apc can be fine tuned (see the official doc for more info), but without any other change, just with this line apc will be enabled on all the vhosts on the server.

Disabling some vhosts from using APC
- if we want to disable APC for a particular vhost we just have to add to the vhost config or to .htaccess:
php_flag apc.cache_by_default Off

Enabling APC only on some vhosts
- if we want to have APC disabled by default globally we will have in php.ini:

extension=apc.so
[apc]
apc.cache_by_default=0 # disable by default
... other apc settings...

and we will enable APC for the particular vhost config or using .htaccess using:
php_flag apc.cache_by_default On

Hopefully you found this post useful and this will give you a reason to use APC with more confidence knowing that you have the granularity to disable/enable it as needed in a shared environment.

First off, this is great.  It should improve the speed of looking up the DNS info of many sites, and if the service takes off, it should take the load off your NS.

The focus on speed if very clear, the Google public DNS server lists this first as one of the advantages.  It also points to the speed problems caused by DNS latency.

Google Public DNS IP addresses

The Google Public DNS IP addresses are as follows:

• 8.8.8.8
• 8.8.4.4

#!/bin/bash

PID=`cat /var/run/sshd.pid`
ps -p $PID -o pid -o rss -o %mem -o cmd >> logname
exit

The output of log file will looks like this:

PID   RSS %MEM CMD
2607  1036  0.0 /usr/sbin/sshd

Note the RSS column, it the values keep increasing with usage of application that would indicate a memory leak.

Another way to do the same thing is with pidstat which is a part of the sysstat package. The package can be installed via yum or aptitude but may not come with pidstat, if so download and install or build the latest version from here.

Pidstat is better way to track resources of an application because it has built in polling as well as combining process’s children statistics into the total. See pidstat man page options. In the script below we will use pidstat to track memory usage for sshd by polling 12 times every 5 minutes and e-mailing a report and writing to a log file. This script can also be run via cron.

#!/bin/bash
# track memory usage of sshd using pidstat and send report
# http://www.linuxsysadminblog.com - MaxV

export PID=/var/run/sshduse.pid
export TIMESTAMP=`date +%Y%m%d_%H%M%S`
export LOGDIR=/var/log/
export SSHD_LOG="${LOGDIR}sshd_memUsage_${TIMESTAMP}"
export SSHD_PID=`cat /var/run/sshd.pid`
export MAILTO=user@domain.com

if [ ! -e ${PID} ]; then

#create pid file
echo $$ > ${PID}

#log begin of script to /var/log/messages
/usr/bin/logger "Starting SSHD Memory Usage Tracker"

# pidstat portion, poll 12 times with 5 minutes apart
/usr/bin/pidstat -r -p ${SSHD_PID} 300 12 >> ${SSHD_LOG}

#e-mail report
mail -s "SSHD memory usage ${TIMESTAMP}" ${MAILTO} < ${SSHD_LOG}

#clean up pid file
if [ -f ${PID} ]; then
rm -rf ${PID}

#log end of script to /var/log/messages
/usr/bin/logger "Ending SSHD Memory Usage Tracker"
fi
exit 0
else

exit 0
fi

The output of this script:

Linux 2.6.18-6-686 (hostname)       06/16/09        _i686_  (2 CPU)

14:21:26          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
14:21:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:22:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:22:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:23:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:23:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:24:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:24:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:25:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:25:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:26:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:26:56         2564      0.47      0.00    4932   1108   0.05  sshd
14:27:26         2564      0.00      0.00    4932   1108   0.05  sshd
Average:         2564      0.04      0.00    4932   1108   0.05  sshd

From the output we can see that “RSS” values are not increasing as time progresses which means that it’s a solid piece of coding.

The good news is that if you have data that does not change very much the query cache can give you an enormous performance boost. It even bypasses the query optimizer so that if the query is complex even more cpu time is saved. Knowing this you can optimize your application by chopping complex queries into smaller queries that only use that data that never changes.

Of course there are some tricks to using the query cache. The first one is the size of the query cache. The default is 16MB which doesn’t do much. However, keep in mind that any memory assigned to the query cache is removed from another cache so it’s very important to strike a good balance. Of course the balance is very application dependent. The second parameter is the maximum allowed result set size. It really doesn’t do any good to allow 16MB result sets into the cache because it would take only one badly written query to flush out the entire cache. 1MB is standard but in my personal experience queries that return 1MB of results on a frequent basis usually indicate that the software needs to be optimized.

So when is the query cache a bad thing? Well, in short, when the cache gets flushed out all the time and only adds to the overhead it’s usually better to assign the memory to storage engine dependent cache. If there are constant updates and inserts to most of your tables it will invalidate the results in the query cache pretty quickly and assigning memory to it is a waste of resources.

Useful tools like MySQL Tuner will give some quick information about the efficiency of the query cache but I do think it is a bit quick in suggesting more memory for the cache.

• Look at the number of requests a page makes to the server
• Use yslow to measure page rendering (often a page performance is perceived, not just based on the server response time)

• Remove search, use alternate solutions such as Apache Solr or Google Search API
• Use CDN as much as possible
• Use Reverse Proxy Cache and memcache
• Obviously use drupal cache

Some other notes that are somewhat related to drupal performance and site performance management in a clustered hosting environment.

Manual updates and rollback

OLD WAY: tar, move/copy untar restart services

OLD WAY: rsync

BETTER WAY: Capistrano

Managing systems:

BETTER WAY: bcfg2

Monitoring Tools

• Capacity Load: analyzing trends, predicting load, checking results of configuration and software changes (cacti, munin)
• Failure: analyzing downtime, notification (nagios – using nrpe agents to monitor diverse services (do we use it this way?) , hyperin)

Use monitoring tools to closely observe cluster replication and cashing as the failures in this area are the most difficult to solve.

Temporary reporting servers
Create a one time copy of an existing production database server to run specific heavy reports. This is ideal for monthly reports since the server only needs to be up and running for several hours per month.

Backup database server
This is a backup solution where the server is only allocated once there is a problem with the primary server which makes a lot of sense because the client only pays for the server once it is used. One downside to this scenario is that the server has to created and loaded with the latest backup which will result in a decent amount of downtime but at least all of this can be automated. A bigger problem is the loss of data since the latest backup.For our high availability sites we have a standby database server replicating all changes from the master so we can switch over at a moment’s notice without losing any data.

Migrations
Performing a migration or a system upgrade usually brings some downtime. Promoting a standby system to primary creates a single point of failure so it makes sense to create a remporary standby of the standby.

Development branches and testing environments
For development branches we usually only need an extra database for a short amount of time although truth be told, those database are not very large in general so we tend to put them on the same development database server anyway. The same is true for testing and QA. These activities usually occur in cycles which means that they are very attractive targets for cloud based servers.

Alternative data center
Yes, it happened to us once that our datacenter went off line due to a very heavy attack. Instead of finding another data center for these eventualities it could be useful to have cloud based backup servers defined. However, this requires the extra effort of keeping these instances up to date for this eventuality. Additionally, DNS caching will stop the switch from being instantaneous. A geographical load balancing solution would be the answer to that but at that point the cost for preparing for this eventuality will have to be compared to the loss due to down time.

In the article below it shows how to patch the slow query log to track queries that last less than a second. Obviously you don’t want to have this running continually in production because the amount of logging would be enormous but in test environments or for a limited time in production this can be very useful. Be prepared to analyse some huge amounts of data though.

http://www.mysqlperformanceblog.com/2006/09/06/slow-query-log-analyzes-tools/

Although this video has been around for a while, its a must for system admins who bang their heads at trying to squeeze performance  out of the 10 to 20 percent of the user wait time they are responsible for.  

Steve’s Performance golden rule:

80% to 90% of the end-user response time is spent on the front end.  Start there.

Here is the older Yahoo video 

;

Followed by the more recent Google talk

Check out Yslow firefox plug in as well as the IBM Page Detailer – its the product he uses to map out the front end work of the browser.  Great tools to start thinking about why your website is slow or why pages are loading slow.

On any weekday around lunch time network traffic between web and data servers was around 80Mb/s. On a 100Mb/s connection that is dangerously high and to address this we’ve been in the process of modifying queries that do a SELECT *. There really hardly ever is a reason to do a SELECT * except when you have very flexible code that automatically deals with extra columns. That was not the case with this application.

On Cyber Monday we found that traffic for this shop was touching 100Mb/s and web site performance really suffered. As a temporary measure we switched the database server to a 1Gb/s connection but both web servers stayed on 100Mb/s connections.

Looking at the slow query log and mtop revealed a ton of similar queries:

# User@Host: username[username] @  [10.0.0.123]
# Query_time: 2  Lock_time: 0  Rows_sent: 1948  Rows_examined: 2047
SELECT * FROM sites, sites_to_bundle WHERE sites_to_bundle.sites_id = sites.sites_id AND sites.site_name ='shop1';

In and by them selves these queries look pretty harmless. 2000 rows is not that much. Maybe more than is needed but not enough to choke up the network either. The problem turns out to be that this query was executed several times for each page in the check out process. The database server was not overloaded since it had the whole result set in its cache, but it had to transfer 2MB of data for each request. When a developer investigated it turned out that only one 2 digit value of the entire result set was used.

We rewrote query and pushed to production on that same day. Yes, I know that a previous blog post tells us not to do that but in this case I’m glad we did. Network traffic dropped below 10Mb/s and the web site was flying. Below is the Cacti graph that shows that difference. Shortly after 15:00 we implemented the optimization and traffic dropped dramatically.

Network traffic decrease after query optimization