So far this is the most attended session.  Standing room only available only before it start.

What does facebook sysadmins have to support?

• Monthly 700 million minutes of time spent on fb
• 6billion pieces of content updated
• 3 billion photos
• 1 million connect implementations
• 1/2 billion active users

Infrastructure Growth

• fb reached a limit on leasing datacenter space
• fb is building their own http://www.facebook.com/prinevilledatacenter
• currently serving out of california and Virginia

Initially a LAMP stack.  LB -> Web Servers -> Services/Memcached/Databases

Originally facebook was a simple Apache PHP site.  When fb started hitting a limit on this, they started compiling PHP into C++ (HipHop for PHP).

FB claims to be the biggest memcache deployment in the world.  They server 300 Terbytes of memcached data out of memory.

MySQL improvements contributed back is flashcache.

Services supported

• News Feed
• Search
• Cache

Service implementation languages

• C++
• PHP – front end
• python
• Ruby
• Java
• erlang (chat room)

How do they talk between these?  Json?  SOAP?  No, fb implemented Thrift – ligtwaith software framework for cross language development, a common glue behind all facebook systems.

For Systems, what does fb have to worry about on a daily basis?

• deployment
• monitoring
• data manaement
• Core operating updates

Facebook OS is…. CentOS!

Systems Management

• Configuration Management
• CFengine for system management
• On Demand

Deployments

• Web Push – new code gets deployed to fb at least once a day.  Its a coordinated push, everyone is aware, notification happens to dev team.  Everyone sites on IRC during the push.  It is undestood by engineers and the rest of the company
  • push software built over on-demand control tools
  • code distributed via internal BitTorrent swarm
  • php gets compiled, the few hundred MB binary gets rapidly pushed bia bit torrent.
  • it takes one minute to push across the entire network
• Backend Deployments – only Engineering and Operations.  Engineers write, test and display
  • Quickly make performance decisions
  • Expose changes to subset of real traffic
  • No ‘commit and quit’
  • Deeply involved in moving services to production
  • Ops ‘embeded’ into engineering teams
• Heavy Change logging – pin pointing code to every push and change

Monitoring and Metrics of servers and performance at facebook

• Ganglia – aggregated metrics
  • fast
  • straightforward
  • nested grids & pools
  • over 5 million monitored metrics
• facebook inhouse monitoring system

Monitoring – facebook still uses Nagios!

To manage complexity and the number of alarms and systems monitoring the fb team uses aggregation.  Initially alarms were managed by email.

Scribe – high performance logging application.  Initially used syslog.  Also used Hadoop and Hive.

How does it work and gets done?

• clear delineation of dependencies and responsibilities
• Constant Failure
• Servers were the first line of defense, then started focusing on racks
• Now is focused on clusters.  Logical delineation based on function (web, db, feed, etc)
• Next stage is datacenters – what to do if a natural disaster strikes?
• Constant Communication – information is shared constantly.
  • IRC
  • lots of automated bots, get and set data
  • internal news updates
  • “Headers” on internal tools
  • Change log/feeds
• Small teams

Interesting fact – each fb server gets an update on average every eight minutes.

Busiest day for FB is day after halloween

URLS to check out:

facebook.com/engineering

facebook.com/opensource

We encourage you to start looking at your site’s speed (the tools above provide a great starting point) — not only to improve your ranking in search engines, but also to improve everyone’s experience on the Internet.

The post lists a number of tools everyone should be using already, such as YSlow, google’s own PageSpeed, online speed waterfall diagram tool and webmaster tools. Webmaster tools recently added a beta feature which provides data about your sites speed relative to other sites on the internet.

Here is a sample report:

webmaster tools page speed chart

Performance overview
On average, pages in your site take 6.3 seconds to load (updated on Apr 11, 2010). This is slower than 83% of sites. These estimates are of medium accuracy (between 100 and 1000 data points). The chart below shows how your site’s average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times.

Linus System admin blog will have a series on page speed improvement.  Stay tuned!

I’ll assume that you have installed apc already, if this is not the case this will probably be something as simple as running
pecl install apc
or downloading the archive from pecl and running:
phpize; ./configure; make; make install

The APC extension needs to be enabled either in php.ini or in one included file with a line like this:
extension=apc.so
there are many other parameters that apc can be fine tuned (see the official doc for more info), but without any other change, just with this line apc will be enabled on all the vhosts on the server.

Disabling some vhosts from using APC
- if we want to disable APC for a particular vhost we just have to add to the vhost config or to .htaccess:
php_flag apc.cache_by_default Off

Enabling APC only on some vhosts
- if we want to have APC disabled by default globally we will have in php.ini:

extension=apc.so
[apc]
apc.cache_by_default=0 # disable by default
... other apc settings...

and we will enable APC for the particular vhost config or using .htaccess using:
php_flag apc.cache_by_default On

Hopefully you found this post useful and this will give you a reason to use APC with more confidence knowing that you have the granularity to disable/enable it as needed in a shared environment.

#!/bin/bash

PID=`cat /var/run/sshd.pid`
ps -p $PID -o pid -o rss -o %mem -o cmd >> logname
exit

The output of log file will looks like this:

PID RSS %MEM CMD
2607  1036  0.0 /usr/sbin/sshd

Note the RSS column, it the values keep increasing with usage of application that would indicate a memory leak.

Another way to do the same thing is with pidstat which is a part of the sysstat package. The package can be installed via yum or aptitude but may not come with pidstat, if so download and install or build the latest version from here.

Pidstat is better way to track resources of an application because it has built in polling as well as combining process’s children statistics into the total. See pidstat man page options. In the script below we will use pidstat to track memory usage for sshd by polling 12 times every 5 minutes and e-mailing a report and writing to a log file. This script can also be run via cron.

#!/bin/bash
# track memory usage of sshd using pidstat and send report
# http://www.linuxsysadminblog.com - MaxV

export PID=/var/run/sshduse.pid
export TIMESTAMP=`date +%Y%m%d_%H%M%S`
export LOGDIR=/var/log/
export SSHD_LOG="${LOGDIR}sshd_memUsage_${TIMESTAMP}"
export SSHD_PID=`cat /var/run/sshd.pid`
export MAILTO=user@domain.com

if [ ! -e ${PID} ]; then

#create pid file
echo $$ > ${PID}

#log begin of script to /var/log/messages
/usr/bin/logger "Starting SSHD Memory Usage Tracker"

# pidstat portion, poll 12 times with 5 minutes apart
/usr/bin/pidstat -r -p ${SSHD_PID} 300 12 >> ${SSHD_LOG}

#e-mail report
mail -s "SSHD memory usage ${TIMESTAMP}" ${MAILTO} < ${SSHD_LOG}

#clean up pid file
if [ -f ${PID} ]; then
rm -rf ${PID}

#log end of script to /var/log/messages
/usr/bin/logger "Ending SSHD Memory Usage Tracker"
fi
exit 0
else

exit 0
fi

The output of this script:

Linux 2.6.18-6-686 (hostname)       06/16/09        _i686_  (2 CPU)

14:21:26          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
14:21:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:22:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:22:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:23:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:23:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:24:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:24:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:25:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:25:56         2564      0.00      0.00    4932   1108   0.05  sshd
14:26:26         2564      0.00      0.00    4932   1108   0.05  sshd
14:26:56         2564      0.47      0.00    4932   1108   0.05  sshd
14:27:26         2564      0.00      0.00    4932   1108   0.05  sshd
Average:         2564      0.04      0.00    4932   1108   0.05  sshd

From the output we can see that “RSS” values are not increasing as time progresses which means that it’s a solid piece of coding.

The good news is that if you have data that does not change very much the query cache can give you an enormous performance boost. It even bypasses the query optimizer so that if the query is complex even more cpu time is saved. Knowing this you can optimize your application by chopping complex queries into smaller queries that only use that data that never changes.

Of course there are some tricks to using the query cache. The first one is the size of the query cache. The default is 16MB which doesn’t do much. However, keep in mind that any memory assigned to the query cache is removed from another cache so it’s very important to strike a good balance. Of course the balance is very application dependent. The second parameter is the maximum allowed result set size. It really doesn’t do any good to allow 16MB result sets into the cache because it would take only one badly written query to flush out the entire cache. 1MB is standard but in my personal experience queries that return 1MB of results on a frequent basis usually indicate that the software needs to be optimized.

So when is the query cache a bad thing? Well, in short, when the cache gets flushed out all the time and only adds to the overhead it’s usually better to assign the memory to storage engine dependent cache. If there are constant updates and inserts to most of your tables it will invalidate the results in the query cache pretty quickly and assigning memory to it is a waste of resources.

Useful tools like MySQL Tuner will give some quick information about the efficiency of the query cache but I do think it is a bit quick in suggesting more memory for the cache.

• Look at the number of requests a page makes to the server
• Use yslow to measure page rendering (often a page performance is perceived, not just based on the server response time)

• Remove search, use alternate solutions such as Apache Solr or Google Search API
• Use CDN as much as possible
• Use Reverse Proxy Cache and memcache
• Obviously use drupal cache

Some other notes that are somewhat related to drupal performance and site performance management in a clustered hosting environment.

Manual updates and rollback

OLD WAY: tar, move/copy untar restart services

OLD WAY: rsync

BETTER WAY: Capistrano

Managing systems:

BETTER WAY: bcfg2

Monitoring Tools

• Capacity Load: analyzing trends, predicting load, checking results of configuration and software changes (cacti, munin)
• Failure: analyzing downtime, notification (nagios – using nrpe agents to monitor diverse services (do we use it this way?) , hyperin)

Use monitoring tools to closely observe cluster replication and cashing as the failures in this area are the most difficult to solve.

Temporary reporting servers
Create a one time copy of an existing production database server to run specific heavy reports. This is ideal for monthly reports since the server only needs to be up and running for several hours per month.

Backup database server
This is a backup solution where the server is only allocated once there is a problem with the primary server which makes a lot of sense because the client only pays for the server once it is used. One downside to this scenario is that the server has to created and loaded with the latest backup which will result in a decent amount of downtime but at least all of this can be automated. A bigger problem is the loss of data since the latest backup.For our high availability sites we have a standby database server replicating all changes from the master so we can switch over at a moment’s notice without losing any data.

Migrations
Performing a migration or a system upgrade usually brings some downtime. Promoting a standby system to primary creates a single point of failure so it makes sense to create a remporary standby of the standby.

Development branches and testing environments
For development branches we usually only need an extra database for a short amount of time although truth be told, those database are not very large in general so we tend to put them on the same development database server anyway. The same is true for testing and QA. These activities usually occur in cycles which means that they are very attractive targets for cloud based servers.

Alternative data center
Yes, it happened to us once that our datacenter went off line due to a very heavy attack. Instead of finding another data center for these eventualities it could be useful to have cloud based backup servers defined. However, this requires the extra effort of keeping these instances up to date for this eventuality. Additionally, DNS caching will stop the switch from being instantaneous. A geographical load balancing solution would be the answer to that but at that point the cost for preparing for this eventuality will have to be compared to the loss due to down time.

In the article below it shows how to patch the slow query log to track queries that last less than a second. Obviously you don’t want to have this running continually in production because the amount of logging would be enormous but in test environments or for a limited time in production this can be very useful. Be prepared to analyse some huge amounts of data though.

http://www.mysqlperformanceblog.com/2006/09/06/slow-query-log-analyzes-tools/