Official Google Blog: This is your pilot speaking. Now, about that holding pattern…

Ok, so I checked out my RSS feeds and went over to google blog to see what the hoopla was about today after slowdown. Yes, I had pings from folks asking me if google was down for them or just me, but I really find the backlinks interesting… from funny to pathetic.

I think uptime still matters.

See for yoursleves (my emphasis):

Google Outage Caused by Asian “Traffic Jam” | John Paczkowski …
If the Web has does have a single point of failure, you’d think it was Google given all the outcry over the the outages suffered by some of the company’s services Thursday. Something went wrong at the company this morning and whatever …
Posted by John Paczkowski at 12:23

Google Slow (or Down) for Some
Thursday, May 14, 2009. Google Slow (or Down) for Some. Some of us are having problems accessing google.com, YouTube, Gmail and others. [This post may update if there's further info.] Update: And it seems to be back up now (18:15 CET). …
Posted by Philipp Lenssen at 10:08

Google Slow, Twitterati Hysterical
UPDATED: Google appears to be having problems across its Gmail, search and even its Blogger platforms, judging by complaints on …
Posted by Stacey Higginbotham at 09:48

It’s Down! The Day Google Stood Still (Updated) – ReadWriteWeb
We have seen our fair share of failures from web based products, but this morning, for a large number of users (at least in the US), it looks …
Posted by Frederic Lardinois at 09:32

Major Google Outages Today: #GoogleFail Or #AT&T Fail?
A bunch of GoogleGoogle reviews services have been failing this morning, and we’ve been trying to figure out why. The hashtag #googlefail on TwitterTwitter.
Posted by Adam Ostrow at 09:25

Google Services Go Down For Many
Currently, many people who use Google’s services, including web search, Gmail, Google Reader and other products are either down or incredibly slow for some.
Posted by Barry Schwartz at 09:15

La panne de Google: une erreur d’aiguillage – Media & Pub – E24.fr
Le moteur de recherche a connu de sérieux problèmes techniques entre 17h et 18h ce jeudi. Des milliers d’internautes ont témoigné sur Twitter des difficultés rencontrées sur Google.
Posted by at 09:08

Google Stumbles, Internet Breaks A Leg
Recent Posts. Google Stumbles, Internet Breaks A Leg · Beer Monday: Redhook’s Slim Chance · Friday Gallimaufry: Migratory Birds · Moms On The Net · Beer Monday: New Glarus Brewing · The Pale Blue Dot · Devo Was Right About Everything …
Posted by forbes blogger at 08:35

Ajax Girl » Blog Archive » Google’s Outage Was Asia’s Fault
Written by on May 14th, 2009 in Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site. Google finally has an explanation for its …
Posted by at 07:34

Tech Science | SearchBeat.com Shout-Out Blog
Top Technology, Computer, Internet and Science News Latest Science News from Around the Web Scientific.
Posted by keithco at 23:12

Tech Central – Times Online – WBLG: Problems with Google today …

On the hardware side you need a DV camera with firewire port (IEEE 1394), firewire port equipped Pentium 4 or equivalent pc or laptop with loaded with Fedora 9 or 10 and a  firewire cable to connect camera to the computer. For software we will only need dvgrab and VLC

Install dvgrab from Fedora update repo:
yum install dvgrab

Install vlc from rpmfusion repo:
sudo rpm -ivh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm
sudo yum install vlc

Set DV camera audio to 16bit (default is 12 bit) to avoid garbled audio.

Turn on and connect video camera to computer and you should see something like this in /var/log/dmesg:
firewire_core: created device fw1: GUID 0800460102721e20, S100

To test that we are able to grab video/audio from camera and display in VLC player pipe output of dvgrab into vlc.

sudo dvgrab - -noavc -nostop | vlc - --no-sub-autodetect-file :demux=rawdv
After issuing this command you should see a 720×480 video feed with 16bit 48000Hz audio stream in vlc on your desktop

Now we setup vlc as a streaming server so that we can view the video/audio when away. Streaming a 720×480 video stream is a bit overkill as the video quality on the DV camera is pretty good when vlc streams video at lower resolutions like 320×240, I also reduce the audio quality to save on bandwidth. Here I used “cvlc” or command vlc to avoid opening a vlc window and “&” to put process into background.

sudo dvgrab - -noavc -nostop | cvlc - --no-sub-autodetect-file :demux=rawdv --sout '#transcode{vcodec=mp4v,vb=600,acodec=mp3,ab=56,scale=1,width=320,height=240,channels=2}:duplicate{dst=std{access=http,mux=ts,dst=192.168.1.102:3323}}' &

To view the feed locally via vlc open a http network location on ip and port you specified in the dst= section of the command above.

To view your feed from the Internet you will need to either configure vlc to stream on an outside interface or configure port forwarding.

Temporary reporting servers
Create a one time copy of an existing production database server to run specific heavy reports. This is ideal for monthly reports since the server only needs to be up and running for several hours per month.

Backup database server
This is a backup solution where the server is only allocated once there is a problem with the primary server which makes a lot of sense because the client only pays for the server once it is used. One downside to this scenario is that the server has to created and loaded with the latest backup which will result in a decent amount of downtime but at least all of this can be automated. A bigger problem is the loss of data since the latest backup.For our high availability sites we have a standby database server replicating all changes from the master so we can switch over at a moment’s notice without losing any data.

Migrations
Performing a migration or a system upgrade usually brings some downtime. Promoting a standby system to primary creates a single point of failure so it makes sense to create a remporary standby of the standby.

Development branches and testing environments
For development branches we usually only need an extra database for a short amount of time although truth be told, those database are not very large in general so we tend to put them on the same development database server anyway. The same is true for testing and QA. These activities usually occur in cycles which means that they are very attractive targets for cloud based servers.

Alternative data center
Yes, it happened to us once that our datacenter went off line due to a very heavy attack. Instead of finding another data center for these eventualities it could be useful to have cloud based backup servers defined. However, this requires the extra effort of keeping these instances up to date for this eventuality. Additionally, DNS caching will stop the switch from being instantaneous. A geographical load balancing solution would be the answer to that but at that point the cost for preparing for this eventuality will have to be compared to the loss due to down time.

I always thought that they just weren’t confident in their system folks and scoffed at this policy as it always made our deadlines shorter.

I think this story convinced me that doing production work these days on the bussiest web days is not a good idea.  Maybe microsoft should have borrowed a page from the mainframe policy manual – don’t do system updates on black monday or black friday as it may cause system outage.

This story: Microsoft Says Sorry For Black Friday Cashback Outage

For Internet users, Black Friday was supposed to be about buying and cashing back, but Microsoft’s Live Search cashback machine apparently broke down just as customers “barged in” to make some early morning purchases.

According to a blog posting, the unexpected outage occurred due to a significant spike in traffic, which caused the system to go down for several hours. It took quite a while for it to come back to life, but apparently that was related to investigating the issue and rebuilding and deploying the databases and indexes that support Microsoft Live Search Cashback.

rm -fr ${DIR}/${SUBDIR}

Hint: add the following alias for all users to prevent this: alias rm=’rm –preserve-root”

We were lucky in two ways. First off, this was not a production server, just a development and testing server and secondly the databases and web sites on that server were unaffected. That’s where the good news ended and Murphy’s Law kicked in. A couple of days before we found that our backup server had a corrupt filesystem on its RAID array. Since we did not have enough space available on other servers to place all the backups on other servers we temporarily suspended (you guessed it) the backups of the development and testing server.

To undelete or not to undelete

To get back up and running we immediately closed off access to the server and considered how we could recover the deleted files. Unfortunately undeleting files on an ext3 file system can only be done under certain circumstances. If the deleted files are still opened by some process the lsof utility can help as is documented on some web sites (just Google “ext3 undelete lsof”) but for larger scale undeletes the first step is to create an image of the partition in question. That image can then be searched for inode entries which can be very useful for finding specific files. However, if you want perform a more general undelete this method is a lot less useful because the file names will not be recovered.

Apart from the limited usefulness that creating this image would yield it would have taken several hours to complete during which development and testing would be at a standstill. We decided not to do this and instead take our losses instead. It is important to note what data we were losing at that point. Among the missing directories were some binary directories (/usr/sbin and such) which were easily recoverable by copying them from similarly configured servers. The most important missing data was the version control repository and a custom scripts directory. All the history of changes in the repository was lost but the latest state of the code was easily restored. We copied the latest code from the developer who had last performed a complete update (which is a part of the daily development process) and put that code into the repository again. Since the versions did not match up anymore after that (all code versions were reinitialized) all developers had to retrieve the complete set of code files again and copy their latest versions over it to keep working.

Although this is definitely a loss for us the impact is limited by the fact that we keep copies of all released code. These copies were unaffected on the server in question but are also present on other servers. If need be we can go through that history to track down a change, but the comments are gone and it’s not a process the developers can do themselves.

Rebooting the server

After all this we were left with one task, rebooting that server. Since we did not know exactly what got deleted this might give us some severe problems. This was scheduled for a quiet night with several system admins present. Unfortunately our hand was forced when a change in the iptables configuration caused a kernel panic. Rebooting the server revealed several more problems, the main one being the privileges on the /tmp directory. This resulted in Apache not being able to write session info there and MySQL not being able to write temporary data either. This was quickly solved of course. Without going into too many details the final action we took was to update our Cpanel. This reinstalled many missing scripts and binaries.

I bet you’re wondering why we don’t use off site backups. Well, we do actually. The problem is that this involves copying many gigabytes over a limited line so we made a selection of what needed to be copied and we focused mainly on all our production servers. The main purpose of our off site backups is to recover production servers in case our data center becomes unavailable.

Conclusions

It’s been an annoying experience and it’s hard to draw positive lessons from mr. Murphy’s teachings but all in all it could have been a lot worse. Production was not down or affected and even testing and development impact was pretty limited. The main things on our agenda after this are to review our backup strategy for essential locations and reviewing the use of root privileges on our servers. Although we use non-root users most of the time there are tasks that are made a lot quicker by changing to root. We all know the danger of this and need to be a lot more aware of it.

• Site Down – those poor bastards (system admins) their life must suck right now
• Site Down – SEE!!! It happens to them too! No one is perfect! No one is immune!

Here is a story that elicited these feelings in today’s Slashdot stories:

+--------------------------------------------------------------------+
| US Amazon.com Website Down For Over 1 Hour |
| from the there-goes-the-bottom-line dept. |
| posted by ScuttleMonkey on Friday June 06, @15:10 (The Internet) |
| http://tech.slashdot.org/article.pl?sid=08/06/06/199211 |
+--------------------------------------------------------------------+

CorporalKlinger writes "CNET News is reporting that Amazon's US website, Amazon.com, has been unreachable since 10:30 AM PDT today. As of posting, visiting www.amazon.com produces an 'Http/1.1 Service Unavailable' message. According to CNET, "Based on last quarter's revenue of $4.13 billion, a full-scale global outage would cost Amazon more than [0]$31,000 per minute on average." Some of Amazon's international websites still appear to be working, and some pages on the US Amazon.com site load if accessed using HTTPS instead of HTTP."

Discuss this story at:
http://tech.slashdot.org/comments.pl?sid=08/06/06/199211