The tool that I’m talking about is XtraBackup by Percona. This tool is originally meant to accompany the XtraDB storage engine which in itself is a patched version of InnoDB. XtraBackup will create online non-blocking backups for both XtraDB and InnoDB databases and best of all, it’s free.

For those of you who are not that familiar with MySQL backups, the standard way of doing backups is with mysqldump. This can be done with the database online but it blocks the tables it’s backing up which is not acceptable for production environments. It also takes a good amount of time to restore a mysqldump since it writes out everything as SQL statements which then have to be processed again. A binary copy is much faster to restore but commonly requires the server to be stopped. The best alternative is to create an LVM snapshot of the binary files but this requires LVM to be set up and enough disk space to perform the LVM snapshot. All in all it’s nice to have a free alternative although I have to add the footnote that I haven’t tested it on any decently sized database to check what the performance impact is.

To fix this you can use the CHANGE MASTER command to set the slave to the master bin log file and position that shows up in the SHOW SLAVE STATUS output. Make sure you use the Relay_Master_Log_File and Exec_Master_Log_Pos fields since they indicate what position in the master binlog the slave actually thought it was executing. Keep in mind that corruption and its effects are hard to predict. It will definitely be useful to compare the master and slave afterward using the MaatKit tools.

As some more background, the server log will be probably show and error like this to indicate there was a network error:
[ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)

And finally, if you do read the entire bug thread you will notice that the original developer of MySQL also has an opinion on this.

The issue that we encountered was that some statistics like the InnoDB buffer pool activity were not displaying anything for one server. Other servers were displaying it just fine and other statistics for that server were also fine.

Among other things the SHOW ENGINE INNODB STATUS command shows deadlock information pertaining to the last deadlock that the InnoDB engine encountered. In some cases this information will be quite extensive and this causes a problem. The output of this command is one giant text field with a limit of 64KB. If the deadlock information is very large other information will get cut off which means certain statistics are lost. The easy fix for this is to restart the database server but in case this is not an option you can always use the innotop utility to wipe the deadlock information by causing a small deadlock.

MySQL conference notes on cloud computing shoot out text summary atranscript

Lew Tucker, Cloud CTO, sun
Monty Taylor, MySQL Drissel Geek, Sun
Jeremy Zawodny, MySQL hacker, craigslist
Chander Kant, CEO, Zmanda
Thorsten von Ficon, CTO, Rightscale
Prashant Malik, Cassandra Dude, Facebook
Mike Culver, Evangerlist, Amazon Web Services

Some interesting subjects touched upon were:

elasticity – allowing even a small company to shoot for the moon without shooting itself in the foot
like utilities, much like electricity – you dont think about the electricyt company running out of electricity
new way of packaging the technology with a pay as you go model, way to provision your application

Different Types of Clouds
Layers of Clouds
Amazon like
Google
SaaS

Who is the cloud for?
even ERP in the cloud?

How would an existing application benefit from the cloud?
- scaling an application
- leverage the collective scalability of the cloud
- forklifting an application out of datacenter or in house colo
- scalability testing?

Cloud adoption barriers
- privacy
- performance
- network latency
- trust and privacy
- mindset on owning your own datacenter
- competition (lack of)

What applications fit best into the cloud
- own

Business Opportunities?
- dev pay – the customer signs up for SaaS or Software as an annuity
- learn how to do performance tuning and optimization and do that for cloud infrastructures (41:50)
-

The good news is that if you have data that does not change very much the query cache can give you an enormous performance boost. It even bypasses the query optimizer so that if the query is complex even more cpu time is saved. Knowing this you can optimize your application by chopping complex queries into smaller queries that only use that data that never changes.

Of course there are some tricks to using the query cache. The first one is the size of the query cache. The default is 16MB which doesn’t do much. However, keep in mind that any memory assigned to the query cache is removed from another cache so it’s very important to strike a good balance. Of course the balance is very application dependent. The second parameter is the maximum allowed result set size. It really doesn’t do any good to allow 16MB result sets into the cache because it would take only one badly written query to flush out the entire cache. 1MB is standard but in my personal experience queries that return 1MB of results on a frequent basis usually indicate that the software needs to be optimized.

So when is the query cache a bad thing? Well, in short, when the cache gets flushed out all the time and only adds to the overhead it’s usually better to assign the memory to storage engine dependent cache. If there are constant updates and inserts to most of your tables it will invalidate the results in the query cache pretty quickly and assigning memory to it is a waste of resources.

Useful tools like MySQL Tuner will give some quick information about the efficiency of the query cache but I do think it is a bit quick in suggesting more memory for the cache.

We use MySQL extnsivly and for very high availability applications so naturally there was a lot of discussion about the future of MySQL.  We believe that it will be for the better.  I hope the MySQL folks who created this super popular database feel the same.

But, how would you feel if you got this email?  Here is the copy.

This is one of the toughest emails I’ve ever had to write.

It’s also one of the most hopeful about Sun’s future in the industry.

For 27 years, Sun has stood for courage, innovation, a willingness to blaze trails, to envision and engineer the future. No matter our ups and downs, we’ve remained committed to those ideals, and to the R&D that’s allowed us to differentiate. We’ve committed to decade-long pursuits, from the evolution of one of the world’s most powerful datacenter operating systems, to one of the world’s most advanced multi-core microelectronics. We’ve never walked away from the wholesale reinvention of business models, the redefinition of technology boundaries or the pursuit of new routes to market.

Because of the unparalleled talent at Sun, we’ve also fueled entire industries with our people and technologies, and fostered extraordinary companies and market successes. Our products and services have driven the discovery of new drugs, transformed social media, and created a better understanding of the world and marketplace around us. All, while we’ve undergone a near constant transformation in the face of a rapidly changing marketplace and global economy. We’ve never walked away from a challenge – or an opportunity.

So today we take another step forward in our journey, but along a different path – by announcing that this weekend, our board of directors and I approved the acquisition of Sun Microsystems by the Oracle Corporation for $9.50/share in cash.

All members of the board present at the meeting to review the transaction voted for it with enthusiasm, and the transaction stands to utterly transform the marketplace – bringing together two companies with a long history of working together to create a newly unified vision of the future.

Oracle’s interest in Sun is very clear – they aspire to help customers simplify the development, deployment and operation of high value business systems, from applications all the way to datacenters. By acquiring Sun, Oracle will be well positioned to help customers solve the most complex technology problems related to running a business.

To me, this proposed acquisition totally redefines the industry, resetting the competitive landscape by creating a company with great reach, expertise and innovation. A combined Oracle/Sun will be capable of cultivating one of the world’s most vibrant and far reaching developer communities, accelerating the convergence of storage, networking and computing, and delivering one of the world’s most powerful and complete portfolios of business and technical software.

I do not consider the announcement to be the end of the road, not by any stretch of the imagination. I believe this is the first step down a different path, one that takes us and our innovations to an even broader market, one that ensures the ubiquitous role we play in the world around us. The deal was announced today, and, after regulatory review and shareholder approval, will take some months to close – until that close occurs, however, we are a separate company, operating independently. No matter how long it takes, the world changed starting today.

But it’s important to note it’s not the acquisition that’s changing the world – it’s the people that fuel both companies. Having spent a considerable amount of time talking to Oracle, let me assure you they are single minded in their focus on the one asset that doesn’t appear in our financial statements: our people. That’s their highest priority – creating an inviting and compelling environment in which our brightest minds can continue to invent and deliver the future.

Thank you for everything you’ve done over the years, and for everything you will do in the future to carry the business forward. I’m incredibly proud of this company and what we’ve accomplished together.

Details will be forthcoming as we work together on the integration planning process.

Jonathan

My favorites are the server swap and a Google provided Personal Transportation Device (PTD).

These are great videos providing a glimpse into the usually locked down giant data centers housing thousands of servers.  This is how its done!

That was the summary video overview of a google data center – container based.

Below are a few more videos from the above mentioned Efficency Data Center Summit.  Here is part one overview the google green grid.

Here is the google provided summary of the event, and the following part 2 and 3 videos of the event.

On April 1^st, we hosted leaders of the IT industry to discuss best practices for improving data center efficiency. The day was spent discussing how to make significant reductions in resource use while meeting service requirements. Saving electricity and water is not just good for the environment, it makes good business sense too. Being “green” reduces operating costs and can keep our industry competitive; it is the economic advantage that makes efficiency truly sustainable.

Temporary reporting servers
Create a one time copy of an existing production database server to run specific heavy reports. This is ideal for monthly reports since the server only needs to be up and running for several hours per month.

Backup database server
This is a backup solution where the server is only allocated once there is a problem with the primary server which makes a lot of sense because the client only pays for the server once it is used. One downside to this scenario is that the server has to created and loaded with the latest backup which will result in a decent amount of downtime but at least all of this can be automated. A bigger problem is the loss of data since the latest backup.For our high availability sites we have a standby database server replicating all changes from the master so we can switch over at a moment’s notice without losing any data.

Migrations
Performing a migration or a system upgrade usually brings some downtime. Promoting a standby system to primary creates a single point of failure so it makes sense to create a remporary standby of the standby.

Development branches and testing environments
For development branches we usually only need an extra database for a short amount of time although truth be told, those database are not very large in general so we tend to put them on the same development database server anyway. The same is true for testing and QA. These activities usually occur in cycles which means that they are very attractive targets for cloud based servers.

Alternative data center
Yes, it happened to us once that our datacenter went off line due to a very heavy attack. Instead of finding another data center for these eventualities it could be useful to have cloud based backup servers defined. However, this requires the extra effort of keeping these instances up to date for this eventuality. Additionally, DNS caching will stop the switch from being instantaneous. A geographical load balancing solution would be the answer to that but at that point the cost for preparing for this eventuality will have to be compared to the loss due to down time.

In the article below it shows how to patch the slow query log to track queries that last less than a second. Obviously you don’t want to have this running continually in production because the amount of logging would be enormous but in test environments or for a limited time in production this can be very useful. Be prepared to analyse some huge amounts of data though.

http://www.mysqlperformanceblog.com/2006/09/06/slow-query-log-analyzes-tools/

On any weekday around lunch time network traffic between web and data servers was around 80Mb/s. On a 100Mb/s connection that is dangerously high and to address this we’ve been in the process of modifying queries that do a SELECT *. There really hardly ever is a reason to do a SELECT * except when you have very flexible code that automatically deals with extra columns. That was not the case with this application.

On Cyber Monday we found that traffic for this shop was touching 100Mb/s and web site performance really suffered. As a temporary measure we switched the database server to a 1Gb/s connection but both web servers stayed on 100Mb/s connections.

Looking at the slow query log and mtop revealed a ton of similar queries:

# User@Host: username[username] @  [10.0.0.123]
# Query_time: 2  Lock_time: 0  Rows_sent: 1948  Rows_examined: 2047
SELECT * FROM sites, sites_to_bundle WHERE sites_to_bundle.sites_id = sites.sites_id AND sites.site_name ='shop1';

In and by them selves these queries look pretty harmless. 2000 rows is not that much. Maybe more than is needed but not enough to choke up the network either. The problem turns out to be that this query was executed several times for each page in the check out process. The database server was not overloaded since it had the whole result set in its cache, but it had to transfer 2MB of data for each request. When a developer investigated it turned out that only one 2 digit value of the entire result set was used.

We rewrote query and pushed to production on that same day. Yes, I know that a previous blog post tells us not to do that but in this case I’m glad we did. Network traffic dropped below 10Mb/s and the web site was flying. Below is the Cacti graph that shows that difference. Shortly after 15:00 we implemented the optimization and traffic dropped dramatically.

Network traffic decrease after query optimization