First off, this is great.  It should improve the speed of looking up the DNS info of many sites, and if the service takes off, it should take the load off your NS.

The focus on speed if very clear, the Google public DNS server lists this first as one of the advantages.  It also points to the speed problems caused by DNS latency.

Google Public DNS IP addresses

The Google Public DNS IP addresses are as follows:

• 8.8.8.8
• 8.8.4.4

Objectives:

1. Format a new EBS (10GB) and mount it on a running instance of private AMI (created on first activity – add link/ref to old post)
2. Setup a MySQL server with the datastore on EBS partition
3. Setup the partition(EBS) to start at boot time of AMI

Here, I will elaborate the steps (mostly commands) and some issues that I encountered along the way. I also included the script (below) that i used for attaching the EBS to AMI at boot time. Reference here. I will add an indicator on where i am running my commands, either on controling machine or on instance. On variables or values i assumed that you already know how to get them, the ec2-describe-instances/volume..etc. If the ec2 commands is not available on your system make sure you have the ec2 api tools or have your environment variables configured.
Objective #1: Format EBS and mount on a running instance

• Run instance of private ami and take note of the zone (default is us-east-1a – not sure )
  controlling machine$: ec2-run-instances -z us-east-1a --key YOURKEYPAIR ami-xxxxx
• Create ebs volume with 10GB size. Note the use of same zone so the volume can be attached to the instance above. Check the EBS docs for more details on Zones.
  controlling machine$: ec2-create-volume -z us-east-1a -s 10
• Attach the zone to your instance, ex: as /dev/sdh
  controlling machine$: ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111
• Login to your instance and format your ebs drive on /dev/sdh. It’s your choice on what filesystem to use. For my activity, i used xfs as i was advised that it is easier/faster to increase/shrink xfs filesystem compared to ext3 – and on the above reference xfs as used.
  controlling machine$: ssh -i ssh_key root@ec2.xxxxx.amazonaws.com
  (host may not be on this format, just refer to the details on your instance)
  instance$: yum install xfsprogs
instance$: modprobe xfs
instance$: mkfs.xfs /dev/sdh
• Mount the ebs volume.
  instance$: mount -t xfs /dev/sdh /ebs

Objective #2: Setup a MySQL server with the datastore on EBS partition

• Install mysql on your running instance, edit /etc/my.cnf and set the value for datadir to /ebs (my example), and start your MySQL.
  instance$: yum install mysql-server
instance$: vi /etc/my.cnf
instance$: /etc/init.d/mysqld start
• Create a sample database to test
  instance$: mysql
mysql> create database ebstest;
mysql> quit
instance$: ls /ebs/

Objective #3: Setup the partition(EBS) to start at boot time of AMI

• I was advised here to create an init script that will attach the ebs volume to my running instance and i was given a sample script (for debian) that i modified to my need (for CentOS) and added some stuff. I encountered several issues here as my init script failed to start correctly, like my environment variable is not available or incorrect paths etc. And was able to bundle four or five times. In short the script (below) does the automation, i only need to add this on my start up – so for the process, please check or continue reading the notes/comments on the script below. Btw, I just added the section to start MySQL inside the init script, but of course you can separate them.
• After creating a init script with the correct variables/filenames, bundle or create new AMI. Commands below are summary from a video tutorial – i forgot the link Run help for each command to get details on the options used, ex: ‘ec2-bundle-vol -h’.
  instance$: cd /mnt
instance$: mkdir ami
instance$: ec2-bundle-vol -d /mnt/ami -k /root/.ec2/pk.xxx.pem -c /root/.ec2/cert.xxx.pem -u xxxx-xxxx-xxxx
instance$: ec2-upload-bundle -b bucket1 -m /mnt/ami/image.manifest.xml -a XXXXXX -s xxxXXXXx
controlling machine$: ec2-register bucket1/image.manifest.xml
• Test your new AMI – run new instance and check if your ebs volume is attached – goodluck!

Init Script Here: mountebs

I based my instructions on previous post on Howto Get Started With Amazon EC2 API Tools, so I won’t give details on some steps. And this post will cover mainly the steps taken to complete my objectives above.

To start, I signed up for an account and enabled EC2 and S3 services, and generate X509 certificate. Next, I selected a test server running CentOS 5.3 with Cpanel and installed java (openjdk 1.6, using yum) as a requirement.

Then, download EC2 API Tools and extract to my working directory at /myhome/.ec2/ and upload your private key and x509 certificates. Don’t forget to follow the filename format of cert-xxx.pem and pk-xx.pem.

Export shell variables (posted on the previous post) and specify the correct private and x509 path. Then run source /myhome/.bashrc or open new terminal to load new environment variables.

Setup EC2 keypair. At first i used the certificate from different account but i got the error below:
Client.AuthFailure: AWS was not able to validate the provided access credentials
I searched for this error and one suggestion is to chmod your certificate and key files to 600 but it didn’t help me. My problem is on our account because one of my teammates changed our account password and probably generated new keys. Anyway, this is where i signed up for a new account and proceeded without issues.

Search for the AMIs to use. Following the steps listed on the instructions, I tried several AMI’s (start/stop processes). I observed some AMI’s took longer to start compare to others but i have no idea why . Btw, you can also search for AMI’s, and start/stop them from Amazon Management Console (EC2 Dashboard).

My next task is to create my private AMIs and here’s a good video tutorial on Customizing an existing AMI and create your own AMI from it. From this part that I need to setup my S3 bucket or directory to store my AMI. There’s a Firefox addon called S3Fox that my friend suggested but unfortunately i can’t install it on my Firefox due to some errors. I found and tried this BucketExplorer for creating my S3 Bucket. Btw, this one is commercial and you can try it for 30 days. I haven’t checked for other apps.

Back to creating my private AMI based on the above video, I ran into issue with ec2-bundle-vol command as it is not included on the AMI that i used, so i search for other AMIs that includes the EC2 Tools and found one from RightScale (CentOS5V1_10.img.manifest.xml).

After this i was able to complete my private AMI and start new instance from it using the above steps without any issues.

Moving Files:

• Copy Drupal file and preserve mode (ownerships, permissions, etc)
  Example: cp -rp drupal_source drupal_destination
  Review your directory permissions on sites/default/files, sites/default/files/civicrm, and other directories.
• Update references to Drupal url, path, and database details (name, user, pass, and host). Sample commands below using grep:
  find /path/to/drupal -type f -exec perl -pi -e "s/example.com/example2.com/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/public_html\/example/public_html\/example2/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/db_name/db_name2/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/db_user/db_user2/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/db_pass/db_pass2/g" {} \;

Moving Database/s:

Case 1: Combined CiviCRM and Drupal Database.

• Create sql dump of source database.
  Example: mysqldump -Q -udb_user -pdbpass db_name > db_name.sql
• Import to destination database.
  Example: mysql -udb_user2 -pdbpass2 db_name2 < db_name.sql
• Update references to Drupal url, path, and database details (name, user, pass, and host) of non-CiviCRM tables. You can use PhpMyAdmin to export this tables, then do the search/replace process on your local editor, and upload back the updates sql. You can also dump the tables from using command line (but you’ll have a long list of tables) and do the grep (same as above) and re-import the updated sql file.
• Update CiviCRM configurations from Drupal Admin section. You need to update the “Resource URLs” and “Directories”.
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Directories (or use the direct url: /civicrm/admin/setting/path?reset=1)
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Resource Urls (or use the direct url: /civicrm/admin/setting/url?reset=1)
• Optional: You can empty Sessions and Cache tables if you want.

Case 2: Separate CiviCRM and Drupal Database (recommended install for CiviCRM).
Process for this setup is almost the same as Case 1, the difference is on the import process for databases. I’ll just provide the complete info below.

• Create sql dump of source databases.
  Examples:
  mysqldump -Q -udb_user -pdbpass db_name_drupal > db_name_drupal.sql
mysqldump -Q -udb_user -pdbpass db_name_civicrm > db_name_civicrm.sql

• Import directly the CiviCRM database.
  Example: mysql -udb_user2 -pdbpass2 db_name2_civicrm < db_name_civicrm.sql
• Update CiviCRM configurations from Drupal Admin section. You need to update the “Resource URLs” and “Directories”.
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Directories (or use the direct url: /civicrm/admin/setting/path?reset=1)
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Resource Urls (or use the direct url: /civicrm/admin/setting/url?reset=1)
• Update references to Drupal url, path, and database details (name, user, pass, and host) of Drupal database dump.
  perl -pi -e "s/example.com/example2.com/g" db_name_drupal.sql
perl -pi -e "s/public_html\/example/public_html\/example2/g" db_name_drupal.sql
• Import the Drupal database.
  Example: mysql -udb_user2 -pdbpass2 db_name2_drupal < db_name_drupal.sql
• Optional: You can empty Sessions and Cache tables if you want.

That’s All!

Getting Started

First of all you will need one Amazon AWS account and enable the EC2 service; in case you don’t have this already now is the time to create your account. Once you do that you can safely return to this doc

Once you have your account working, while still on the aws site, go and create a new X.509 certificate (under the AWS Access Identifiers page, in the X.509 certificate section near the bottom, click Create New). Once this is done, you will want to download locally the private key file and X.509 certificate.

EC2 API tools

Next you will have to download and install the Amazon EC2 api tools on one system (controlling machine) that will be used to start your EC2 army of servers, and control their usage. You will want to use the latest version (2009-05-15 at this time) as it will support all the features Amazon is offering for the EC2 service.

The only real dependency of the EC2 API tools is java (at least version 1.5) so we will want to install that first. If you are running debian you can easily do this just by running (for lenny):
aptitude install sun-java6-jre
while for etch you will have to use: aptitude install sun-java5-jre
For other distributions you can either use their internal packaging mechanism (in case they provide sun-java packages) or just download the binary from sun and install it manually.

Extract the EC2 APi tools (it is a zip archive called ec2-api-tools.zip) and move it under a folder of your preferece. I like to use ~/.ec2 for this, but you can use any folder you prefer. Also copy the private key and X.509 certificate in the same directory. Those files will look like cert-xxx.pem and pk-xxx.pem.

Next we will have to export some shell variables. A good place to put this is in ~/.bashrc:
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=$EC2_HOME/pk-xxx.pem
export EC2_CERT=$EC2_HOME/cert-xxx.pem
#Java home for debian default install path:
export JAVA_HOME=/usr
#add ec2 tools to default path
export PATH=~/.ec2/bin:$PATH

Finally source the file to have the changes active in your current shell session:
source ~/.bashrc
or just open a new shell before starting to use the API tools.

EC2 Keypair

We will need to create one keypair that will be used to connect using ssh to the EC2 instances we will be using. We will use the ec2-add-keypair utility to create the key and register it with amazon:
ec2-add-keypair my-keypair
This will print out the private key that we will have to save in a file:
cat > ~/.ec2/id_rsa-my-keypair
#paste the private key content
sudo chmod 600 ~/.ec2/id_rsa-my-keypair

Running your first EC2 instance

Amazon EC2 uses the concept of AMIs = Amazon Machine Images. Any EC2 instance is started from one AMI. You can either use standard, public AMIs or create and customize your own private images. Creating or modifying existing AMIs is beyond the scope of this article, but just as a general information this is done using special AMI tools. Also before building your AMI you will want to ensure if you want to use a ’small’ type of image (i386 os) or a ‘large’ type of instance (64bit os). These are described under http://aws.amazon.com/ec2/instance-types/

For the scope of our article we will find a standard public image and start one instance of it to see that all is working properly with the EC2 api tools. You can see all the public images using:
ec2-describe-images -a
(over 2,300 images ). You should probably grep the result to get any useful information. There are many good public images to use, like for example the alestic ones (for debian and ubuntu)
Having the AMI id of the image we want to use we are ready to start our fist EC2 instance:
ec2-run-instances ami-e348af8a -k my-keypair
that will start a small instance with a 32bit debian lenny server instance from alestic.com.

ec2-describe-instances
- this will describe the status of all the running instances, with their hostname, instance id, etc.

ec2-authorize default -p 22
- in order to connect to your instance you will need to customize the ‘default’ firewall rules for your account. The above rule will allow ssh on port 22 from anywhere. If you want to open http traffic you will have to add a rule like this:
ec2-authorize default -p 80

Finally we can ssh to the ec2 instance using:
ssh -i ~/.ec2/id_rsa-my-keypair root@ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com
where ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com is the actual hostname of the instance as obtained from ec2-describe-instances.

Note: don’t forget to stop your instance when you no longer need it. EC2 is a service paid as you use, hence if you forget your instance running you will be billed for it . You can do this by running shutdown inside the instance or by using:
ec2-terminate-instances i-yourinstance
and verify with ec2-describe-instances that the instance is indeed stopped.

Next step is to create/customize your own EC2 AMI images based on your needs. This will be covered in a future article. Hopefully you found this article useful, and it will get you on track quickly with Amazon EC2 api tools.

MySQL conference notes on cloud computing shoot out text summary atranscript

Lew Tucker, Cloud CTO, sun
Monty Taylor, MySQL Drissel Geek, Sun
Jeremy Zawodny, MySQL hacker, craigslist
Chander Kant, CEO, Zmanda
Thorsten von Ficon, CTO, Rightscale
Prashant Malik, Cassandra Dude, Facebook
Mike Culver, Evangerlist, Amazon Web Services

Some interesting subjects touched upon were:

elasticity – allowing even a small company to shoot for the moon without shooting itself in the foot
like utilities, much like electricity – you dont think about the electricyt company running out of electricity
new way of packaging the technology with a pay as you go model, way to provision your application

Different Types of Clouds
Layers of Clouds
Amazon like
Google
SaaS

Who is the cloud for?
even ERP in the cloud?

How would an existing application benefit from the cloud?
- scaling an application
- leverage the collective scalability of the cloud
- forklifting an application out of datacenter or in house colo
- scalability testing?

Cloud adoption barriers
- privacy
- performance
- network latency
- trust and privacy
- mindset on owning your own datacenter
- competition (lack of)

What applications fit best into the cloud
- own

Business Opportunities?
- dev pay – the customer signs up for SaaS or Software as an annuity
- learn how to do performance tuning and optimization and do that for cloud infrastructures (41:50)
-

That’s when I thought – wow – this term really has penetrated into the mainstream.  Cloud Computing has crossed the chasm, except that it seems to be more of a catch phrase for the latest technical fad, except that no one seems to be talking about exactly the same thing.  

As if on cue the next morning the Wall Street Journal:  The Internet Industry Is on a Cloud — Whatever That May Mean by Geoffrey A. Fowler and Ben Worthen landed on my lap.  Ok, it landed on my porch, and then I read it.

 I hope the Journal doesnt mind if I borrow their image of the cloud – very fitting for the article.  Basically its a tale of marketers trying to – or should I say – fighting for integration of the term into its product jargon.  

Some of my favorite quotes from the piece are bleow.  I recommended the quick article as a nice distraction, but don’t expect to come away being clear on the term “Cloud Computing”.  

“I have no idea what anyone is talking about,” said Oracle Corp. Chief Executive Larry Ellison, when talking about cloud computing at a financial analyst conference in September. “It’s really just complete gibberish. What is it?” He added: “When is this idiocy going to stop?”

 

In the full fiscal year since Salesforce started using the term cloud computing, its revenue grew 44%. “I think it’s the most powerful term in the industry,” 

 

Dell Inc. applied to trademark the term cloud computing last year. The U.S. Patent and Trademark Office initially approved the application. But it changed its mind in response to an outburst of criticism, including from bloggers incensed that the term could fall under one company’s control.

Additionally I have to say that the scripting abilities offered by RightScale seem quite extensive and very useful.The scripting allows for some very powerful ways of setting up servers. The prices associated with those services however seem a bit steep compared to the bare bones services offered by Amazon.

Outages pose problems for Google as it tries to persuade companies to buy into its cloud-computing vision, in which applications are hosted on the Internet rather than on corporate computers. But Google argues its service availability is competitive with most organizations’ abilities to run their own e-mail servers.

Clearly the google cloud isn’t going to be the only one having an outage from time to time.  It seems to me that while still in the infancy, these services are vulnerable to unexpected problems, kind of like the famous first internet worm – the Morris worm -  that brought the internet to its knees back in the 1980s.

While cloud services offer a lot of promise, and overall should offer a better level of redundancy and up time, this shows that the cloud is also not immune to some down time.

Here is the link to the original Google cloud problem blog post.

• the desktop version will have a new look (more beautiful)
• on the server side as everybody these days Ubuntu’s future will be targeting the “cloud”. They will have official supported Ubuntu Amazon EC2 AMIs, as ready-to-run appliances.  Also they will support the open source project Eucalyptus that enables you to create an EC2-style cloud using your own hardware. Eucalyptus, will be included in ubuntu repos allowing users to build and manage private clouds more easily.

For full details: https://lists.ubuntu.com/archives/ubuntu-devel-announce/2009-February/000536.html