Since the mail server for these accounts are pointing to other servers, I deleted all the email accounts, forwarders, mailinglists, etc. After this I’ve seen sending error/failure messages from mail logs of the said accounts It shows ‘fixed_login authenticator failed for hostxx’ [535 Incorrect authentication data].

Also this release features many bug fixes and security updates, and should be an easy upgrade for users running centos5.x:
yum update

For the full list of packages changed/added please see the centos5.4 release notes: http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.4

In order to enable error reporting for your php script or application include inside your code the following lines:
error_reporting(E_ALL);
ini_set("display_errors", 1);
and this will result in displaying in the browser any errors your application might have.

ps: once you are done with this and fixed the issue, don’t forget to remove the error reporting lines, as we don’t want our users/clients to see errors in the browser in case something went wrong.

There are no major changes in this update, mostly bug fixes and security fixes, and it should be a quick and easy upgrade for most people still running the 4.x branch (you should really consider upgrading to 5.x ).

Objectives:

1. Format a new EBS (10GB) and mount it on a running instance of private AMI (created on first activity – add link/ref to old post)
2. Setup a MySQL server with the datastore on EBS partition
3. Setup the partition(EBS) to start at boot time of AMI

Here, I will elaborate the steps (mostly commands) and some issues that I encountered along the way. I also included the script (below) that i used for attaching the EBS to AMI at boot time. Reference here. I will add an indicator on where i am running my commands, either on controling machine or on instance. On variables or values i assumed that you already know how to get them, the ec2-describe-instances/volume..etc. If the ec2 commands is not available on your system make sure you have the ec2 api tools or have your environment variables configured.
Objective #1: Format EBS and mount on a running instance

• Run instance of private ami and take note of the zone (default is us-east-1a – not sure )
  controlling machine$: ec2-run-instances -z us-east-1a --key YOURKEYPAIR ami-xxxxx
• Create ebs volume with 10GB size. Note the use of same zone so the volume can be attached to the instance above. Check the EBS docs for more details on Zones.
  controlling machine$: ec2-create-volume -z us-east-1a -s 10
• Attach the zone to your instance, ex: as /dev/sdh
  controlling machine$: ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111
• Login to your instance and format your ebs drive on /dev/sdh. It’s your choice on what filesystem to use. For my activity, i used xfs as i was advised that it is easier/faster to increase/shrink xfs filesystem compared to ext3 – and on the above reference xfs as used.
  controlling machine$: ssh -i ssh_key root@ec2.xxxxx.amazonaws.com
  (host may not be on this format, just refer to the details on your instance)
  instance$: yum install xfsprogs
instance$: modprobe xfs
instance$: mkfs.xfs /dev/sdh
• Mount the ebs volume.
  instance$: mount -t xfs /dev/sdh /ebs

Objective #2: Setup a MySQL server with the datastore on EBS partition

• Install mysql on your running instance, edit /etc/my.cnf and set the value for datadir to /ebs (my example), and start your MySQL.
  instance$: yum install mysql-server
instance$: vi /etc/my.cnf
instance$: /etc/init.d/mysqld start
• Create a sample database to test
  instance$: mysql
mysql> create database ebstest;
mysql> quit
instance$: ls /ebs/

Objective #3: Setup the partition(EBS) to start at boot time of AMI

• I was advised here to create an init script that will attach the ebs volume to my running instance and i was given a sample script (for debian) that i modified to my need (for CentOS) and added some stuff. I encountered several issues here as my init script failed to start correctly, like my environment variable is not available or incorrect paths etc. And was able to bundle four or five times. In short the script (below) does the automation, i only need to add this on my start up – so for the process, please check or continue reading the notes/comments on the script below. Btw, I just added the section to start MySQL inside the init script, but of course you can separate them.
• After creating a init script with the correct variables/filenames, bundle or create new AMI. Commands below are summary from a video tutorial – i forgot the link Run help for each command to get details on the options used, ex: ‘ec2-bundle-vol -h’.
  instance$: cd /mnt
instance$: mkdir ami
instance$: ec2-bundle-vol -d /mnt/ami -k /root/.ec2/pk.xxx.pem -c /root/.ec2/cert.xxx.pem -u xxxx-xxxx-xxxx
instance$: ec2-upload-bundle -b bucket1 -m /mnt/ami/image.manifest.xml -a XXXXXX -s xxxXXXXx
controlling machine$: ec2-register bucket1/image.manifest.xml
• Test your new AMI – run new instance and check if your ebs volume is attached – goodluck!

Init Script Here: mountebs

First you need to see where the login process gets hung up:
ssh -vvv server_address
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-with-mic,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address server_address.
debug1: Unspecified GSS failure. Minor code may provide more information
No credentials cache found
debug1: Unspecified GSS failure. Minor code may provide more information
No credentials cache found
debug1: Unspecified GSS failure. Minor code may provide more information
debug2: we did not send a packet, disable method

and check if a PTR record exists:
[max@linux ~]$ dig -x server_address
; <<>> DiG 9.5.1-P2-RedHat-9.5.1-2.P2.fc10 <<>> -x server_address
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20960
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;sserdda_revres.in-addr.arpa. IN PTR

;; Query time: 87 msec

Here we see that in fact we are being hung on the gssapi-with-mic method and there is no PTR record for the host. The quickest and simples way to resolve this is to disable gssapi-with-mic authmethod globally on the client.
In RedHat/Fedora Linux edit /etc/ssh/ssh_config and make sure you have an uncommented "GSSAPIAuthentication no" line for Host *

# Host *
# ForwardAgent no
# ForwardX11 no
# RhostsRSAAuthentication no
# RSAAuthentication yes
# PasswordAuthentication yes
# HostbasedAuthentication no
GSSAPIAuthentication no
# GSSAPIDelegateCredentials no

If you are using host-based configuration be sure to put this at the top of the file so it takes priority over the defaults below it.
Host server_name
HostName server_address
Port 22
User max
GSSAPIAuthentication no

I based my instructions on previous post on Howto Get Started With Amazon EC2 API Tools, so I won’t give details on some steps. And this post will cover mainly the steps taken to complete my objectives above.

To start, I signed up for an account and enabled EC2 and S3 services, and generate X509 certificate. Next, I selected a test server running CentOS 5.3 with Cpanel and installed java (openjdk 1.6, using yum) as a requirement.

Then, download EC2 API Tools and extract to my working directory at /myhome/.ec2/ and upload your private key and x509 certificates. Don’t forget to follow the filename format of cert-xxx.pem and pk-xx.pem.

Export shell variables (posted on the previous post) and specify the correct private and x509 path. Then run source /myhome/.bashrc or open new terminal to load new environment variables.

Setup EC2 keypair. At first i used the certificate from different account but i got the error below:
Client.AuthFailure: AWS was not able to validate the provided access credentials
I searched for this error and one suggestion is to chmod your certificate and key files to 600 but it didn’t help me. My problem is on our account because one of my teammates changed our account password and probably generated new keys. Anyway, this is where i signed up for a new account and proceeded without issues.

Search for the AMIs to use. Following the steps listed on the instructions, I tried several AMI’s (start/stop processes). I observed some AMI’s took longer to start compare to others but i have no idea why . Btw, you can also search for AMI’s, and start/stop them from Amazon Management Console (EC2 Dashboard).

My next task is to create my private AMIs and here’s a good video tutorial on Customizing an existing AMI and create your own AMI from it. From this part that I need to setup my S3 bucket or directory to store my AMI. There’s a Firefox addon called S3Fox that my friend suggested but unfortunately i can’t install it on my Firefox due to some errors. I found and tried this BucketExplorer for creating my S3 Bucket. Btw, this one is commercial and you can try it for 30 days. I haven’t checked for other apps.

Back to creating my private AMI based on the above video, I ran into issue with ec2-bundle-vol command as it is not included on the AMI that i used, so i search for other AMIs that includes the EC2 Tools and found one from RightScale (CentOS5V1_10.img.manifest.xml).

After this i was able to complete my private AMI and start new instance from it using the above steps without any issues.

Moving Files:

• Copy Drupal file and preserve mode (ownerships, permissions, etc)
  Example: cp -rp drupal_source drupal_destination
  Review your directory permissions on sites/default/files, sites/default/files/civicrm, and other directories.
• Update references to Drupal url, path, and database details (name, user, pass, and host). Sample commands below using grep:
  find /path/to/drupal -type f -exec perl -pi -e "s/example.com/example2.com/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/public_html\/example/public_html\/example2/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/db_name/db_name2/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/db_user/db_user2/g" {} \;
find /path/to/drupal -type f -exec perl -pi -e "s/db_pass/db_pass2/g" {} \;

Moving Database/s:

Case 1: Combined CiviCRM and Drupal Database.

• Create sql dump of source database.
  Example: mysqldump -Q -udb_user -pdbpass db_name > db_name.sql
• Import to destination database.
  Example: mysql -udb_user2 -pdbpass2 db_name2 < db_name.sql
• Update references to Drupal url, path, and database details (name, user, pass, and host) of non-CiviCRM tables. You can use PhpMyAdmin to export this tables, then do the search/replace process on your local editor, and upload back the updates sql. You can also dump the tables from using command line (but you’ll have a long list of tables) and do the grep (same as above) and re-import the updated sql file.
• Update CiviCRM configurations from Drupal Admin section. You need to update the “Resource URLs” and “Directories”.
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Directories (or use the direct url: /civicrm/admin/setting/path?reset=1)
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Resource Urls (or use the direct url: /civicrm/admin/setting/url?reset=1)
• Optional: You can empty Sessions and Cache tables if you want.

Case 2: Separate CiviCRM and Drupal Database (recommended install for CiviCRM).
Process for this setup is almost the same as Case 1, the difference is on the import process for databases. I’ll just provide the complete info below.

• Create sql dump of source databases.
  Examples:
  mysqldump -Q -udb_user -pdbpass db_name_drupal > db_name_drupal.sql
mysqldump -Q -udb_user -pdbpass db_name_civicrm > db_name_civicrm.sql

• Import directly the CiviCRM database.
  Example: mysql -udb_user2 -pdbpass2 db_name2_civicrm < db_name_civicrm.sql
• Update CiviCRM configurations from Drupal Admin section. You need to update the “Resource URLs” and “Directories”.
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Directories (or use the direct url: /civicrm/admin/setting/path?reset=1)
  CiviCRM Admin Settings: Administer Civicrm > Global Settings > Resource Urls (or use the direct url: /civicrm/admin/setting/url?reset=1)
• Update references to Drupal url, path, and database details (name, user, pass, and host) of Drupal database dump.
  perl -pi -e "s/example.com/example2.com/g" db_name_drupal.sql
perl -pi -e "s/public_html\/example/public_html\/example2/g" db_name_drupal.sql
• Import the Drupal database.
  Example: mysql -udb_user2 -pdbpass2 db_name2_drupal < db_name_drupal.sql
• Optional: You can empty Sessions and Cache tables if you want.

That’s All!

Installation process is very straightforward provided you met the requirements like php-pear, gnupg, libgpg-error, and gpgme. You can install it directly from WHM -> Module Installers -> PHP Pecl, enter ‘gnupg‘ and click the Install button. You can do the same via shell with this command ‘pecl install gnupg‘. Then restart your Apache. Be sure to double check if the extension was added to you php.ini.

On Cpanel servers most of the requirements are present, like php-pear (if you compiled your apache/php with pear), gnupg, libgpg-error, etc, except for gpgme which is the main library used by gnupg extension. Our system is running CentOS 4.7 with latest STABLE branch of Cpanel (S35075 at this time of writing), PHP 5.2.9 with pear and libgpg-error (version 1.0-1 only). Only missing requirement was gpgme which i installed from source as it was not available on our default yum repository. You can also install it from other repositories like centos.karan.org. At this time the latest gpgme version is 1.1.8 but I installed version 1.1.2 because it’s the latest version compatible with our libgpg.

I also encountered an error library path when I check the extension, "error while loading shared libraries: libgpgme.so.11: cannot open shared object file: No such file or directory", so I had to add ‘/usr/local/lib‘ to /etc/ld.so.conf and run ldconfig, so that libraries in this directory are detected automatically.

To overcome this issue, and at least have rpm report the proper versions we have to add in our rpmmacros file a new line like: “%_query_all_fmt %%{name}-%%{version}-%%{release}.%%{arch}” that will add to the rpm output the architecture and allow us to see the this:

cat >> ~/.rpmmacros
%_query_all_fmt %%{name}-%%{version}-%%{release}.%%{arch}

and now running the same command will return a more intuitive and meaningful:
rpm -qa | grep ncurses
ncurses-5.5-24.20060715.x86_64
ncurses-5.5-24.20060715.i386

This doesn’t fix anything in how yum will install duplicate programs or libraries, but at least it will allow us to see the full name of the packages in rpm commands. Theoretically people should be able to add into yum.conf (this is the default anyway, so you might have it already):
exactarch=1
and yum will install by default the packages of the arch it is running on (x86_64 in our case). Still, this will not prevent i386 dependencies to show up and be installed. In case you want to completely ignore other arch packages add in the [main] section of /etc/yum.conf to exclude all 32bit packages,:
exclude=*.i386 *.i586 *.i686
and this will completely exclude them completely from yum operations. Please use this with care, and only if you have a full understanding of the implications to exclude those packages.

Even if you don’t exclude the 32bit packages as shown above, it is a good idea to add the arch to all yum operations (like install, remove, etc.), like:
yum install ncurses.x86_64

Hopefully you found this post useful, and have now a better understanding on how rhel/centos use the i368 and x86_64 packages and libraries with rpm and yum on a 64bit installation.