Equallogic: Scripting snapshots made easy!

Out-of-the-box, Equallogic SANs provide the ability to create snapshots, clones, and replicas of volumes. This represents an important part of their value proposition, and the success of that idea fueled their growth dramatically, and led to their acquisition by Dell.

Snapshots for filesystems can be automated through a scheduler in the Equallogic web-based GUI. But what about transaction-based applications, such as databases, or products built on top of databases? While Equallogic’s host integration tools (aka HIT kit) provides great support for application-aware snapshots, clones, and replicas for Microsoft products, e.g. Exchange and SQLserver, what can you do for other products centered on transcactions, e.g. Oracle, MySQL, etc., whether on Windows or Linux? Fortunately, Equallogic provides a scripting API and sample scripts that can be used from either UNIX-based systems or Windows. Let’s take a look at how this can be used, and also consider some of the weaknesses of the provided approach.

Since my application was for a MySQL database on Linux, I chose to implement on a Linux system. The goal was to automate database consistent snapshots of the filesystems on which database files and logfiles are stored.

Script Setup

When you unravel the scripting kit, you will have a tree structure that looks like the following:

Follow these steps to implement a scheduled snapshot mechanism:

1. Create a directory to hold the Equallogic scripting kit. E.g. mkdir -p /usr/local/bin/eqlscript will create a directory named eqlscript at /usr/local/bin.
2. Optional: set the setgid bit on the just-created directory, e.g. chmod g+s to this permission bit.
3. Move or copy the contents of the eqlscript kit unix directory to this location.
4. Rename the samples directory under /usr/local/bin/eqlscript to scripts. The contents of the directory should now include what is shown below:

   

5. Modify the CreateSnapshot.pl script to include your PeerStorage Group hostname, and credentials to access the SAN.
   • Look for the lines near the time with variable names $host, $user, and $pass and change the contents appropriately.
   • NOTE: Consider creating a special user on the SAN for this purpose. Use these credentials rather than the group admin account (grpadmin). Do this from the Equallogic web-based GUI — select the ‘Group Configuration’ item on the left menu bar, and then the ‘Administration’ tab. If you add a user called ‘eqlscript’, it could look something like this:
     
6. Modify the permissions on the script so that it is NOT readable by others (it now contains credentials that provide access to your SAN; restricting access to your SAN is another topic for another time).
7. Create a new shell script in the scripts directory to call CreateSnapshot.pl, e.g. mysql-snapshots.sh. Sample contents are shown below.
8. Test both the CreateSnapshot.pl and mysql-snapshots.sh scripts from the command line to make sure they work as expected.
9. Add a crontab entry for root

The crontab entry might look like this:

0 6 * * 1-5 /usr/local/bin/eqlscript/scripts/mysql-snapshot.sh >/dev/null 2>&1

Script Contents

A sample script to control the snapshot process is below, but first some explanation.

The MAILLOG variable is set if any of the commands produced any errors. It is checked at the end of the script to determine whether or not to mail the logfile to a DBA or SAN management personnel. The script assumes two SAN volumes exist: MySQL-DB and MySQL-Logs.

#!/bin/sh

PATH=/bin:/usr/bin:/usr/local/sbin:/usr/local/bin/eqlscript/scripts
export PATH

LOG=/var/log/mysql-snapshot

MAILLOG=0

if [ -e $LOG ]; then
mv $LOG $LOG.prev
fi

/etc/init.d/mysql stop > $LOG 2>&1
STATUS=$?
if [ $STATUS -ne 0 ]; then
MAILLOG=1
fi

CreateSnapshot.pl MySQL-DB >> $LOG
STATUS=$?
if [ $STATUS -ne 0 ]; then
MAILLOG=1
fi

CreateSnapshot.pl MySQL-Logs >> $LOG
STATUS=$?
if [ $STATUS -ne 0 ]; then
MAILLOG=1
fi

/etc/init.d/mysql start >> $LOG 2>&1
STATUS=$?
if [ $STATUS -ne 0 ]; then
MAILLOG=1
fi

if [ $MAILLOG -ne 0 ]; then
mailx -s “MySQL Snapshot Errors” dba@example.com <$LOG
exit 1
fi

exit 0

Comments

All of this seems very nice, and indeed it is very useful. Since an application-aware agent is not available, it does require a brief period of downtime to ensure consistency. With some applications, you may have the ability to complete any active transactions, but freeze, or hold off any new ones until you release the application to continue — either way, you need an external mechanism to quiesce the data source to ensure your snapshots are usable. The Equallogic scripting framework is simple, but it has some weaknesses.

First, it is based on telnet, an application susceptible to network sniffing. This requires that CLI access be added for the telnet protocol (you may have noticed in the image above showing users). Since this could open the door to DoS attacks, or brute force password cracking, you should have other means of restricting network access to your SAN. It would be preferably to have the API use SSH, so that credentials do not have to be exposed. Even better would be to enable public-key authentication so that the application needs access to a cryptographic key to gain access. Enabling SSH probably means re-writing of the API, while public-key authentication would require changing the operating system configuration on the SAN itself (also probably rather trivial, given its NetBSD heritage).

Second, the permissions for administration accounts are not granular enough to restrict an account to specific tasks, e.g. create snapshots. Ideally, any user for scripting the creation of snapshots (or clones, or replicas) should not be able to do much else. It would be better if some level of permission existing that allowed a user to create new entities, but not delete existing ones.

Third, while the scheduler in the GUI can limit the maximum number of snapshots retained for a volume (the HIT scheduler for Exchange and SQLserver volumes have the same ability), this functionality is obviously built into that higher-level component and not a part of an underlying volume management layer. This means that you will have to occasionally purge snapshots from your volumes. Not a big deal, but something that would be nice to have some day.

The bottom line… this is useful, even necessary functionality, but understand the security implications, and be cautious. How you architect your SAN solution is up to you — all you hope for from a vendor is that they supply you with the most important pieces, enabling your success. Equallogic gets high marks from me!