MySQL backup as separate sql files with rotation
A few days ago I accidentally dropped a MySQL database from my home development server. While my computer does automatically run a daily backup using:
mysqldump --all-databasesIt does mean that all my databases are backed up into one massive single file; so it makes it a little bit tricky to pull out the database I was after, however I stumbled across a MySQL backup bash script that will automatically separate out the databases into their own files making it allot easier to manage.
I decided to change the script slightly so that the mysqldump command is a bit more reliable (handling UTF-8, stored procedures, not breaking foreign key indexes etc) and so that it can handle multiple config files. It can also automatically rotate archive backups for x number of months, weeks and days.
Being the open source kinda guy that I am; you can download modified script here:
mysqlbackup.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 | #! /bin/bash # MySQL database backup (databases in separate files) with daily, weekly and monthly rotation # Sebastian Flippence (http://seb.flippence.net) originally based on code from: Ameir Abdeldayem (http://www.ameir.net) # You are free to modify and distribute this code, # so long as you keep the authors name and URL in it. # By default it will search for the config file in the same directory as this script, otherwise you can choose it (e.g. ./mysqlbackup.sh /path/to/mysqlbackup.conf) # Read the config file if [ "$1" = "" ]; then CONFIG_FILE="`dirname $0`/mysqlbackup.conf" else CONFIG_FILE="$1" fi if [ -f "$CONFIG_FILE" ]; then echo "Loading config file ($CONFIG_FILE)" . $CONFIG_FILE else echo "Config file not found ($CONFIG_FILE)" exit 1 fi # Setup some command defaults (can be overriden by the config) MYSQL=${MYSQL:-`which mysql`} MYSQLDUMP=${MYSQLDUMP:-`which mysqldump`} PHP=${PHP:-`which php`} # Date format that is appended to filename DATE=`date +'%Y-%m-%d'` # Setup paths ARCHIVE_PATH="${BACKDIR}/${ARCHIVE_PATH}" ORGBACKDIR="${BACKDIR}" BACKDIR="${BACKDIR}/${LATEST_PATH}/${DATE}" # Check backup directory exists # if not, create it if [ -e $BACKDIR ]; then echo "Backup directory exists (${BACKDIR})" else mkdir -p $BACKDIR echo "Created backup directory (${BACKDIR})" fi if [ $DUMPALL = "y" ]; then echo "Creating list of databases on: ${HOST}..." $MYSQL -N -h $HOST --user=$USER --password=$PASS -e "show databases;" > ${BACKDIR}/dbs_on_${SERVER}.txt # redefine list of databases to be backed up DBS=`sed -e ':a;N;$!ba;s/\n/ /g' -e 's/Database //g' ${BACKDIR}/dbs_on_${SERVER}.txt` fi echo "Backing up MySQL databases..." for database in $DBS; do echo "${database}..." $MYSQLDUMP --host=$HOST --user=$USER --password=$PASS --default-character-set=utf8 --skip-set-charset --routines --disable-keys --force --single-transaction --allow-keywords --dump-date $database > ${BACKDIR}/${SERVER}-MySQL-backup-$database-${DATE}.sql bzip2 -f ${BACKDIR}/${SERVER}-MySQL-backup-$database-${DATE}.sql done if [ $DUMPALL = "y" ]; then rm ${BACKDIR}/dbs_on_${SERVER}.txt fi if [ $MOVETAR = "y" ]; then echo "Moving sql.bz2 files to tar" for file in `ls *.bz2`; do tar -rf ${BACKDIR}/${SERVER}-MySQL-backup-${DATE}.tar $file rm $file done EXT="tar" else EXT="sql.bz2" fi # If you have the mail program 'mutt' installed on # your server, this script will have mutt attach the backup # and send it to the email addresses in $EMAILS if [ $MAIL = "y" ] && [ $EMAILSENDON = $EMAILTODAY ]; then BODY="MySQL backup is ready" ATTACH=`for file in ${BACKDIR}/*${DATE}.${EXT}; do echo -n "-a ${file} "; done` echo "${BODY}" | mutt -s "${SUBJECT}" $ATTACH $EMAILS echo -e "MySQL backup has been emailed" fi if [ $FTP = "y" ]; then echo "Initiating FTP connection..." cd $BACKDIR ATTACH=`for file in ${BACKDIR}/*${DATE}.${EXT}; do echo -n -e "put ${file}\n"; done` ftp -nv < <EOF open $FTPHOST user $FTPUSER $FTPPASS cd $FTPDIR $ATTACH quit EOF echo -e "FTP transfer complete" fi if [ $ROTATE = "y" ]; then echo "Performing backup rotation..." # Convert the number of weeks and months to days MAX_WEEKS=$(($MAX_WEEKS * 7)) MAX_MONTHS=$(($MAX_MONTHS * 31)) # Daily backups if [ ! -d $ARCHIVE_PATH/$DAILY_PATH/$DATE ] && [ "$MAX_DAYS" -gt "0" ]; then mkdir -p $ARCHIVE_PATH/$DAILY_PATH/$DATE # Copy files into archive dir find $BACKDIR -name "*.$EXT" -exec cp {} $ARCHIVE_PATH/$DAILY_PATH/$DATE/. \; fi # Delete old daily backups if [ -d $ARCHIVE_PATH/$DAILY_PATH ]; then find $ARCHIVE_PATH/$DAILY_PATH/ -maxdepth 1 -type d ! -name $DAILY_PATH -mtime +$MAX_DAYS -exec rm -Rf {} \; if [ "$MAX_DAYS" -lt "1" ]; then rm -Rf $ARCHIVE_PATH/$DAILY_PATH/ fi fi # Weekly backups WEEK_NO=`$PHP -r 'echo ceil(date("j", time())/7);'` DATE_WEEK="`date +'%Y-%m-'`$WEEK_NO" if [ ! -d $ARCHIVE_PATH/$WEEKLY_PATH/$DATE_WEEK ] && [ "$MAX_WEEKS" -gt "0" ]; then mkdir -p $ARCHIVE_PATH/$WEEKLY_PATH/$DATE_WEEK # Copy files into archive dir find $BACKDIR -name "*.$EXT" -exec cp {} $ARCHIVE_PATH/$WEEKLY_PATH/$DATE_WEEK/. \; fi # Delete old weekly backups if [ -d $ARCHIVE_PATH/$WEEKLY_PATH ]; then find $ARCHIVE_PATH/$WEEKLY_PATH/ -maxdepth 1 -type d ! -name $WEEKLY_PATH -mtime +$MAX_WEEKS -exec rm -Rf {} \; if [ "$MAX_WEEKS" -lt "1" ]; then rm -Rf $ARCHIVE_PATH/$WEEKLY_PATH/ fi fi # Monthly backups DATE_MONTH=`date +'%Y-%m'` if [ ! -d $ARCHIVE_PATH/$MONTHLY_PATH/$DATE_MONTH ] && [ "$MAX_MONTHS" -gt "0" ]; then mkdir -p $ARCHIVE_PATH/$MONTHLY_PATH/$DATE_MONTH # Copy files into archive dir find $BACKDIR -name "*.$EXT" -exec cp {} $ARCHIVE_PATH/$MONTHLY_PATH/$DATE_MONTH/. \; fi # Delete old monthly backups if [ -d $ARCHIVE_PATH/$MONTHLY_PATH ]; then find $ARCHIVE_PATH/$MONTHLY_PATH/ -maxdepth 1 -type d ! -name $MONTHLY_PATH -mtime +$MAX_MONTHS -exec rm -Rf {} \; if [ "$MAX_MONTHS" -lt "1" ]; then rm -Rf $ARCHIVE_PATH/$MONTHLY_PATH/ fi fi # Delete old backups in latest folder (-mtime +0 is 24 hours or older) find $ORGBACKDIR/$LATEST_PATH/ -maxdepth 1 -type d ! -name $LATEST_PATH -mtime +0 -exec rm -Rf {} \; echo "Backups rotation complete" fi echo "MySQL backup is complete" |
mysqlbackup.conf
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | # MySQL backup config file #----------------------General Settings--------------------# # MySQL server's name SERVER="Servers-hostname" # Directory to backup to BACKDIR="/path/to/backup/folder" #----------------------MySQL Settings--------------------# # MySQL server's hostname or IP address HOST="localhost" # MySQL username USER="username" # MySQL password PASS="password" # List all of the MySQL databases that you want to backup, # each separated by a space. Or set the option below to backup all database DBS="db1 db2" # Set to 'y' if you want to backup all your databases. This will override # the database selection above. DUMPALL="y" # Move compressed bzip2 database dumps into a single tar file # otherwise move them into a year-month-day folder MOVETAR="n" # Custom path to system commands (enable these if you want use a different # location for PHP and MySQL or if you are having problems running this script) #PHP="/opt/local/bin/php" #MYSQL="/opt/local/bin/mysql" #MYSQLDUMP="/opt/local/bin/mysqldump" #----------------------Mail Settings--------------------# # Set to 'y' if you'd like to be emailed the backup (requires mutt) MAIL="n" # Email addresses to send backups to, separated by a space EMAILS="your-email@example.com" SUBJECT="MySQL backup on $SERVER ($DATE)" # Only email on the first day of the month EMAILSENDON="01" EMAILTODAY=`date +'%d'` #----------------------FTP Settings--------------------# # Set "FTP=y" if you want to enable FTP backups FTP="n" # FTP server settings; should be self-explanatory FTPHOST="ftp.server.com" FTPUSER="username" FTPPASS="password" # Directory to backup to. if it doesn't exist, file will be uploaded to # first logged-in directory FTPDIR="backups" #-------------------Backup Rotation Settings-------------------# # Rotate old files? ROTATE="y" # How many backups do you want to keep? (rotate setting above must be y) MAX_DAYS=5 #default 5 MAX_WEEKS=4 #default 4 MAX_MONTHS=3 #default 3 # Paths LATEST_PATH="latest" ARCHIVE_PATH="archive" DAILY_PATH="daily" WEEKLY_PATH="weekly" MONTHLY_PATH="monthly" #----------------------End of Settings------------------# |
March 23rd, 2010 at 9:56 pm
Thank you for this share, it will certainly come in handy.
April 22nd, 2010 at 12:29 pm
i recieve this error
./mysqlbackup.sh: line 99: syntax error near unexpected token `<'
./mysqlbackup.sh: line 99: ` ftp -nv < <EOF'
April 22nd, 2010 at 8:22 pm
Might have been due to the space between the here tag: “< <” I have updated it to: “<<“