Homeserver update: The ultimate backup-system

A couple of months my primary harddisk of my Homeserver crashed. As a result I lost the main system which was Windows Home Server. I didn’t want to set up everything again because I thought there will be the same case again – harddrives are born to die sooner or later. So I decided I to set up a more agile system that I can back up easy. A system that can boot up from an attached USB flash drive. So I switched over to Ubuntu Server.

The challenge was not the fact that the new system would be Linux since I earned some Linux experience in the past. Instead I knew that my old system was working pretty fine with WHS Drive Extender and a fully automatic incremental backup system that had been approved several times in the past.

But I knew I had to write and configure my own system for my own case:

Actually I am using four harddrives that serve a capacity of 4.2 TB. I don’t use any RAID technology since I separate every backup in ‘shares’. Eg. a Windows-share is a backup, but also my laptop is a share.

Home Automation, Unit Tests, Eclipse & Documentation

Also what’s new, we connected the system to our EZ-Control home automation system so that we are able to trigger the system every week, or by web interface, or by iPhone, or by web interface via VPN via iPhone (how cool is that, heh? 😀 ) The link to the home automation also gives me the ability to hard-reset the system in case of a freeze or whatever. I never used Wake On Lan because it was never working on the Asus-mainboard I am using and the Home automation gives me more control.

In the past I really deleted a lot of data that had been already copied, just because I was trying some new rsync parameters. Or the Mediacenter was offline so the system couldn’t mount anything. Or whatever. So started setting up virtual machines in VMware Fusion that I used to set up my own testcases and to develop my scripts in a sandbox. Also from my job at Accenture I am used to Compuserve’s Workbench for IBM Mainframes. I was looking for such a cool tool and found Eclipse’ Remote System Explorer<link>. But also I knew before I make any changes I should document and backup everything twice, because I start to forget what where and when I configured something how many times.

Challenges in Bash Scripting

I knew that I had done some Bash-scripting in the past, but it was to far away, so I had to reintroduce myself into conditions, variables and functions. Some challenges I had to google:

  • Conditions
  • Variables
  • Extracting variables with awk
  • grep
  • Mailing from shell
  • starting scripts delayed for a relative time with at
  • System logging

Just to name the most important for me.

The Main Idea

  • The whole system is triggered every Sunday at 10 o’clock by a home automation system
  • … that can be used to start the system by a web interface
  • … but also can be switched on like a normal computer.

After the system started

  • Network shares are mounted automatically
  • 3 minutes later a script starts (so in case I’ve got 5 minutes to abort the script for debugging)
  • that sends out a Hello-mail so that we know the system is running
  • looks for changes in the target directories
  • decides if a backup should be done or not

If everything is finished:

  • a script sends out backup-logfiles and disk usage by mail
  • and shuts down the whole system after 30 minutes.

Here I will provide all my scripting. Keep in mind that i had to add some line breaks (since my self develope WordPress Theme is not ready for code :D) that I’ve marked with an \-Backslash.

/etc/rc.local


#!/bin/sh -e

echo “Starte synceverything.sh”

# Eclipse Remote System Explorer-Daemon
cd  /opt/rseserver
perl ./daemon.pl &

# My cool script
cd /home/andremotz/Documents/shellscripts
at now + 5 minutes -f synceverything.sh
exit 0

/etc/fstab

# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=5347ad31-f9ad-4171-9237-69ceb9bf240f /               ext4    errors=remount-ro 0       1

#1.5tb1
UUID=bd77019d-b857-455d-a65f-4ca120ed6083 /media/1.5tb1 ext4    defaults        0       2

#1.5tb2
UUID=ac13878b-9d7a-44f8-be06-d9aae4fa7d23 /media/1.5tb2 ext4    defaults        0       2

#750gb
UUID=80ad76a4-ffe6-4a08-8b95-33d082e70ffa /media/750gb  ext4    defaults        0       2

#500gb
UUID=2f2896ec-7f29-47d0-b3ea-cb246b522f65 /media/500gb ext4     defaults        0       2

#Netzwerkshares
//10.3.0.6/camcorder /mnt/Mediacenter_camcorder cifs credentials=/root/.smbMediacenter,iocharset=utf8     0     0
//10.3.0.6/photos /mnt/Mediacenter_photos cifs credentials=/root/.smbMediacenter,iocharset=utf8     0     0
//10.3.0.6/Musik /mnt/Mediacenter_musik cifs credentials=/root/.smbMediacenter,iocharset=utf8     0     0

//10.4.0.2/MacBook_Backup /mnt/Synology_Andre_MacBook cifs credentials=/root/.smbSynology,iocharset=utf8  0       0
//10.4.0.2/iMac_Backup  /mnt/Synology_iMac cifs credentials=/root/.smbSynology,iocharset=utf8        0       0
//10.4.0.2/WesternDigitalHDBackup       /mnt/Synology_WesternDigitalHD cifs credentials=/root/.smbSynology,iocharset=utf8        0       0

synceverything.sh

#!/bin/bash
logmsg="/usr/bin/logger -s -t SyncAllScript"
$logmsg "Starting SyncAllScript"

BOOTUP_MAIL_CONTENT=$(cat /home/andremotz/Documents/shellscripts/mail_at_bootup.txt)
MAIL_OPTIONS="-f <sendermail> -s andremotz.com -xu <user> -xp <pass>"

sendemail -t <mailadresses> -u "Homeserver on" $MAIL_OPTIONS -m $BOOTUP_MAIL_CONTENT 

SRC_MEDIACENTER_CAMCORDER="/mnt/Mediacenter_camcorder/"
TRGT_MEDIACENTER_CAMCORDER="/media/1.5tb2/Mediacenter_Camcorder/"
LOG_MEDIACENTER_CAMCORDER="/var/log/rsync-mediacenter-camcorder.log"

SRC_MEDIACENTER_PHOTOS="/mnt/Mediacenter_photos/"
TRGT_MEDIACENTER_PHOTOS="/media/750gb/Mediacenter_Photos/"
LOG_MEDIACENTER_PHOTOS="/var/log/rsync-mediacenter-photos.log"

SRC_MEDIACENTER_MUSIK="/mnt/Mediacenter_musik/"
TRGT_MEDIACENTER_MUSIK="/media/750gb/Mediacenter_Musik/"
LOG_MEDIACENTER_MUSIK="/var/log/rsync-mediacenter-musik.log"

backupFunction () {
  # How many Bytes would be transferred?
  WILLTRANSFERBYTES=$(rsync -an --stats $1 $2 | grep 'Total transferred file size'\
      | awk '{print $5}')

  # How many files would be deleted?
  WILLDELETEFILES=$(rsync -avn --delete-after $1 $2 | grep 'deleting' | wc -l)

  # If more than 40GB = 40000000000 bytes
  WARNINGS=""
  if [ $WILLTRANSFERBYTES -gt 40000000000 ]
    then
      WARNING="More than 40 GB changes! "
      WARNINGS="$WARNINGS $WARNING"
      echo $WARNING
  fi

  NUMTOTAL_OF_FILES=$(find $2 -type f | wc -l)

  # a quarter of files total
  QUARTERNUMBER_OF_FILES=$(expr $NUMTOTAL_OF_FILES / 4)

  if [ $WILLDELETEFILES -gt $QUARTERNUMBER_OF_FILES ]
    then
      WARNING="More than 25% of all files will be deleted!"
      WARNINGS="$WARNINGS $WARNING"
      echo $WARNING
  fi

  if [ -n "$WARNINGS" ] ; then
    echo "$WARNINGS - aborting script, please force run."
    echo "$WARNINGS - aborting script, please force run." > $3
  else
    rsync -azP --delete --size-only $1 $2 > $3 2>&1
  fi
}

$logmsg "Starte Mediacenter Camcorder Backup"
backupFunction $SRC_MEDIACENTER_CAMCORDER $TRGT_MEDIACENTER_CAMCORDER $LOG_MEDIACENTER_CAMCORDER

$logmsg "Starte Mediacenter Photos Backup"
backupFunction $SRC_MEDIACENTER_PHOTOS $TRGT_MEDIACENTER_PHOTOS $LOG_MEDIACENTER_PHOTOS

$logmsg "Starte Mediacenter Musik Backup"
backupFunction $SRC_MEDIACENTER_MUSIK $TRGT_MEDIACENTER_MUSIK $LOG_MEDIACENTER_MUSIK

./maileverything.sh
echo "shutdown -h 0" | at now + 30 minutes

maileverything.sh

#/bin/bash

MAIL_OPTIONS="-f <sender mail> -s andremotz.com -xu <user> -xp <pass>"

# How often was my mom's laptop backuped?
LOG_CLIVIALAPTOP=$(cat /var/log/syslog | grep -c 'cliviamotz')
CLIVIALAOPTOP_LOGINCOUNT=$(expr $LOG_CLIVIALAPTOP / 2)

# Mediacenter Logfiles
LOG_MEDIACENTER_CAMCORDER_PATH="/var/log/rsync-mediacenter-camcorder.log"
LOG_MEDIACENTER_PHOTOS_PATH="/var/log/rsync-mediacenter-photos.log"
LOG_MEDIACENTER_MUSIK_PATH="/var/log/rsync-mediacenter-musik.log"

# Content of Logs
LOG_MEDIACENTER_CAMCORDER=$(cat $LOG_MEDIACENTER_CAMCORDER_PATH)
LOG_MEDIACENTER_PHOTOS=$(cat $LOG_MEDIACENTER_PHOTOS_PATH)
LOG_MEDIACENTER_MUSIK=$(cat $LOG_MEDIACENTER_MUSIK_PATH)

# Change Dates of Logfiles
LOG_MEDIACENTER_CAMCORDER_DATE=$(stat $LOG_MEDIACENTER_CAMCORDER_PATH | grep 'Change')
LOG_MEDIACENTER_PHOTOS_DATE=$(stat $LOG_MEDIACENTER_PHOTOS_PATH | grep 'Change')
LOG_MEDIACENTER_MUSIK_DATE=$(stat $LOG_MEDIACENTER_MUSIK_PATH | grep 'Change')

# Concatenate everything for mail
MAILCONTENT_CLIVIALAPTOP="<p><b>Clivia Laptop</b> Backups: $CLIVIALAOPTOP_LOGINCOUNT </p>"

MAILCONTENT_MEDIACENTER_CAMCORDER="<p><b>$LOG_MEDIACENTER_CAMCORDER_PATH:\
  </b> <br /> $LOG_MEDIACENTER_CAMCORDER_DATE </b> <br />\
  $LOG_MEDIACENTER_CAMCORDER </p>"

MAILCONTENT_MEDIACENTER_PHOTOS="<p><b>$LOG_MEDIACENTER_PHOTOS_PATH: </b> \
  <br /> $LOG_MEDIACENTER_PHOTOS_DATE </b> <br /> $LOG_MEDIACENTER_PHOTOS </p>"

MAILCONTENT_MEDIACENTER_MUSIK="<p><b>$LOG_MEDIACENTER_MUSIK_PATH: </b> \
  <br /> $LOG_MEDIACENTER_MUSIK_DATE </b> <br /> $LOG_MEDIACENTER_MUSIK </p>"

MAILCONTENT_DF=$(df -h)

MAIL_CONTENT="<html><body><p>Ich bin's nochmal, hier mein Sync Report:</p> "\
"$MAILCONTENT_CLIVIALAPTOP"\
"$MAILCONTENT_MEDIACENTER_CAMCORDER"\
"$MAILCONTENT_MEDIACENTER_PHOTOS"\
"$MAILCONTENT_MEDIACENTER_MUSIK"\
"<hr /><p>$MAILCONTENT_DF</p>"\
"<p>Ich laufe noch ca. eine halbe Stunde, vielleicht wollt ihr euch ja noch bei mir einloggen.</p>"\
"<p>Liebe Gr&uuml;sse und bis zum n&auml;chsten Mal!</p></body></html>"

echo $MAIL_CONTENT > /home/andremotz/Documents/shellscripts/message.temp

sendemail -t <mailadresses> -u "Shutting down - Sync Report" $MAIL_OPTIONS \
  -o message-file=/home/andremotz/Documents/shellscripts/message.temp

One thought on “Homeserver update: The ultimate backup-system”

Leave a Reply

Your email address will not be published. Required fields are marked *