This is an old revision of the document!
Back-up Server Old Setup
This has been more difficult than I would have thought. I have used rsnapshot the past couple of years (2020-21), but have never really been happy with it for many reasons. The main reason is that it has never worked well, the rsync is reasonably fast, but the rsnapshot takes far too long. Other problems include; difficulties in set-up, and concerns will reliability. Recently I noticed that Debian no longer includes it in its repositories as of Debian 11 due to poor maintenance….. So I have no moved to Restic, which is described here: Back-up Server Setup.
overlayfs
I was disappointed with overlayfs, it does not seem to allow for a true merge file system, just overlay file system. The information available to set up was some of the worst I have found to date for Linux in the modern day. It is the only solution built in the Kernel. I looks like it may be of some use with containers, such as Docker. It does not really interest me at this time and I am not using.
After some experimenting and basic research I found the following seemed to work:
sudo mount -t overlay -o lowerdir=/media/Disk2/backup,upperdir=/media/Disk1/backup,workdir=/media/Disk1/.backup overlay /media/Disk1/backup
The syntax is reasonably evident from the command. The workdir is a directory on the same media as the upperdir, and is where the working files for overlayfs are kept.
Overlayfs is not what I wish to use for my back-up server.
mergerfs
In the end I has stopped using mergerfs. There are other options to combine drives, e.g. lvm or simplely using symlinks.
Mergerfs looks more like what I wish to achieve it is a way to pool drive into a common mount point. Below are some links with information and and possible setups.
- Systemd startup Mergerfs Union Filesystem on Arch Linux
Down load the current correct version of mergerfs from: https://github.com/trapexit/mergerfs/releases/, hint look at the link of the file of interest….:
wget https://github.com/trapexit/mergerfs/releases/download/2.29.0/mergerfs_2.29.0.ubuntu-xenial_amd64.deb
Then load it into linux: sudo dpkg -i mergerfs_2.29.0.ubuntu-xenial_amd64.deb
Then create a mount point directory, e.g. /srv/storage, sudo mkdir /srv/storage
sudo vim /etc/fstab
UUID=d426f5e0-7fbd-4e7f-915e-b7c78eb70877 / ext4 errors=remount-ro 0 1 # swap was on /dev/sdd5 during installation UUID=e3442da3-4eaf-477d-94af-6ace66097a54 none swap sw 0 0 UUID=2fedae7e-8214-4fec-a6dd-85d7026d474f /media/Disk1 ext4 defaults 0 0 UUID=38f52a66-fe96-4eb4-a794-6fe1d02bab4a /media/Disk2 ext4 defaults 0 0 UUID=1cb1509c-ec09-46cd-ab30-23f3be5b554e /media/Disk3 ext4 defaults 0 0 UUID=2ba18a1e-d48d-4122-bff8-30c2bc359749 /media/Disk4 ext4 defaults 0 0 /media/Disk1:/media/Disk2 /srv/storage fuse.mergerfs use_ino,defaults,allow_other,minfreespace=50G,fsname=mergerfs 0 0 #/media/Disk1 /export/Disk1 bind bind 0 0 #/media/Disk2 /export/Disk2 bind bind 0 0 /media/Disk4 /export/Disk4 bind bind 0 0 /home/shared /export/shared bind bind 0 0
Directly mount from CLI:
sudo mergerfs -o defaults,allow_other,use_ino,fsname=mergerFS /media/Disk1:/media/Disk2 /srv/storage
To unmount: sudo umount /srv/storage
To check mounted disks df -Th
Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 16G 0 16G 0% /dev tmpfs tmpfs 3.2G 2.7M 3.2G 1% /run /dev/sdd1 ext4 443G 161G 260G 39% / tmpfs tmpfs 16G 0 16G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup mergerfs fuse.mergerfs 11T 177M 11T 1% /srv/storage /dev/sdb1 ext4 5.5T 89M 5.2T 1% /media/Disk2 /dev/sda1 ext4 5.5T 89M 5.2T 1% /media/Disk3 /dev/sdc1 ext4 5.5T 89M 5.2T 1% /media/Disk1 /dev/sde1 ext4 9.1T 1.2T 7.5T 14% /media/Disk4 tmpfs tmpfs 3.2G 0 3.2G 0% /run/user/1000
LVM
With the availability of Gparted to adjust partitions on the run I do not know if LVM is required any more. It does however look like a flexible system.
| Command | example | LVM | PV | VG | LG |
|---|---|---|---|---|---|
| s | pvs | ✘ | ✔ | ✔ | ✔ |
| display | pvdisplay | ✘ | ✔ | ✔ | ✔ |
| create | pvcreate | ✘ | ✔ | ✔ | ✔ |
| rename | vgrename | ✘ | ✘ | ✔ | ✔ |
| change | lvchange | ✔ | ✔ | ✔ | ✔ |
| move | vgmove | ✘ | ✔ | ✔ | ✘ |
| extend | lvextend | ✘ | ✘ | ✔ | ✔ |
| reduce | lvreduce | ✘ | ✘ | ✔ | ✔ |
| resize | lvresize | ✘ | ✔ | ✘ | ✔ |
| split | vgsplit | ✘ | ✘ | ✔ | ✘ |
| merge | vgmerge | ✘ | ✘ | ✔ | ✘ |
| convert | vgconvert | ✘ | ✘ | ✔ | ✘ |
| import | vgimport | ✘ | ✘ | ✔ | ✘ |
| export | vgexport | ✘ | ✘ | ✔ | ✘ |
| importclone | vgimportclone | ✘ | ✘ | ✔ | ✘ |
| cfgbackup | vgcfgbackup | ✘ | ✘ | ✔ | ✘ |
| cfgrestore | vgcfgrestore | ✘ | ✘ | ✔ | ✘ |
| ck | vgck | ✘ | ✔ | ✔ | ✘ |
| scan | vgscan | 1 | ✔ | ✔ | ✔ |
| mknodes | vgmknode | ✘ | ✘ | ✔ | ✘ |
| remove | vgremove | ✘ | ✔ | ✔ | ✔ |
| dump | vgdump | ✔ | ✘ | ✘ | ✘ |
-diskscan
Links:
NFS
To mount a NFS manually:
sudo mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1sudo mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2
Where:
-t nfs: type nfs-o: options, options are separated by commas with no spacesrw: read / writevers=4: version 4 of nfs
192.168.1.5:/export/Disk1is the remote exported NFS mount point/mnt/backup1: is the local mount point, note this directory must exist, ideally empty, any existing contents are usually invisible and otherwise ignored
To unmount:
sudo umount /mnt/backup1sudo umount /mnt/backup2
Where:
/mnt/backup1is the local mount point to be unmounted
#!/bin/bash #input 1 is the proposed mount point if [ -z "$1" ] then exit 11 #no mount point input, exit with error else mount1=$1 #Use mount point input fi if grep -qs "$mount1" /proc/mounts; then exit 0 #Already mounted, exit with success else mount "$mount1" if [ $? -eq 0 ]; then exit 0 # Successfully mounted, exit with success else exit 12 # mount failed, exit with error fi fi
Created script to mount backup1 sudo bash ~/Myscripts/mount_backup1.sh to run
''sudo vim ~/Myscripts/mount_backup1.sh''
#!/bin/bash #Script to mount NFS remote Disk1 to local /mnt/backup1 if grep -qs "/mnt/backup1" /proc/mounts; then exit 0 #Already mounted, exit with success else mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1 if [ $? -eq 0 ]; then exit 0 # Successfully mounted, exit with success else exit 12 # mount failed, exit with error fi fi
Created script to mount backup2 sudo bash ~/Myscripts/mount_backup2.sh to run
''sudo vim ~/Myscripts/mount_backup2.sh''
#!/bin/bash #Script to mount NFS remote Disk2 to local /mnt/backup2 if grep -qs "/mnt/backup2" /proc/mounts; then exit 0 #Already mounted, exit with success else mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2 if [ $? -eq 0 ]; then exit 0 # Successfully mounted, exit with success else exit 12 # mount failed, exit with error fi fi
As we intend to shutdown the backup server it is good to unmount the back-up drives first.
Created script to unmount backup1 sudo ~/Myscripts/unmount_backup1.sh
''sudo vim ~/Myscripts/unmount_backup1.sh''
#!/bin/bash #Script to mount NFS remote Disk2 to local /mnt/backup2 if grep -qs "/mnt/backup1" /proc/mounts; then umount /mnt/backup1 if [ $? -eq 0 ]; then exit 0 # Successfully unmounted, exit with success else exit 12 # umount failed, exit with error fi else exit 0 # Not actually mounted, exit with success fi
Created script to unmount backup2 sudo ~/Myscripts/unmount_backup2.sh
''sudo vim ~/Myscripts/unmount_backup2.sh''
#!/bin/bash #Script to mount NFS remote Disk2 to local /mnt/backup2 if grep -qs "/mnt/backup2" /proc/mounts; then umount /mnt/backup2 if [ $? -eq 0 ]; then exit 0 # Successfully unmounted, exit with success else exit 12 # umount failed, exit with error fi else exit 0 # Not actually mounted, exit with success fi
Backup main Media drive (Rsync / Rsnapshot method)
Sadly I have found Rsnapshot and Rsync is not suitable for large backup service. Rsnapshot is simply too slow and I am concerned with reliability. Rsnapshot configuration is overly complex, with strange requirements such as tab for white space. As of Debian 11 it is no longer even included in the Debian repositories for lack of maintenance. It feels effectively abandoned.
I do not use Rsnapshot anymore. I just do not feel comfortable using it.
The following is the command I use to backup my main media drive: sudo rsync -av -u --delete /media/disk2/Media/ baumkp@192.168.1.5::all/media/Disk4/disk2 --exclude=snapraid.* --exclude=.Trash* --exclude=lost* >tmp.txt
I have rsync setup running as a daemon on 192.168.1.5 with login details for baumkp setup on the sever accepting input from the local area network to just the one address 192.168.1.10.
On the remote machine 192.168.1.5:
sudo vim /etc/rsyncd.conf
# create new # any name you like [all] # destination directory to copy path = / # hosts you allow to access hosts allow = 192.168.1.10 #hosts deny = * list = true uid = root gid = root read only = false #auth users = baumkp #secrets file = /etc/rsyncd.secrets
sudo vim /etc/rsyncd.secrets
username:userpassword
Use systemd systemctl from control of the deamon: sudo systemctl status rsync.service [status, start, stop, restart, reload, disable, enable] I have enabled so the rsync deamon this normally running.
The local directory /media/disk2/Media/ contents are to be backed up.
rsnapshot backups
I have 2 rsnapshot backup configurations;
/etc/rsnapshot1.conf
To run locally: sudo rsnapshot -c /etc/rsnapshot1.conf daily
sudo vim /etc/rsnapshot1.conf
################################################# # rsnapshot.conf - rsnapshot configuration file # ################################################# # # # PLEASE BE AWARE OF THE FOLLOWING RULE: # # # # This file requires tabs between elements # # # ################################################# ####################### # CONFIG FILE VERSION # ####################### config_version 1.2 ########################### # SNAPSHOT ROOT DIRECTORY # ########################### # All snapshots will be stored under this root directory. # # snapshot_root /var/cache/rsnapshot/ #default snapshot_root /mnt/backup1 # If no_create_root is enabled, rsnapshot will not automatically create the # snapshot_root directory. This is particularly useful if you are backing # up to removable media, such as a FireWire or USB drive. # #no_create_root 1 ################################# # EXTERNAL PROGRAM DEPENDENCIES # ################################# # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features. # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility. # # See the README file or the man page for more details. # cmd_cp /bin/cp # uncomment this to use the rm program instead of the built-in perl routine. # cmd_rm /bin/rm # rsync must be enabled for anything to work. This is the only command that # must be enabled. # cmd_rsync /usr/bin/rsync # Uncomment this to enable remote ssh backups over rsync. # #cmd_ssh /usr/bin/ssh # Comment this out to disable syslog support. # cmd_logger /usr/bin/logger # Uncomment this to specify the path to "du" for disk usage checks. # If you have an older version of "du", you may also want to check the # "du_args" parameter below. # cmd_du /usr/bin/du # Uncomment this to specify the path to rsnapshot-diff. # cmd_rsnapshot_diff /usr/bin/rsnapshot-diff # Specify the path to a script (and any optional arguments) to run right # before rsnapshot syncs files # #cmd_preexec /path/to/preexec/script # Specify the path to a script (and any optional arguments) to run right # after rsnapshot syncs files # #cmd_postexec /path/to/postexec/script # Paths to lvcreate, lvremove, mount and umount commands, for use with # Linux LVMs. # #linux_lvm_cmd_lvcreate /sbin/lvcreate #linux_lvm_cmd_lvremove /sbin/lvremove #linux_lvm_cmd_mount /bin/mount #linux_lvm_cmd_umount /bin/umount ######################################### # BACKUP LEVELS / INTERVALS # # Must be unique and in ascending order # # e.g. alpha, beta, gamma, etc. # ######################################### #retain alpha 6 #retain beta 7 #retain gamma 4 #retain delta 3 #retain hourly 6 retain daily 7 retain weekly 4 retain monthly 3 ############################################ # GLOBAL OPTIONS # # All are optional, with sensible defaults # ############################################ # Verbose level, 1 through 5. # 1 Quiet Print fatal errors only # 2 Default Print errors and warnings only # 3 Verbose Show equivalent shell commands being executed # 4 Extra Verbose Show extra verbose information # 5 Debug mode Everything # verbose 2 # Same as "verbose" above, but controls the amount of data sent to the # logfile, if one is being used. The default is 3. # If you want the rsync output, you have to set it to 4 # loglevel 3 # If you enable this, data will be written to the file you specify. The # amount of data written is controlled by the "loglevel" parameter. # logfile /var/log/rsnapshot.log # If enabled, rsnapshot will write a lockfile to prevent two instances # from running simultaneously (and messing up the snapshot_root). # If you enable this, make sure the lockfile directory is not world # writable. Otherwise anyone can prevent the program from running. # lockfile /var/run/rsnapshot.pid # By default, rsnapshot check lockfile, check if PID is running # and if not, consider lockfile as stale, then start # Enabling this stop rsnapshot if PID in lockfile is not running # #stop_on_stale_lockfile 0 # Default rsync args. All rsync commands have at least these options set. # rsync_short_args -av rsync_long_args --delete --numeric-ids --relative --delete-excluded # ssh has no args passed by default, but you can specify some here. # #ssh_args -p 22 # Default arguments for the "du" program (for disk space reporting). # The GNU version of "du" is preferred. See the man page for more details. # If your version of "du" doesn't support the -h flag, try -k flag instead. # du_args -csh # If this is enabled, rsync won't span filesystem partitions within a # backup point. This essentially passes the -x option to rsync. # The default is 0 (off). # #one_fs 0 # The include and exclude parameters, if enabled, simply get passed directly # to rsync. If you have multiple include/exclude patterns, put each one on a # separate line. Please look up the --include and --exclude options in the # rsync man page for more details on how to specify file name patterns. # #include ??? #include ??? exclude /lost+found exclude snapraid.* exclude lost* exclude .Trash* exclude /proc/* exclude /run/* exclude /tmp/* exclude /VMs/* # The include_file and exclude_file parameters, if enabled, simply get # passed directly to rsync. Please look up the --include-from and # --exclude-from options in the rsync man page for more details. # #include_file /path/to/include/file #exclude_file /path/to/exclude/file # If your version of rsync supports --link-dest, consider enabling this. # This is the best way to support special files (FIFOs, etc) cross-platform. # The default is 0 (off). # #link_dest 0 # When sync_first is enabled, it changes the default behaviour of rsnapshot. # Normally, when rsnapshot is called with its lowest interval # (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest # intervals. With sync_first enabled, "rsnapshot sync" handles the file sync, # and all interval calls simply rotate files. See the man page for more # details. The default is 0 (off). # #sync_first 0 # If enabled, rsnapshot will move the oldest directory for each interval # to [interval_name].delete, then it will remove the lockfile and delete # that directory just before it exits. The default is 0 (off). # #use_lazy_deletes 0 # Number of rsync re-tries. If you experience any network problems or # network card issues that tend to cause ssh to fail with errors like # "Corrupted MAC on input", for example, set this to a non-zero value # to have the rsync operation re-tried. # #rsync_numtries 0 # LVM parameters. Used to backup with creating lvm snapshot before backup # and removing it after. This should ensure consistency of data in some special # cases # # LVM snapshot(s) size (lvcreate --size option). # #linux_lvm_snapshotsize 100M # Name to be used when creating the LVM logical volume snapshot(s). # #linux_lvm_snapshotname rsnapshot # Path to the LVM Volume Groups. # #linux_lvm_vgpath /dev # Mount point to use to temporarily mount the snapshot(s). # #linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup ############################### ### BACKUP POINTS / SCRIPTS ### ############################### # LOCALHOST backup /media/disk1/KarlData KarlData/ backup /home/ home/ #backup /home/ localhost/ #backup /etc/ localhost/ #backup /usr/local/ localhost/ #backup /var/log/rsnapshot localhost/ #backup /etc/passwd localhost/ #backup /home/foo/My Documents/ localhost/ #backup /foo/bar/ localhost/ one_fs=1, rsync_short_args=-urltvpog #backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/ # You must set linux_lvm_* parameters below before using lvm snapshots #backup lvm://vg0/xen-home/ lvm-vg0/xen-home/ # EXAMPLE.COM #backup_exec /bin/date "+ backup of example.com started at %c" #backup root@example.com:/home/ example.com/ +rsync_long_args=--bwlimit=16,exclude=core #backup root@example.com:/etc/ example.com/ exclude=mtab,exclude=core #backup_exec ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql" #backup root@example.com:/var/db/dump/ example.com/ #backup_exec /bin/date "+ backup of example.com ended at %c" # CVS.SOURCEFORGE.NET #backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh rsnapshot.cvs.sourceforge.net/ # RSYNC.SAMBA.ORG #backup rsync://rsync.samba.org/rsyncftp/ rsync.samba.org/rsyncftp/
config_version 1.2 snapshot_root /mnt/backup1 cmd_cp /bin/cp cmd_rm /bin/rm cmd_rsync /usr/bin/rsync cmd_logger /usr/bin/logger cmd_du /usr/bin/du cmd_rsnapshot_diff /usr/bin/rsnapshot-diff retain daily 7 retain weekly 4 retain monthly 3 verbose 2 loglevel 3 logfile /var/log/rsnapshot.log lockfile /var/run/rsnapshot.pid rsync_short_args -av rsync_long_args --delete --numeric-ids --relative --delete-excluded exclude /lost+found exclude snapraid.* exclude lost* exclude .Trash* exclude /proc/* exclude /run/* exclude /tmp/* exclude /VMs/* backup /media/disk1/KarlData KarlData/ backup /home/ home/
Remember! !The rsnapshot configuration file separates with tabs not spaces!
/etc/rsnapshot2.conf
To run locally: sudo rsnapshot -c /etc/rsnapshot2.conf daily
sudo vim /etc/rsnapshot2.conf
################################################# # rsnapshot.conf - rsnapshot configuration file # ################################################# # # # PLEASE BE AWARE OF THE FOLLOWING RULE: # # # # This file requires tabs between elements # # # ################################################# ####################### # CONFIG FILE VERSION # ####################### config_version 1.2 ########################### # SNAPSHOT ROOT DIRECTORY # ########################### # All snapshots will be stored under this root directory. # # snapshot_root /var/cache/rsnapshot/ #default snapshot_root /mnt/backup2 # If no_create_root is enabled, rsnapshot will not automatically create the # snapshot_root directory. This is particularly useful if you are backing # up to removable media, such as a FireWire or USB drive. # #no_create_root 1 ################################# # EXTERNAL PROGRAM DEPENDENCIES # ################################# # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features. # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility. # # See the README file or the man page for more details. # cmd_cp /bin/cp # uncomment this to use the rm program instead of the built-in perl routine. # cmd_rm /bin/rm # rsync must be enabled for anything to work. This is the only command that # must be enabled. # cmd_rsync /usr/bin/rsync # Uncomment this to enable remote ssh backups over rsync. # #cmd_ssh /usr/bin/ssh # Comment this out to disable syslog support. # cmd_logger /usr/bin/logger # Uncomment this to specify the path to "du" for disk usage checks. # If you have an older version of "du", you may also want to check the # "du_args" parameter below. # cmd_du /usr/bin/du # Uncomment this to specify the path to rsnapshot-diff. # cmd_rsnapshot_diff /usr/bin/rsnapshot-diff # Specify the path to a script (and any optional arguments) to run right # before rsnapshot syncs files # #cmd_preexec /path/to/preexec/script # Specify the path to a script (and any optional arguments) to run right # after rsnapshot syncs files # #cmd_postexec /path/to/postexec/script # Paths to lvcreate, lvremove, mount and umount commands, for use with # Linux LVMs. # #linux_lvm_cmd_lvcreate /sbin/lvcreate #linux_lvm_cmd_lvremove /sbin/lvremove #linux_lvm_cmd_mount /bin/mount #linux_lvm_cmd_umount /bin/umount ######################################### # BACKUP LEVELS / INTERVALS # # Must be unique and in ascending order # # e.g. alpha, beta, gamma, etc. # ######################################### #retain alpha 6 #retain beta 7 #retain gamma 4 #retain delta 3 #retain hourly 6 retain daily 7 retain weekly 4 retain monthly 3 ############################################ # GLOBAL OPTIONS # # All are optional, with sensible defaults # ############################################ # Verbose level, 1 through 5. # 1 Quiet Print fatal errors only # 2 Default Print errors and warnings only # 3 Verbose Show equivalent shell commands being executed # 4 Extra Verbose Show extra verbose information # 5 Debug mode Everything # verbose 2 # Same as "verbose" above, but controls the amount of data sent to the # logfile, if one is being used. The default is 3. # If you want the rsync output, you have to set it to 4 # loglevel 3 # If you enable this, data will be written to the file you specify. The # amount of data written is controlled by the "loglevel" parameter. # logfile /var/log/rsnapshot.log # If enabled, rsnapshot will write a lockfile to prevent two instances # from running simultaneously (and messing up the snapshot_root). # If you enable this, make sure the lockfile directory is not world # writable. Otherwise anyone can prevent the program from running. # lockfile /var/run/rsnapshot.pid # By default, rsnapshot check lockfile, check if PID is running # and if not, consider lockfile as stale, then start # Enabling this stop rsnapshot if PID in lockfile is not running # #stop_on_stale_lockfile 0 # Default rsync args. All rsync commands have at least these options set. # rsync_short_args -av rsync_long_args --delete --numeric-ids --relative --delete-excluded # ssh has no args passed by default, but you can specify some here. # #ssh_args -p 22 # Default arguments for the "du" program (for disk space reporting). # The GNU version of "du" is preferred. See the man page for more details. # If your version of "du" doesn't support the -h flag, try -k flag instead. # du_args -csh # If this is enabled, rsync won't span filesystem partitions within a # backup point. This essentially passes the -x option to rsync. # The default is 0 (off). # #one_fs 0 # The include and exclude parameters, if enabled, simply get passed directly # to rsync. If you have multiple include/exclude patterns, put each one on a # separate line. Please look up the --include and --exclude options in the # rsync man page for more details on how to specify file name patterns. # #include ??? #include ??? exclude /lost+found exclude snapraid.* exclude lost* exclude .Trash* exclude /proc/* exclude /run/* exclude /tmp/* exclude /VMs/* # The include_file and exclude_file parameters, if enabled, simply get # passed directly to rsync. Please look up the --include-from and # --exclude-from options in the rsync man page for more details. # #include_file /path/to/include/file #exclude_file /path/to/exclude/file # If your version of rsync supports --link-dest, consider enabling this. # This is the best way to support special files (FIFOs, etc) cross-platform. # The default is 0 (off). # #link_dest 0 # When sync_first is enabled, it changes the default behaviour of rsnapshot. # Normally, when rsnapshot is called with its lowest interval # (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest # intervals. With sync_first enabled, "rsnapshot sync" handles the file sync, # and all interval calls simply rotate files. See the man page for more # details. The default is 0 (off). # #sync_first 0 # If enabled, rsnapshot will move the oldest directory for each interval # to [interval_name].delete, then it will remove the lockfile and delete # that directory just before it exits. The default is 0 (off). # #use_lazy_deletes 0 # Number of rsync re-tries. If you experience any network problems or # network card issues that tend to cause ssh to fail with errors like # "Corrupted MAC on input", for example, set this to a non-zero value # to have the rsync operation re-tried. # #rsync_numtries 0 # LVM parameters. Used to backup with creating lvm snapshot before backup # and removing it after. This should ensure consistency of data in some special # cases # # LVM snapshot(s) size (lvcreate --size option). # #linux_lvm_snapshotsize 100M # Name to be used when creating the LVM logical volume snapshot(s). # #linux_lvm_snapshotname rsnapshot # Path to the LVM Volume Groups. # #linux_lvm_vgpath /dev # Mount point to use to temporarily mount the snapshot(s). # #linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup ############################### ### BACKUP POINTS / SCRIPTS ### ############################### # LOCALHOST backup /media/disk1/Media Media/ #backup /home/ localhost/ #backup /home/ localhost/ #backup /etc/ localhost/ #backup /usr/local/ localhost/ #backup /var/log/rsnapshot localhost/ #backup /etc/passwd localhost/ #backup /home/foo/My Documents/ localhost/ #backup /foo/bar/ localhost/ one_fs=1, rsync_short_args=-urltvpog #backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/ # You must set linux_lvm_* parameters below before using lvm snapshots #backup lvm://vg0/xen-home/ lvm-vg0/xen-home/ # EXAMPLE.COM #backup_exec /bin/date "+ backup of example.com started at %c" #backup root@example.com:/home/ example.com/ +rsync_long_args=--bwlimit=16,exclude=core #backup root@example.com:/etc/ example.com/ exclude=mtab,exclude=core #backup_exec ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql" #backup root@example.com:/var/db/dump/ example.com/ #backup_exec /bin/date "+ backup of example.com ended at %c" # CVS.SOURCEFORGE.NET #backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh rsnapshot.cvs.sourceforge.net/ # RSYNC.SAMBA.ORG #backup rsync://rsync.samba.org/rsyncftp/ rsync.samba.org/rsyncftp/
config_version 1.2 snapshot_root /mnt/backup2 cmd_cp /bin/cp cmd_rm /bin/rm cmd_rsync /usr/bin/rsync cmd_logger /usr/bin/logger cmd_du /usr/bin/du cmd_rsnapshot_diff /usr/bin/rsnapshot-diff retain daily 7 retain weekly 4 retain monthly 3 verbose 2 loglevel 3 logfile /var/log/rsnapshot.log lockfile /var/run/rsnapshot.pid rsync_short_args -av rsync_long_args --delete --numeric-ids --relative --delete-excluded exclude /lost+found exclude snapraid.* exclude lost* exclude .Trash* exclude /proc/* exclude /run/* exclude /tmp/* exclude /VMs/* backup /media/disk1/Media Media/
Remember! !The rsnapshot configuration file separates with tabs not spaces!
As these backup are made to nfs mounts it is very important the at nfs drive are mounted, or the backups will go into the local mount points and fill the local disks.
String the Scripts Together into a Backup Master Script (Rsnapshot)
Do a manual run through first:
- Start-up Back up Computer
sudo bash ./Myscripts/ipmi_start_05_40.sh ; echo $?…:… See IPMI startup script for description of this script. - Mount the back-up drives to the back-up server …:… See NFS
- Mount the first back-up drive
sudo bash ~/Myscripts/mount_backup1.sh ; echo $? - Mount the Second back-up drive
sudo bash ~/Myscripts/mount_backup2.sh ; echo $?
- Perform the various backup
sudo rsnapshot -c /etc/rsnapshot1.conf dailysudo rsnapshot -c /etc/rsnapshot2.conf dailysudo rsync -av -u --delete /media/disk2/Media/ baumkp@192.168.5::all/media/Disk4/disk2 --exclude=snapraid.* --exclude=.Trash* --exclude=lost* >tmp.txt
- Unmount the backup drives on the backup servers
- sudo bash ~/Myscripts/unmount_backup1.sh
- sudo bash ~/Myscripts/unmount_backup2.sh
- Turn off the remote back-up sever, see IPMI Stop script
sudo bash ~/Myscripts/ipmi_stop_05_40.sh ; echo $?
To use:
sudo bash ~/Myscripts/server_backup.sh [<weekly> | monthly | quarterly]
sudo vim ~/Myscripts/server_backup.sh
#!/bin/bash if [ "$1" != "" ]; then case "$1" in weekly | monthly | quarterly) period="$1" ;; *) period="weekly" ;; esac else period="weekly" fi strf='/var/log/backup.log' str0='/home/baumkp' str1='/Myscripts/' ipmi_start='ipmi_start_05_40.sh' ipmi_stop='ipmi_stop_05_40.sh' # $rtn is the saved global error code from last command (failed) # $func1 is the glabal of function name that failed # $period is the rsnapshot period to be used: weekly, monthly, or quarterly : default is weekly function successtype { # This function performs the success response # $rtn is the inputed return # $strf is the log file # function input 1: # 0 : return on error with error status # 1 : (default & not 0) exit on error with error status if [ "$1" != "" ]; then case "$1" in 0) return_type="$1" ;; *) return_type=1 ;; esac else return_type=1 fi echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" if [ "$rtn" == "0" ] then echo "Successful" >>"$strf" return "$rtn" else echo "Unsuccessful" "$rtn" >>"$strf" if [ "$return_type" == "0" ] then return "$rtn" else exit "$rtn" fi fi } #Script to Startup remote IPMI Server echo "--------------------------------------------------" >>"$strf" echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "Start Backup Server" "$period" >>"$strf" func1="Remote Backup Server Started" echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "$func1" | awk '{printf "%s ", $0}' >>"$strf" cmd1="${str0}${str1}${ipmi_start}" bash "$cmd1" rtn=$? #rtn=10 # for error on test successtype #Script to mount remote backup drive func1="Mount /mntbackup1" echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "$func1" | awk '{printf "%s ", $0}' >>"$strf" if grep -qs "/mnt/backup1" /proc/mounts; then #Already mounted, log success echo "already mounted" >>"$strf" else mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1 rtn=$? successtype fi function run_cmd { # use global $strf for log file # use global $cmd1 for bash command to be run # use global $str3 for log descriptor string # use global $rtn to store function return value echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "${str3}" | awk '{printf "%s ", $0}' >>"$strf" eval "${cmd1}" #echo "$cmd1" | awk '{printf "%s ", $0}' >>"$strf" # test script rtn=$? #rtn=10 # for error on test successtype 0 return "$rtn" } #Script to run Rsnapshot Karl Data 1a cmd1="rsync -a -u --delete '/media/disk1/KarlData/Karl Data 1' 'baumkp@192.168.1.5::all/media/Disk1/Karl Data 1/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*" str3="rsync /media/disk1/KarlData/Karl Data 1" run_cmd cmd1="rsnapshot -c /etc/rsnapshot1a.conf ${period}" str3="/etc/rsnapshot1a" #backup /media/disk1/KarlData/Karl Data 1 KarlData/Karl Data 1 run_cmd #Script to run Rsnapshot Karl Data 1b cmd1="rsync -a -u --delete '/media/disk1/KarlData/Karl Data 2' 'baumkp@192.168.1.5::all/media/Disk1/Karl Data 2/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*" str3="rsync /media/disk1/KarlData/Karl Data 2" run_cmd cmd1="rsnapshot -c /etc/rsnapshot1b.conf ${period}" str3="/etc/rsnapshot1b" #backup /media/disk1/KarlData/Karl Data 2 KarlData/Karl Data 2 run_cmd #Script to run Rsnapshot Karl Data 1c cmd1="rsync -a -u --delete '/home/' 'baumkp@192.168.1.5::all/media/Disk1/home/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*" str3="rsync /home/" run_cmd cmd1="rsnapshot -c /etc/rsnapshot1c.conf ${period}" str3="/etc/rsnapshot1c" #backup /home/ home/ run_cmd #Script to unmount NFS on /mnt/backup1 echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" if grep -qs "/mnt/backup1" /proc/mounts; then /bin/umount /mnt/backup1 if [ $? -eq 0 ]; then echo "Successfully unmounted /mnt/backup1" >>"$strf" else echo "umount /mnt/backup1 failed" >>"$strf" fi else echo "/mnt/backup1 not actually mounted" >>"$strf" fi #Script to mount remote backup drive func1="Mount /mntbackup2" echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "$func1" | awk '{printf "%s ", $0}' >>"$strf" if grep -qs "/mnt/backup2" /proc/mounts; then #Already mounted, log success echo "already mounted" >>"$strf" else mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2 rtn=$? successtype fi #Script to run Rsnapshot Karl Data 2 cmd1="rsync -a -u --delete '/media/disk1/Media' 'baumkp@192.168.1.5::all/media/Disk2/Media/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*" str3="rsync /media/disk1/Media" run_cmd cmd1="/usr/bin/rsnapshot -c /etc/rsnapshot2.conf ${period}" str3="rsnapshot2" #backup /media/disk1/Media Media/ run_cmd #Script to unmount NFS on /mnt/backup2 echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" if grep -qs "/mnt/backup2" /proc/mounts; then /bin/umount /mnt/backup2 if [ $? -eq 0 ]; then echo "Successfully unmounted /mnt/backup2" >>"$strf" else echo "umount /mnt/backup2 failed" >>"$strf" fi else echo "/mnt/backup2 not actually mounted" >>"$strf" fi #Script to mount remote Storage drive func1="Mount /mnt/Sorage" echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "$func1" | awk '{printf "%s ", $0}' >>"$strf" if grep -qs "/mnt/Storage" /proc/mounts; then #Already mounted, log success echo "already mounted" >>"$strf" else mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk4 /mnt/storage rtn=$? successtype fi #rsync main media files to backup (rsync daemon is enabled service on backup server) cmd1="rsync -a -u --delete /media/disk2/Media/ baumkp@192.168.1.5::all/media/Disk4/Media/.sync --exclude=snapraid.* --exclude=.Trash* --exclude=lost*" str3="rsync /media/disk2/Media/" run_cmd cmd1="/usr/bin/rsnapshot -c /etc/rsnapshot4.conf ${period}" str3="rsnapshot4" #backup /media/disk2/Media Media/ #run_cmd # decided not to history backup this one. #Script to unmount NFS on /mnt/storage echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" if grep -qs "/mnt/storage" /proc/mounts; then /bin/umount /mnt/storage if [ $? -eq 0 ]; then echo "Successfully unmounted /mnt/storage" >>"$strf" else echo "umount /mnt/storage failed" >>"$strf" fi else echo "/mnt/storage not actually mounted" >>"$strf" fi #Script to shutdown func1="Remote Backup Server Stop" echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "$func1" | awk '{printf "%s ", $0}' >>"$strf" cmd1="${str0}${str1}${ipmi_stop}" bash "$cmd1" rtn=$? #rtn=10 # for error on test successtype echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf" echo "Backup Server Script Ended" >>"$strf" exit 0