{{tag>linux debian backup LVM filesystem server NFS mount showmount}}
======Back-up Server Old Setup======
This has been more difficult than I would have thought. I have used rsnapshot the past couple of years (2020-21), but have never really been happy with it for many reasons. The main reason is that it has never worked well, the rsync is reasonably fast, but the rsnapshot takes far too long. Other problems include; difficulties in set-up, and concerns will reliability. Recently I noticed that Debian no longer includes it in its repositories as of Debian 11 due to poor maintenance..... So I have no moved to Restic, which is described here: [[home_server:home_server_setup:other_services:back-up_server]].
=====overlayfs=====
I was disappointed with overlayfs, it does not seem to allow for a true merge file system, just overlay file system. The information available to set up was some of the worst I have found to date for Linux in the modern day. It is the only solution built in the Kernel. I looks like it may be of some use with containers, such as Docker. It does not really interest me at this time and I am not using.
++++tl;dr;|
After some experimenting and basic research I found the following seemed to work:
''sudo mount -t overlay -o lowerdir=/media/Disk2/backup,upperdir=/media/Disk1/backup,workdir=/media/Disk1/.backup overlay /media/Disk1/backup''
The syntax is reasonably evident from the command. The workdir is a directory on the same media as the upperdir, and is where the working files for overlayfs are kept.
Overlayfs is not what I wish to use for my back-up server.
++++
=====mergerfs=====
In the end I has stopped using mergerfs. There are other options to combine drives, e.g. lvm or simplely using symlinks.
++++tl;dr;|
Mergerfs looks more like what I wish to achieve it is a way to pool drive into a common mount point.
Below are some links with information and and possible setups.
*[[https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/|Combining Different Sized Drives with mergerfs and SnapRAID]]
*[[https://zackreed.me/mergerfs-another-good-option-to-pool-your-snapraid-disks/|Mergerfs – another good option to pool your SnapRAID disks]]
*[[https://zackreed.me/mergerfs-neat-tricks/|https://zackreed.me/mergerfs-neat-tricks/]]
*[[https://github.com/trapexit/mergerfs|trapexit / mergerfs]]
*[[https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/|Disk Pooling in Linux with mergerFS]]
*[[https://blog.linuxserver.io/2019/07/16/perfect-media-server-2019/|The Perfect Media Server - 2019 Edition]]
*Systemd startup [[https://dominicm.com/mergerfs-union-filesystem-arch-linux/|Mergerfs Union Filesystem on Arch Linux]]
*[[http://manpages.ubuntu.com/manpages/bionic/man1/mergerfs.1.html|Ubuntu Manuals mergerfs]]
Down load the current correct version of mergerfs from: ''https://github.com/trapexit/mergerfs/releases/'', hint look at the link of the file of interest....:
''wget https://github.com/trapexit/mergerfs/releases/download/2.29.0/mergerfs_2.29.0.ubuntu-xenial_amd64.deb''
Then load it into linux: ''sudo dpkg -i mergerfs_2.29.0.ubuntu-xenial_amd64.deb''
Then create a mount point directory, e.g. /srv/storage, ''sudo mkdir /srv/storage''
''sudo vim /etc/fstab''
UUID=d426f5e0-7fbd-4e7f-915e-b7c78eb70877 / ext4 errors=remount-ro 0 1
# swap was on /dev/sdd5 during installation
UUID=e3442da3-4eaf-477d-94af-6ace66097a54 none swap sw 0 0
UUID=2fedae7e-8214-4fec-a6dd-85d7026d474f /media/Disk1 ext4 defaults 0 0
UUID=38f52a66-fe96-4eb4-a794-6fe1d02bab4a /media/Disk2 ext4 defaults 0 0
UUID=1cb1509c-ec09-46cd-ab30-23f3be5b554e /media/Disk3 ext4 defaults 0 0
UUID=2ba18a1e-d48d-4122-bff8-30c2bc359749 /media/Disk4 ext4 defaults 0 0
/media/Disk1:/media/Disk2 /srv/storage fuse.mergerfs use_ino,defaults,allow_other,minfreespace=50G,fsname=mergerfs 0 0
#/media/Disk1 /export/Disk1 bind bind 0 0
#/media/Disk2 /export/Disk2 bind bind 0 0
/media/Disk4 /export/Disk4 bind bind 0 0
/home/shared /export/shared bind bind 0 0
Directly mount from CLI:
''sudo mergerfs -o defaults,allow_other,use_ino,fsname=mergerFS /media/Disk1:/media/Disk2 /srv/storage''
To unmount: ''sudo umount /srv/storage''
To check mounted disks ''df -Th''
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 16G 0 16G 0% /dev
tmpfs tmpfs 3.2G 2.7M 3.2G 1% /run
/dev/sdd1 ext4 443G 161G 260G 39% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
mergerfs fuse.mergerfs 11T 177M 11T 1% /srv/storage
/dev/sdb1 ext4 5.5T 89M 5.2T 1% /media/Disk2
/dev/sda1 ext4 5.5T 89M 5.2T 1% /media/Disk3
/dev/sdc1 ext4 5.5T 89M 5.2T 1% /media/Disk1
/dev/sde1 ext4 9.1T 1.2T 7.5T 14% /media/Disk4
tmpfs tmpfs 3.2G 0 3.2G 0% /run/user/1000
++++
=====LVM=====
With the availability of Gparted to adjust partitions on the run I do not know if LVM is required any more. It does however look like a flexible system.
|<60px 60px 20px 20px 20px 20px>|
^ Command ^ example ^ LVM ^ PV ^ VG ^ LG ^
| s | ''pvs'' | ✘ | ✔ | ✔ | ✔ |
| display | ''pvdisplay'' | ✘ | ✔ | ✔ | ✔ |
| create | ''pvcreate'' | ✘ | ✔ | ✔ | ✔ |
| rename | ''vgrename'' | ✘ | ✘ | ✔ | ✔ |
| change | ''lvchange'' | ✔ | ✔ | ✔ | ✔ |
| move | ''vgmove'' | ✘ | ✔ | ✔ | ✘ |
| extend | ''lvextend'' | ✘ | ✘ | ✔ | ✔ |
| reduce | ''lvreduce'' | ✘ | ✘ | ✔ | ✔ |
| resize | ''lvresize'' | ✘ | ✔ | ✘ | ✔ |
| split | ''vgsplit'' | ✘ | ✘ | ✔ | ✘ |
| merge | ''vgmerge'' | ✘ | ✘ | ✔ | ✘ |
| convert | ''vgconvert'' | ✘ | ✘ | ✔ | ✘ |
| import | ''vgimport'' | ✘ | ✘ | ✔ | ✘ |
| export | ''vgexport'' | ✘ | ✘ | ✔ | ✘ |
| importclone | ''vgimportclone'' | ✘ | ✘ | ✔ | ✘ |
| cfgbackup | ''vgcfgbackup'' | ✘ | ✘ | ✔ | ✘ |
| cfgrestore | ''vgcfgrestore'' | ✘ | ✘ | ✔ | ✘ |
| ck | ''vgck'' | ✘ | ✔ | ✔ | ✘ |
| scan | ''vgscan'' | 1 | ✔ | ✔ | ✔ |
| mknodes | ''vgmknode'' | ✘ | ✘ | ✔ | ✘ |
| remove | ''vgremove'' | ✘ | ✔ | ✔ | ✔ |
| dump | ''dump'' | ✔ | ✘ | ✘ | ✘ |
1. ''diskscan''
\\
Links:
*[[https://linuxhandbook.com/lvm-guide/|The Complete Beginner's Guide to LVM in Linux]]
*[[https://linuxconfig.org/using-lvm-to-manage-physical-volumes-volume-groups-and-logical-volumes-rhcsa-objective-preparation|Using LVM to manage physical volumes, volume groups, and logical volumes]]
*[[https://www.howtogeek.com/howto/40702/how-to-manage-and-use-lvm-logical-volume-management-in-ubuntu/|How to Manage and Use LVM (Logical Volume Management) in Ubuntu]]
=====NFS=====
To check available server drives use ''sudo showmount -e 192.168.1.5''
To mount a NFS manually:
*''sudo mount -t nfs -o rw,vers=4 192.168.1.5:/Disk1 /mnt/backup1''
*''sudo mount -t nfs -o rw,vers=4 192.168.1.5:/Disk2 /mnt/backup2''
Where:
* ''-t nfs'' : type nfs
* ''-o'' : options, options are separated by commas with no spaces
* ''rw'' : read / write
* ''vers=4'' : version 4 of nfs
* ''192.168.1.5:/Disk1'' is the remote exported NFS mount point (/export/Disk1 is no longer accepted syntax in NFS4.)
* ''/mnt/backup1'' : is the local mount point, note this directory must exist, ideally empty, any existing contents are usually invisible and otherwise ignored
To unmount:
*''sudo umount /mnt/backup1''
*''sudo umount /mnt/backup2''
Where:
*''/mnt/backup1'' is the local mount point to be unmounted
++++Possible mount script|
#!/bin/bash
#input 1 is the proposed mount point
if [ -z "$1" ]
then
exit 11 #no mount point input, exit with error
else
mount1=$1 #Use mount point input
fi
if grep -qs "$mount1" /proc/mounts; then
exit 0 #Already mounted, exit with success
else
mount "$mount1"
if [ $? -eq 0 ]; then
exit 0 # Successfully mounted, exit with success
else
exit 12 # mount failed, exit with error
fi
fi
++++
Created script to mount backup1 ''sudo bash ~/Myscripts/mount_backup1.sh'' to run
++++''sudo vim ~/Myscripts/mount_backup1.sh''|
#!/bin/bash
#Script to mount NFS remote Disk1 to local /mnt/backup1
if grep -qs "/mnt/backup1" /proc/mounts; then
exit 0 #Already mounted, exit with success
else
mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1
if [ $? -eq 0 ]; then
exit 0 # Successfully mounted, exit with success
else
exit 12 # mount failed, exit with error
fi
fi
++++
Created script to mount backup2 ''sudo bash ~/Myscripts/mount_backup2.sh'' to run
++++''sudo vim ~/Myscripts/mount_backup2.sh''|
#!/bin/bash
#Script to mount NFS remote Disk2 to local /mnt/backup2
if grep -qs "/mnt/backup2" /proc/mounts; then
exit 0 #Already mounted, exit with success
else
mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2
if [ $? -eq 0 ]; then
exit 0 # Successfully mounted, exit with success
else
exit 12 # mount failed, exit with error
fi
fi
++++
As we intend to shutdown the backup server it is good to unmount the back-up drives first.
Created script to unmount backup1 ''sudo ~/Myscripts/unmount_backup1.sh''
++++''sudo vim ~/Myscripts/unmount_backup1.sh''|
#!/bin/bash
#Script to mount NFS remote Disk2 to local /mnt/backup2
if grep -qs "/mnt/backup1" /proc/mounts; then
umount /mnt/backup1
if [ $? -eq 0 ]; then
exit 0 # Successfully unmounted, exit with success
else
exit 12 # umount failed, exit with error
fi
else
exit 0 # Not actually mounted, exit with success
fi
++++
Created script to unmount backup2 ''sudo ~/Myscripts/unmount_backup2.sh''
++++''sudo vim ~/Myscripts/unmount_backup2.sh''|
#!/bin/bash
#Script to mount NFS remote Disk2 to local /mnt/backup2
if grep -qs "/mnt/backup2" /proc/mounts; then
umount /mnt/backup2
if [ $? -eq 0 ]; then
exit 0 # Successfully unmounted, exit with success
else
exit 12 # umount failed, exit with error
fi
else
exit 0 # Not actually mounted, exit with success
fi
++++
=====Backup main Media drive (Rsync / Rsnapshot method)=====
Sadly I have found Rsnapshot and Rsync is not suitable for large backup service. Rsnapshot is simply too slow and I am concerned with reliability. Rsnapshot configuration is overly complex, with strange requirements such as tab for white space. As of Debian 11 it is no longer even included in the Debian repositories for lack of maintenance. It feels effectively abandoned.\\
++++Superseded, tl;dr;|
I do not use Rsnapshot anymore. I just do not feel comfortable using it.\\
The following is the command I use to backup my main media drive: ''sudo rsync -av -u %%--%%delete /media/disk2/Media/ baumkp@192.168.1.5::all/media/Disk4/disk2 %%--%%exclude=snapraid.* %%--%%exclude=.Trash* %%--%%exclude=lost* >tmp.txt''
I have rsync setup running as a daemon on 192.168.1.5 with login details for baumkp setup on the sever accepting input from the local area network to just the one address 192.168.1.10.
On the remote machine 192.168.1.5:
''sudo vim /etc/rsyncd.conf''
# create new
# any name you like
[all]
# destination directory to copy
path = /
# hosts you allow to access
hosts allow = 192.168.1.10
#hosts deny = *
list = true
uid = root
gid = root
read only = false
#auth users = baumkp
#secrets file = /etc/rsyncd.secrets
''sudo vim /etc/rsyncd.secrets''
username:userpassword
Use systemd systemctl from control of the deamon: ''sudo systemctl status rsync.service'' [status, start, stop, restart, reload, disable, enable] I have enabled so the rsync deamon this normally running.
The local directory /media/disk2/Media/ contents are to be backed up.
++++
----
====rsnapshot backups====
I have 2 rsnapshot backup configurations;
''/etc/rsnapshot1.conf''
To run locally: ''sudo rsnapshot -c /etc/rsnapshot1.conf daily''
''sudo vim /etc/rsnapshot1.conf''
++++with all comments|
#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
# #
# PLEASE BE AWARE OF THE FOLLOWING RULE: #
# #
# This file requires tabs between elements #
# #
#################################################
#######################
# CONFIG FILE VERSION #
#######################
config_version 1.2
###########################
# SNAPSHOT ROOT DIRECTORY #
###########################
# All snapshots will be stored under this root directory.
#
# snapshot_root /var/cache/rsnapshot/ #default
snapshot_root /mnt/backup1
# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
#no_create_root 1
#################################
# EXTERNAL PROGRAM DEPENDENCIES #
#################################
# LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features.
# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
#
# See the README file or the man page for more details.
#
cmd_cp /bin/cp
# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm /bin/rm
# rsync must be enabled for anything to work. This is the only command that
# must be enabled.
#
cmd_rsync /usr/bin/rsync
# Uncomment this to enable remote ssh backups over rsync.
#
#cmd_ssh /usr/bin/ssh
# Comment this out to disable syslog support.
#
cmd_logger /usr/bin/logger
# Uncomment this to specify the path to "du" for disk usage checks.
# If you have an older version of "du", you may also want to check the
# "du_args" parameter below.
#
cmd_du /usr/bin/du
# Uncomment this to specify the path to rsnapshot-diff.
#
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
#
#cmd_preexec /path/to/preexec/script
# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
#
#cmd_postexec /path/to/postexec/script
# Paths to lvcreate, lvremove, mount and umount commands, for use with
# Linux LVMs.
#
#linux_lvm_cmd_lvcreate /sbin/lvcreate
#linux_lvm_cmd_lvremove /sbin/lvremove
#linux_lvm_cmd_mount /bin/mount
#linux_lvm_cmd_umount /bin/umount
#########################################
# BACKUP LEVELS / INTERVALS #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc. #
#########################################
#retain alpha 6
#retain beta 7
#retain gamma 4
#retain delta 3
#retain hourly 6
retain daily 7
retain weekly 4
retain monthly 3
############################################
# GLOBAL OPTIONS #
# All are optional, with sensible defaults #
############################################
# Verbose level, 1 through 5.
# 1 Quiet Print fatal errors only
# 2 Default Print errors and warnings only
# 3 Verbose Show equivalent shell commands being executed
# 4 Extra Verbose Show extra verbose information
# 5 Debug mode Everything
#
verbose 2
# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
# If you want the rsync output, you have to set it to 4
#
loglevel 3
# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile /var/log/rsnapshot.log
# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile /var/run/rsnapshot.pid
# By default, rsnapshot check lockfile, check if PID is running
# and if not, consider lockfile as stale, then start
# Enabling this stop rsnapshot if PID in lockfile is not running
#
#stop_on_stale_lockfile 0
# Default rsync args. All rsync commands have at least these options set.
#
rsync_short_args -av
rsync_long_args --delete --numeric-ids --relative --delete-excluded
# ssh has no args passed by default, but you can specify some here.
#
#ssh_args -p 22
# Default arguments for the "du" program (for disk space reporting).
# The GNU version of "du" is preferred. See the man page for more details.
# If your version of "du" doesn't support the -h flag, try -k flag instead.
#
du_args -csh
# If this is enabled, rsync won't span filesystem partitions within a
# backup point. This essentially passes the -x option to rsync.
# The default is 0 (off).
#
#one_fs 0
# The include and exclude parameters, if enabled, simply get passed directly
# to rsync. If you have multiple include/exclude patterns, put each one on a
# separate line. Please look up the --include and --exclude options in the
# rsync man page for more details on how to specify file name patterns.
#
#include ???
#include ???
exclude /lost+found
exclude snapraid.*
exclude lost*
exclude .Trash*
exclude /proc/*
exclude /run/*
exclude /tmp/*
exclude /VMs/*
# The include_file and exclude_file parameters, if enabled, simply get
# passed directly to rsync. Please look up the --include-from and
# --exclude-from options in the rsync man page for more details.
#
#include_file /path/to/include/file
#exclude_file /path/to/exclude/file
# If your version of rsync supports --link-dest, consider enabling this.
# This is the best way to support special files (FIFOs, etc) cross-platform.
# The default is 0 (off).
#
#link_dest 0
# When sync_first is enabled, it changes the default behaviour of rsnapshot.
# Normally, when rsnapshot is called with its lowest interval
# (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync,
# and all interval calls simply rotate files. See the man page for more
# details. The default is 0 (off).
#
#sync_first 0
# If enabled, rsnapshot will move the oldest directory for each interval
# to [interval_name].delete, then it will remove the lockfile and delete
# that directory just before it exits. The default is 0 (off).
#
#use_lazy_deletes 0
# Number of rsync re-tries. If you experience any network problems or
# network card issues that tend to cause ssh to fail with errors like
# "Corrupted MAC on input", for example, set this to a non-zero value
# to have the rsync operation re-tried.
#
#rsync_numtries 0
# LVM parameters. Used to backup with creating lvm snapshot before backup
# and removing it after. This should ensure consistency of data in some special
# cases
#
# LVM snapshot(s) size (lvcreate --size option).
#
#linux_lvm_snapshotsize 100M
# Name to be used when creating the LVM logical volume snapshot(s).
#
#linux_lvm_snapshotname rsnapshot
# Path to the LVM Volume Groups.
#
#linux_lvm_vgpath /dev
# Mount point to use to temporarily mount the snapshot(s).
#
#linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup
###############################
### BACKUP POINTS / SCRIPTS ###
###############################
# LOCALHOST
backup /media/disk1/KarlData KarlData/
backup /home/ home/
#backup /home/ localhost/
#backup /etc/ localhost/
#backup /usr/local/ localhost/
#backup /var/log/rsnapshot localhost/
#backup /etc/passwd localhost/
#backup /home/foo/My Documents/ localhost/
#backup /foo/bar/ localhost/ one_fs=1, rsync_short_args=-urltvpog
#backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/
# You must set linux_lvm_* parameters below before using lvm snapshots
#backup lvm://vg0/xen-home/ lvm-vg0/xen-home/
# EXAMPLE.COM
#backup_exec /bin/date "+ backup of example.com started at %c"
#backup root@example.com:/home/ example.com/ +rsync_long_args=--bwlimit=16,exclude=core
#backup root@example.com:/etc/ example.com/ exclude=mtab,exclude=core
#backup_exec ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql"
#backup root@example.com:/var/db/dump/ example.com/
#backup_exec /bin/date "+ backup of example.com ended at %c"
# CVS.SOURCEFORGE.NET
#backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh rsnapshot.cvs.sourceforge.net/
# RSYNC.SAMBA.ORG
#backup rsync://rsync.samba.org/rsyncftp/ rsync.samba.org/rsyncftp/
++++
++++with no comments|
config_version 1.2
snapshot_root /mnt/backup1
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
retain daily 7
retain weekly 4
retain monthly 3
verbose 2
loglevel 3
logfile /var/log/rsnapshot.log
lockfile /var/run/rsnapshot.pid
rsync_short_args -av
rsync_long_args --delete --numeric-ids --relative --delete-excluded
exclude /lost+found
exclude snapraid.*
exclude lost*
exclude .Trash*
exclude /proc/*
exclude /run/*
exclude /tmp/*
exclude /VMs/*
backup /media/disk1/KarlData KarlData/
backup /home/ home/
**Remember! //!The rsnapshot configuration file separates with tabs not spaces!//**
++++
''/etc/rsnapshot2.conf''
To run locally: ''sudo rsnapshot -c /etc/rsnapshot2.conf daily''
''sudo vim /etc/rsnapshot2.conf''
++++with all comments|
#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
# #
# PLEASE BE AWARE OF THE FOLLOWING RULE: #
# #
# This file requires tabs between elements #
# #
#################################################
#######################
# CONFIG FILE VERSION #
#######################
config_version 1.2
###########################
# SNAPSHOT ROOT DIRECTORY #
###########################
# All snapshots will be stored under this root directory.
#
# snapshot_root /var/cache/rsnapshot/ #default
snapshot_root /mnt/backup2
# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
#no_create_root 1
#################################
# EXTERNAL PROGRAM DEPENDENCIES #
#################################
# LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features.
# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
#
# See the README file or the man page for more details.
#
cmd_cp /bin/cp
# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm /bin/rm
# rsync must be enabled for anything to work. This is the only command that
# must be enabled.
#
cmd_rsync /usr/bin/rsync
# Uncomment this to enable remote ssh backups over rsync.
#
#cmd_ssh /usr/bin/ssh
# Comment this out to disable syslog support.
#
cmd_logger /usr/bin/logger
# Uncomment this to specify the path to "du" for disk usage checks.
# If you have an older version of "du", you may also want to check the
# "du_args" parameter below.
#
cmd_du /usr/bin/du
# Uncomment this to specify the path to rsnapshot-diff.
#
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
#
#cmd_preexec /path/to/preexec/script
# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
#
#cmd_postexec /path/to/postexec/script
# Paths to lvcreate, lvremove, mount and umount commands, for use with
# Linux LVMs.
#
#linux_lvm_cmd_lvcreate /sbin/lvcreate
#linux_lvm_cmd_lvremove /sbin/lvremove
#linux_lvm_cmd_mount /bin/mount
#linux_lvm_cmd_umount /bin/umount
#########################################
# BACKUP LEVELS / INTERVALS #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc. #
#########################################
#retain alpha 6
#retain beta 7
#retain gamma 4
#retain delta 3
#retain hourly 6
retain daily 7
retain weekly 4
retain monthly 3
############################################
# GLOBAL OPTIONS #
# All are optional, with sensible defaults #
############################################
# Verbose level, 1 through 5.
# 1 Quiet Print fatal errors only
# 2 Default Print errors and warnings only
# 3 Verbose Show equivalent shell commands being executed
# 4 Extra Verbose Show extra verbose information
# 5 Debug mode Everything
#
verbose 2
# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
# If you want the rsync output, you have to set it to 4
#
loglevel 3
# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile /var/log/rsnapshot.log
# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile /var/run/rsnapshot.pid
# By default, rsnapshot check lockfile, check if PID is running
# and if not, consider lockfile as stale, then start
# Enabling this stop rsnapshot if PID in lockfile is not running
#
#stop_on_stale_lockfile 0
# Default rsync args. All rsync commands have at least these options set.
#
rsync_short_args -av
rsync_long_args --delete --numeric-ids --relative --delete-excluded
# ssh has no args passed by default, but you can specify some here.
#
#ssh_args -p 22
# Default arguments for the "du" program (for disk space reporting).
# The GNU version of "du" is preferred. See the man page for more details.
# If your version of "du" doesn't support the -h flag, try -k flag instead.
#
du_args -csh
# If this is enabled, rsync won't span filesystem partitions within a
# backup point. This essentially passes the -x option to rsync.
# The default is 0 (off).
#
#one_fs 0
# The include and exclude parameters, if enabled, simply get passed directly
# to rsync. If you have multiple include/exclude patterns, put each one on a
# separate line. Please look up the --include and --exclude options in the
# rsync man page for more details on how to specify file name patterns.
#
#include ???
#include ???
exclude /lost+found
exclude snapraid.*
exclude lost*
exclude .Trash*
exclude /proc/*
exclude /run/*
exclude /tmp/*
exclude /VMs/*
# The include_file and exclude_file parameters, if enabled, simply get
# passed directly to rsync. Please look up the --include-from and
# --exclude-from options in the rsync man page for more details.
#
#include_file /path/to/include/file
#exclude_file /path/to/exclude/file
# If your version of rsync supports --link-dest, consider enabling this.
# This is the best way to support special files (FIFOs, etc) cross-platform.
# The default is 0 (off).
#
#link_dest 0
# When sync_first is enabled, it changes the default behaviour of rsnapshot.
# Normally, when rsnapshot is called with its lowest interval
# (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync,
# and all interval calls simply rotate files. See the man page for more
# details. The default is 0 (off).
#
#sync_first 0
# If enabled, rsnapshot will move the oldest directory for each interval
# to [interval_name].delete, then it will remove the lockfile and delete
# that directory just before it exits. The default is 0 (off).
#
#use_lazy_deletes 0
# Number of rsync re-tries. If you experience any network problems or
# network card issues that tend to cause ssh to fail with errors like
# "Corrupted MAC on input", for example, set this to a non-zero value
# to have the rsync operation re-tried.
#
#rsync_numtries 0
# LVM parameters. Used to backup with creating lvm snapshot before backup
# and removing it after. This should ensure consistency of data in some special
# cases
#
# LVM snapshot(s) size (lvcreate --size option).
#
#linux_lvm_snapshotsize 100M
# Name to be used when creating the LVM logical volume snapshot(s).
#
#linux_lvm_snapshotname rsnapshot
# Path to the LVM Volume Groups.
#
#linux_lvm_vgpath /dev
# Mount point to use to temporarily mount the snapshot(s).
#
#linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup
###############################
### BACKUP POINTS / SCRIPTS ###
###############################
# LOCALHOST
backup /media/disk1/Media Media/
#backup /home/ localhost/
#backup /home/ localhost/
#backup /etc/ localhost/
#backup /usr/local/ localhost/
#backup /var/log/rsnapshot localhost/
#backup /etc/passwd localhost/
#backup /home/foo/My Documents/ localhost/
#backup /foo/bar/ localhost/ one_fs=1, rsync_short_args=-urltvpog
#backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/
# You must set linux_lvm_* parameters below before using lvm snapshots
#backup lvm://vg0/xen-home/ lvm-vg0/xen-home/
# EXAMPLE.COM
#backup_exec /bin/date "+ backup of example.com started at %c"
#backup root@example.com:/home/ example.com/ +rsync_long_args=--bwlimit=16,exclude=core
#backup root@example.com:/etc/ example.com/ exclude=mtab,exclude=core
#backup_exec ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql"
#backup root@example.com:/var/db/dump/ example.com/
#backup_exec /bin/date "+ backup of example.com ended at %c"
# CVS.SOURCEFORGE.NET
#backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh rsnapshot.cvs.sourceforge.net/
# RSYNC.SAMBA.ORG
#backup rsync://rsync.samba.org/rsyncftp/ rsync.samba.org/rsyncftp/
++++
++++with no comments|
config_version 1.2
snapshot_root /mnt/backup2
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
retain daily 7
retain weekly 4
retain monthly 3
verbose 2
loglevel 3
logfile /var/log/rsnapshot.log
lockfile /var/run/rsnapshot.pid
rsync_short_args -av
rsync_long_args --delete --numeric-ids --relative --delete-excluded
exclude /lost+found
exclude snapraid.*
exclude lost*
exclude .Trash*
exclude /proc/*
exclude /run/*
exclude /tmp/*
exclude /VMs/*
backup /media/disk1/Media Media/
**Remember! //!The rsnapshot configuration file separates with tabs not spaces!//**
++++
As these backup are made to nfs mounts it is very important the at nfs drive are mounted, or the backups will go into the local mount points and fill the local disks.
----
====String the Scripts Together into a Backup Master Script (Rsnapshot)====
++++Superseded, tl;dr;|
Do a manual run through first:
- Start-up Back up Computer ''sudo bash ./Myscripts/ipmi_start_05_40.sh ; echo $?'' ...:... See [[https://wiki.kptree.net/doku.php?id=home_server:home_server_setup:other_services:ipmi#ipmi_start_script|IPMI startup script]] for description of this script.
- Mount the back-up drives to the back-up server ...:... See [[https://wiki.kptree.net/doku.php?id=home_server:home_server_setup:other_services:back-up_server#nfs| NFS]]
- Mount the first back-up drive ''sudo bash ~/Myscripts/mount_backup1.sh ; echo $?''
- Mount the Second back-up drive ''sudo bash ~/Myscripts/mount_backup2.sh ; echo $?''
- Perform the various backup
- ''sudo rsnapshot -c /etc/rsnapshot1.conf daily''
- ''sudo rsnapshot -c /etc/rsnapshot2.conf daily''
- ''sudo rsync -av -u %%--%%delete /media/disk2/Media/ baumkp@192.168.5::all/media/Disk4/disk2 %%--%%exclude=snapraid.* %%--%%exclude=.Trash* %%--%%exclude=lost* >tmp.txt''
- Unmount the backup drives on the backup servers
- sudo bash ~/Myscripts/unmount_backup1.sh
- sudo bash ~/Myscripts/unmount_backup2.sh
- Turn off the remote back-up sever, see [[https://wiki.kptree.net/doku.php?id=home_server:home_server_setup:other_services:ipmi#ipmi_stop_script|IPMI Stop script]]
- ''sudo bash ~/Myscripts/ipmi_stop_05_40.sh ; echo $?''
To use:
''sudo bash ~/Myscripts/server_backup.sh [ | monthly | quarterly]''
++++
''sudo vim ~/Myscripts/server_backup.sh''
++++ server_backup.sh code|
#!/bin/bash
if [ "$1" != "" ]; then
case "$1" in
weekly | monthly | quarterly)
period="$1"
;;
*)
period="weekly"
;;
esac
else
period="weekly"
fi
strf='/var/log/backup.log'
str0='/home/baumkp'
str1='/Myscripts/'
ipmi_start='ipmi_start_05_40.sh'
ipmi_stop='ipmi_stop_05_40.sh'
# $rtn is the saved global error code from last command (failed)
# $func1 is the glabal of function name that failed
# $period is the rsnapshot period to be used: weekly, monthly, or quarterly : default is weekly
function successtype
{
# This function performs the success response
# $rtn is the inputed return
# $strf is the log file
# function input 1:
# 0 : return on error with error status
# 1 : (default & not 0) exit on error with error status
if [ "$1" != "" ]; then
case "$1" in
0)
return_type="$1"
;;
*)
return_type=1
;;
esac
else
return_type=1
fi
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
if [ "$rtn" == "0" ]
then
echo "Successful" >>"$strf"
return "$rtn"
else
echo "Unsuccessful" "$rtn" >>"$strf"
if [ "$return_type" == "0" ]
then
return "$rtn"
else
exit "$rtn"
fi
fi
}
#Script to Startup remote IPMI Server
echo "--------------------------------------------------" >>"$strf"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "Start Backup Server" "$period" >>"$strf"
func1="Remote Backup Server Started"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "$func1" | awk '{printf "%s ", $0}' >>"$strf"
cmd1="${str0}${str1}${ipmi_start}"
bash "$cmd1"
rtn=$?
#rtn=10 # for error on test
successtype
#Script to mount remote backup drive
func1="Mount /mntbackup1"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "$func1" | awk '{printf "%s ", $0}' >>"$strf"
if grep -qs "/mnt/backup1" /proc/mounts; then
#Already mounted, log success
echo "already mounted" >>"$strf"
else
mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1
rtn=$?
successtype
fi
function run_cmd
{
# use global $strf for log file
# use global $cmd1 for bash command to be run
# use global $str3 for log descriptor string
# use global $rtn to store function return value
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "${str3}" | awk '{printf "%s ", $0}' >>"$strf"
eval "${cmd1}"
#echo "$cmd1" | awk '{printf "%s ", $0}' >>"$strf" # test script
rtn=$?
#rtn=10 # for error on test
successtype 0
return "$rtn"
}
#Script to run Rsnapshot Karl Data 1a
cmd1="rsync -a -u --delete '/media/disk1/KarlData/Karl Data 1' 'baumkp@192.168.1.5::all/media/Disk1/Karl Data 1/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk1/KarlData/Karl Data 1"
run_cmd
cmd1="rsnapshot -c /etc/rsnapshot1a.conf ${period}"
str3="/etc/rsnapshot1a"
#backup /media/disk1/KarlData/Karl Data 1 KarlData/Karl Data 1
run_cmd
#Script to run Rsnapshot Karl Data 1b
cmd1="rsync -a -u --delete '/media/disk1/KarlData/Karl Data 2' 'baumkp@192.168.1.5::all/media/Disk1/Karl Data 2/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk1/KarlData/Karl Data 2"
run_cmd
cmd1="rsnapshot -c /etc/rsnapshot1b.conf ${period}"
str3="/etc/rsnapshot1b"
#backup /media/disk1/KarlData/Karl Data 2 KarlData/Karl Data 2
run_cmd
#Script to run Rsnapshot Karl Data 1c
cmd1="rsync -a -u --delete '/home/' 'baumkp@192.168.1.5::all/media/Disk1/home/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /home/"
run_cmd
cmd1="rsnapshot -c /etc/rsnapshot1c.conf ${period}"
str3="/etc/rsnapshot1c"
#backup /home/ home/
run_cmd
#Script to unmount NFS on /mnt/backup1
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
if grep -qs "/mnt/backup1" /proc/mounts; then
/bin/umount /mnt/backup1
if [ $? -eq 0 ]; then
echo "Successfully unmounted /mnt/backup1" >>"$strf"
else
echo "umount /mnt/backup1 failed" >>"$strf"
fi
else
echo "/mnt/backup1 not actually mounted" >>"$strf"
fi
#Script to mount remote backup drive
func1="Mount /mntbackup2"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "$func1" | awk '{printf "%s ", $0}' >>"$strf"
if grep -qs "/mnt/backup2" /proc/mounts; then
#Already mounted, log success
echo "already mounted" >>"$strf"
else
mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2
rtn=$?
successtype
fi
#Script to run Rsnapshot Karl Data 2
cmd1="rsync -a -u --delete '/media/disk1/Media' 'baumkp@192.168.1.5::all/media/Disk2/Media/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk1/Media"
run_cmd
cmd1="/usr/bin/rsnapshot -c /etc/rsnapshot2.conf ${period}"
str3="rsnapshot2"
#backup /media/disk1/Media Media/
run_cmd
#Script to unmount NFS on /mnt/backup2
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
if grep -qs "/mnt/backup2" /proc/mounts; then
/bin/umount /mnt/backup2
if [ $? -eq 0 ]; then
echo "Successfully unmounted /mnt/backup2" >>"$strf"
else
echo "umount /mnt/backup2 failed" >>"$strf"
fi
else
echo "/mnt/backup2 not actually mounted" >>"$strf"
fi
#Script to mount remote Storage drive
func1="Mount /mnt/Sorage"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "$func1" | awk '{printf "%s ", $0}' >>"$strf"
if grep -qs "/mnt/Storage" /proc/mounts; then
#Already mounted, log success
echo "already mounted" >>"$strf"
else
mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk4 /mnt/storage
rtn=$?
successtype
fi
#rsync main media files to backup (rsync daemon is enabled service on backup server)
cmd1="rsync -a -u --delete /media/disk2/Media/ baumkp@192.168.1.5::all/media/Disk4/Media/.sync --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk2/Media/"
run_cmd
cmd1="/usr/bin/rsnapshot -c /etc/rsnapshot4.conf ${period}"
str3="rsnapshot4"
#backup /media/disk2/Media Media/
#run_cmd # decided not to history backup this one.
#Script to unmount NFS on /mnt/storage
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
if grep -qs "/mnt/storage" /proc/mounts; then
/bin/umount /mnt/storage
if [ $? -eq 0 ]; then
echo "Successfully unmounted /mnt/storage" >>"$strf"
else
echo "umount /mnt/storage failed" >>"$strf"
fi
else
echo "/mnt/storage not actually mounted" >>"$strf"
fi
#Script to shutdown
func1="Remote Backup Server Stop"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "$func1" | awk '{printf "%s ", $0}' >>"$strf"
cmd1="${str0}${str1}${ipmi_stop}"
bash "$cmd1"
rtn=$?
#rtn=10 # for error on test
successtype
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s ", $0}' >>"$strf"
echo "Backup Server Script Ended" >>"$strf"
exit 0
++++
----
<- home_server:home_server_setup:other_services:nut|Prev ^ home_server:home_server_setup:other_services:index|Start page ^ home_server:home_server_setup:other_services:back-up_server|Next ->