KPTree Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{tag>linux debian backup LVM filesystem server NFS mount showmount}}
======Back-up Server Old Setup======
This has been more difficult than I would have thought. I have used rsnapshot the past couple of years (2020-21), but have never really been happy with it for many reasons. The main reason is that it has never worked well, the rsync is reasonably fast, but the rsnapshot takes far too long. Other problems include; difficulties in set-up, and concerns will reliability. Recently I noticed that Debian no longer includes it in its repositories as of Debian 11 due to poor maintenance.....  So I have no moved to Restic, which is described here: [[home_server:home_server_setup:other_services:back-up_server]].

=====overlayfs=====
I was disappointed with overlayfs, it does not seem to allow for a true merge file system, just overlay file system.  The information available to set up was some of the worst I have found to date for Linux in the modern day.  It is the only solution built in the Kernel.  I looks like it may be of some use with containers, such as Docker.  It does not really interest me at this time and I am not using.
++++tl;dr;|
After some experimenting and basic research I found the following seemed to work:

''sudo mount -t overlay -o lowerdir=/media/Disk2/backup,upperdir=/media/Disk1/backup,workdir=/media/Disk1/.backup overlay /media/Disk1/backup''

The syntax is reasonably evident from the command.  The workdir is a directory on the same media as the upperdir, and is where the working files for overlayfs are kept. 

Overlayfs is not what I wish to use for my back-up server.
++++

=====mergerfs=====
In the end I has stopped using mergerfs. There are other options to combine drives, e.g. lvm or simplely using symlinks.
++++tl;dr;|
Mergerfs looks more like what I wish to achieve it is a way to pool drive into a common mount point.
Below are some links with information and and possible setups.
  *[[https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/|Combining Different Sized Drives with mergerfs and SnapRAID]]
  *[[https://zackreed.me/mergerfs-another-good-option-to-pool-your-snapraid-disks/|Mergerfs – another good option to pool your SnapRAID disks]]
  *[[https://zackreed.me/mergerfs-neat-tricks/|https://zackreed.me/mergerfs-neat-tricks/]]
  *[[https://github.com/trapexit/mergerfs|trapexit / mergerfs]]
  *[[https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/|Disk Pooling in Linux with mergerFS]]
  *[[https://blog.linuxserver.io/2019/07/16/perfect-media-server-2019/|The Perfect Media Server - 2019 Edition]]
  *Systemd startup [[https://dominicm.com/mergerfs-union-filesystem-arch-linux/|Mergerfs Union Filesystem on Arch Linux]]
  *[[http://manpages.ubuntu.com/manpages/bionic/man1/mergerfs.1.html|Ubuntu Manuals mergerfs]]

Down load the current correct version of mergerfs from: ''https://github.com/trapexit/mergerfs/releases/'', hint look at the link of the file of interest....:  
''wget https://github.com/trapexit/mergerfs/releases/download/2.29.0/mergerfs_2.29.0.ubuntu-xenial_amd64.deb''

Then load it into linux: ''sudo dpkg -i mergerfs_2.29.0.ubuntu-xenial_amd64.deb''

Then create a mount point directory, e.g. /srv/storage,  ''sudo mkdir /srv/storage''
''sudo vim /etc/fstab''
<code>
UUID=d426f5e0-7fbd-4e7f-915e-b7c78eb70877 /                   ext4    errors=remount-ro 0       1
# swap was on /dev/sdd5 during installation
UUID=e3442da3-4eaf-477d-94af-6ace66097a54        none           swap    sw              0       0
UUID=2fedae7e-8214-4fec-a6dd-85d7026d474f       /media/Disk1    ext4    defaults        0       0
UUID=38f52a66-fe96-4eb4-a794-6fe1d02bab4a       /media/Disk2    ext4    defaults        0       0
UUID=1cb1509c-ec09-46cd-ab30-23f3be5b554e       /media/Disk3    ext4    defaults        0       0
UUID=2ba18a1e-d48d-4122-bff8-30c2bc359749       /media/Disk4    ext4    defaults        0       0
/media/Disk1:/media/Disk2 /srv/storage fuse.mergerfs use_ino,defaults,allow_other,minfreespace=50G,fsname=mergerfs 0 0
#/media/Disk1           /export/Disk1   bind    bind    0       0
#/media/Disk2           /export/Disk2   bind    bind    0       0
/media/Disk4            /export/Disk4   bind    bind    0       0
/home/shared            /export/shared  bind    bind    0       0
</code>

Directly mount from CLI: 
''sudo mergerfs -o defaults,allow_other,use_ino,fsname=mergerFS /media/Disk1:/media/Disk2 /srv/storage''

To unmount: ''sudo umount /srv/storage''

To check mounted disks ''df -Th''
<code>
Filesystem     Type           Size  Used Avail Use% Mounted on
udev           devtmpfs        16G     0   16G   0% /dev
tmpfs          tmpfs          3.2G  2.7M  3.2G   1% /run
/dev/sdd1      ext4           443G  161G  260G  39% /
tmpfs          tmpfs           16G     0   16G   0% /dev/shm
tmpfs          tmpfs          5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs           16G     0   16G   0% /sys/fs/cgroup
mergerfs       fuse.mergerfs   11T  177M   11T   1% /srv/storage
/dev/sdb1      ext4           5.5T   89M  5.2T   1% /media/Disk2
/dev/sda1      ext4           5.5T   89M  5.2T   1% /media/Disk3
/dev/sdc1      ext4           5.5T   89M  5.2T   1% /media/Disk1
/dev/sde1      ext4           9.1T  1.2T  7.5T  14% /media/Disk4
tmpfs          tmpfs          3.2G     0  3.2G   0% /run/user/1000
</code>
++++

=====LVM=====
With the availability of Gparted to adjust partitions on the run I do not know if LVM is required any more. It does however look like a flexible system.
|<60px 60px 20px 20px 20px 20px>|
^ Command     ^  example    ^  LVM  ^  PV  ^  VG  ^  LG  ^
| s           |  ''pvs''            |  ✘  |  ✔  |  ✔  |  ✔  |
| display     |  ''pvdisplay''      |  ✘  |  ✔  |  ✔  |  ✔  |
| create      |  ''pvcreate''       |  ✘  |  ✔  |  ✔  |  ✔  |
| rename      |  ''vgrename''       |  ✘  |  ✘  |  ✔  |  ✔  |
| change      |  ''lvchange''       |  ✔  |  ✔  |  ✔  |  ✔  |
| move        |  ''vgmove''         |  ✘  |  ✔  |  ✔  |  ✘  |
| extend      |  ''lvextend''       |  ✘  |  ✘  |  ✔  |  ✔  |
| reduce      |  ''lvreduce''       |  ✘  |  ✘  |  ✔  |  ✔  |
| resize      |  ''lvresize''       |  ✘  |  ✔  |  ✘  |  ✔  |
| split       |  ''vgsplit''        |  ✘  |  ✘  |  ✔  |  ✘  |
| merge       |  ''vgmerge''        |  ✘  |  ✘  |  ✔  |  ✘  |
| convert     |  ''vgconvert''      |  ✘  |  ✘  |  ✔  |  ✘  |
| import      |  ''vgimport''       |  ✘  |  ✘  |  ✔  |  ✘  |
| export      |  ''vgexport''       |  ✘  |  ✘  |  ✔  |  ✘  |
| importclone |  ''vgimportclone''  |  ✘  |  ✘  |  ✔  |  ✘  |
| cfgbackup   |  ''vgcfgbackup''    |  ✘  |  ✘  |  ✔  |  ✘  |
| cfgrestore  |  ''vgcfgrestore''   |  ✘  |  ✘  |  ✔  |  ✘  |
| ck          |  ''vgck''           |  ✘  |  ✔  |  ✔  |  ✘  |
| scan        |  ''vgscan''         |  1  |  ✔  |  ✔  |  ✔  |
| mknodes     |  ''vgmknode''       |  ✘  |  ✘  |  ✔  |  ✘  |
| remove      |  ''vgremove''       |  ✘  |  ✔  |  ✔  |  ✔  |
| dump        |  ''dump''           |  ✔  |  ✘  |  ✘  |  ✘  |
1. ''diskscan''
 \\

Links:
  *[[https://linuxhandbook.com/lvm-guide/|The Complete Beginner's Guide to LVM in Linux]]
  *[[https://linuxconfig.org/using-lvm-to-manage-physical-volumes-volume-groups-and-logical-volumes-rhcsa-objective-preparation|Using LVM to manage physical volumes, volume groups, and logical volumes]]
  *[[https://www.howtogeek.com/howto/40702/how-to-manage-and-use-lvm-logical-volume-management-in-ubuntu/|How to Manage and Use LVM (Logical Volume Management) in Ubuntu]]
=====NFS=====
To check available server drives use ''sudo showmount -e 192.168.1.5''


To mount a NFS manually: 
  *''sudo mount -t nfs -o rw,vers=4 192.168.1.5:/Disk1 /mnt/backup1''
  *''sudo mount -t nfs -o rw,vers=4 192.168.1.5:/Disk2 /mnt/backup2''

Where:
  * ''-t nfs'' : type nfs
  * ''-o'' : options, options are separated by commas with no spaces
    * ''rw'' : read / write
    * ''vers=4'' : version 4 of nfs
  * ''192.168.1.5:/Disk1'' is the remote exported NFS mount point (/export/Disk1 is no longer accepted syntax in NFS4.)
  * ''/mnt/backup1'' : is the local mount point, note this directory must exist, ideally empty, any existing contents are usually invisible and otherwise ignored

To unmount: 
  *''sudo umount /mnt/backup1''
  *''sudo umount /mnt/backup2''

Where:
  *''/mnt/backup1'' is the local mount point to be unmounted

++++Possible mount script|
<code bash>
#!/bin/bash

#input 1 is the proposed mount point

if [ -z "$1" ]
then 
   exit 11  #no mount point input, exit with error
else
   mount1=$1 #Use mount point input
fi

if grep -qs "$mount1" /proc/mounts; then
  exit 0  #Already mounted, exit with success
else
  mount "$mount1"
  if [ $? -eq 0 ]; then
   exit 0  # Successfully mounted, exit with success
  else
   exit 12 # mount failed, exit with error
  fi
fi
</code>
++++

Created script to mount backup1 ''sudo bash ~/Myscripts/mount_backup1.sh'' to run
++++''sudo vim ~/Myscripts/mount_backup1.sh''|
<code bash>
#!/bin/bash
#Script to mount NFS remote Disk1 to local /mnt/backup1

if grep -qs "/mnt/backup1" /proc/mounts; then
  exit 0  #Already mounted, exit with success
else
  mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1
  if [ $? -eq 0 ]; then
   exit 0  # Successfully mounted, exit with success
  else
   exit 12 # mount failed, exit with error
  fi
fi
</code>
++++

Created script to mount backup2 ''sudo bash ~/Myscripts/mount_backup2.sh'' to run
++++''sudo vim ~/Myscripts/mount_backup2.sh''|
<code bash>
#!/bin/bash
#Script to mount NFS remote Disk2 to local /mnt/backup2

if grep -qs "/mnt/backup2" /proc/mounts; then
  exit 0  #Already mounted, exit with success
else
  mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2
  if [ $? -eq 0 ]; then
   exit 0  # Successfully mounted, exit with success
  else
   exit 12 # mount failed, exit with error
  fi
fi
</code>
++++

As we intend to shutdown the backup server it is good to unmount the back-up drives first.

Created script to unmount backup1 ''sudo ~/Myscripts/unmount_backup1.sh''
++++''sudo vim ~/Myscripts/unmount_backup1.sh''|
<code bash>
#!/bin/bash
#Script to mount NFS remote Disk2 to local /mnt/backup2

if grep -qs "/mnt/backup1" /proc/mounts; then
  umount /mnt/backup1
  if [ $? -eq 0 ]; then
   exit 0  # Successfully unmounted, exit with success
  else
   exit 12 # umount failed, exit with error
  fi
else
  exit 0 # Not actually mounted, exit with success
fi
</code>
++++

Created script to unmount backup2 ''sudo ~/Myscripts/unmount_backup2.sh''
++++''sudo vim ~/Myscripts/unmount_backup2.sh''|
<code bash>
#!/bin/bash
#Script to mount NFS remote Disk2 to local /mnt/backup2

if grep -qs "/mnt/backup2" /proc/mounts; then
  umount /mnt/backup2
  if [ $? -eq 0 ]; then
   exit 0  # Successfully unmounted, exit with success
  else
   exit 12 # umount failed, exit with error
  fi
else
  exit 0 # Not actually mounted, exit with success
fi
</code>
++++

=====Backup main Media drive (Rsync / Rsnapshot method)=====
Sadly I have found Rsnapshot and Rsync is not suitable for large backup service.  Rsnapshot is simply too slow and I am concerned with reliability. Rsnapshot configuration is overly complex, with strange requirements such as tab for white space.  As of Debian 11 it is no longer even included in the Debian repositories for lack of maintenance. It feels effectively abandoned.\\

++++Superseded, tl;dr;|
<fs large><fc #ff0000>I do not use Rsnapshot anymore.  I just do not feel comfortable using it.</fc></fs>\\
The following is the command I use to backup my main media drive: ''sudo rsync -av -u %%--%%delete /media/disk2/Media/ baumkp@192.168.1.5::all/media/Disk4/disk2 %%--%%exclude=snapraid.* %%--%%exclude=.Trash* %%--%%exclude=lost* >tmp.txt''

I have rsync setup running as a daemon on 192.168.1.5 with login details for baumkp setup on the sever accepting input from the local area network to just the one address 192.168.1.10.
On the remote machine 192.168.1.5:
''sudo vim /etc/rsyncd.conf''
<code>
# create new
  
# any name you like
[all]
# destination directory to copy
path = /
# hosts you allow to access
hosts allow = 192.168.1.10
#hosts deny = *
list = true
uid = root
gid = root
read only = false
#auth users = baumkp
#secrets file = /etc/rsyncd.secrets
</code>

''sudo vim /etc/rsyncd.secrets''
<code>username:userpassword</code>

Use systemd systemctl from control of the deamon: ''sudo systemctl status rsync.service''   [status, start, stop, restart, reload, disable, enable]  I have enabled so the rsync deamon this normally running.


The local directory /media/disk2/Media/ contents are to be backed up.
++++
----
====rsnapshot backups====
I have 2 rsnapshot backup configurations;

''/etc/rsnapshot1.conf''
To run locally: ''sudo rsnapshot -c /etc/rsnapshot1.conf daily''

''sudo vim /etc/rsnapshot1.conf''
++++with all comments|
<code rsnapshot>
#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
#                                               #
# PLEASE BE AWARE OF THE FOLLOWING RULE:        #
#                                               #
# This file requires tabs between elements      #
#                                               #
#################################################

#######################
# CONFIG FILE VERSION #
#######################

config_version  1.2

###########################
# SNAPSHOT ROOT DIRECTORY #
###########################

# All snapshots will be stored under this root directory.
#
# snapshot_root /var/cache/rsnapshot/ #default 
snapshot_root   /mnt/backup1

# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
#no_create_root 1

#################################
# EXTERNAL PROGRAM DEPENDENCIES #
#################################

# LINUX USERS:   Be sure to uncomment "cmd_cp". This gives you extra features.
# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
#
# See the README file or the man page for more details.
#
cmd_cp          /bin/cp

# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm          /bin/rm

# rsync must be enabled for anything to work. This is the only command that
# must be enabled.
#
cmd_rsync       /usr/bin/rsync

# Uncomment this to enable remote ssh backups over rsync.
#
#cmd_ssh        /usr/bin/ssh

# Comment this out to disable syslog support.
#
cmd_logger      /usr/bin/logger

# Uncomment this to specify the path to "du" for disk usage checks.
# If you have an older version of "du", you may also want to check the
# "du_args" parameter below.
#
cmd_du          /usr/bin/du

# Uncomment this to specify the path to rsnapshot-diff.
#
cmd_rsnapshot_diff      /usr/bin/rsnapshot-diff

# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
#
#cmd_preexec    /path/to/preexec/script

# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
#
#cmd_postexec   /path/to/postexec/script

# Paths to lvcreate, lvremove, mount and umount commands, for use with
# Linux LVMs.
#
#linux_lvm_cmd_lvcreate /sbin/lvcreate
#linux_lvm_cmd_lvremove /sbin/lvremove
#linux_lvm_cmd_mount    /bin/mount
#linux_lvm_cmd_umount   /bin/umount
#########################################
#     BACKUP LEVELS / INTERVALS         #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc.         #
#########################################

#retain alpha   6
#retain beta    7
#retain gamma   4
#retain delta   3

#retain hourly  6
retain  daily   7
retain  weekly  4
retain  monthly 3


############################################
#              GLOBAL OPTIONS              #
# All are optional, with sensible defaults #
############################################

# Verbose level, 1 through 5.
# 1     Quiet           Print fatal errors only
# 2     Default         Print errors and warnings only
# 3     Verbose         Show equivalent shell commands being executed
# 4     Extra Verbose   Show extra verbose information
# 5     Debug mode      Everything
#
verbose         2

# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
# If you want the rsync output, you have to set it to 4
#
loglevel        3

# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile         /var/log/rsnapshot.log

# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile        /var/run/rsnapshot.pid

# By default, rsnapshot check lockfile, check if PID is running
# and if not, consider lockfile as stale, then start
# Enabling this stop rsnapshot if PID in lockfile is not running
#
#stop_on_stale_lockfile         0

# Default rsync args. All rsync commands have at least these options set.
#
rsync_short_args        -av
rsync_long_args --delete        --numeric-ids   --relative      --delete-excluded

# ssh has no args passed by default, but you can specify some here.
#
#ssh_args       -p 22

# Default arguments for the "du" program (for disk space reporting).
# The GNU version of "du" is preferred. See the man page for more details.
# If your version of "du" doesn't support the -h flag, try -k flag instead.
#
du_args         -csh

# If this is enabled, rsync won't span filesystem partitions within a
# backup point. This essentially passes the -x option to rsync.
# The default is 0 (off).
#
#one_fs         0

# The include and exclude parameters, if enabled, simply get passed directly
# to rsync. If you have multiple include/exclude patterns, put each one on a
# separate line. Please look up the --include and --exclude options in the
# rsync man page for more details on how to specify file name patterns. 
# 
#include        ???
#include        ???
exclude         /lost+found
exclude         snapraid.*
exclude         lost*
exclude         .Trash*
exclude         /proc/*
exclude         /run/*
exclude         /tmp/*
exclude         /VMs/*

# The include_file and exclude_file parameters, if enabled, simply get
# passed directly to rsync. Please look up the --include-from and
# --exclude-from options in the rsync man page for more details.
#
#include_file   /path/to/include/file
#exclude_file   /path/to/exclude/file

# If your version of rsync supports --link-dest, consider enabling this.
# This is the best way to support special files (FIFOs, etc) cross-platform.
# The default is 0 (off).
#
#link_dest      0

# When sync_first is enabled, it changes the default behaviour of rsnapshot.
# Normally, when rsnapshot is called with its lowest interval
# (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync,
# and all interval calls simply rotate files. See the man page for more
# details. The default is 0 (off).
#
#sync_first     0

# If enabled, rsnapshot will move the oldest directory for each interval
# to [interval_name].delete, then it will remove the lockfile and delete
# that directory just before it exits. The default is 0 (off).
#
#use_lazy_deletes       0

# Number of rsync re-tries. If you experience any network problems or
# network card issues that tend to cause ssh to fail with errors like
# "Corrupted MAC on input", for example, set this to a non-zero value
# to have the rsync operation re-tried.
#
#rsync_numtries 0

# LVM parameters. Used to backup with creating lvm snapshot before backup
# and removing it after. This should ensure consistency of data in some special
# cases
#
# LVM snapshot(s) size (lvcreate --size option).
#
#linux_lvm_snapshotsize 100M

# Name to be used when creating the LVM logical volume snapshot(s).
#
#linux_lvm_snapshotname rsnapshot

# Path to the LVM Volume Groups.
#
#linux_lvm_vgpath       /dev

# Mount point to use to temporarily mount the snapshot(s).
#
#linux_lvm_mountpath    /path/to/mount/lvm/snapshot/during/backup

###############################
### BACKUP POINTS / SCRIPTS ###
###############################

# LOCALHOST
backup  /media/disk1/KarlData   KarlData/
backup  /home/                  home/
#backup /home/          localhost/
#backup /etc/           localhost/
#backup /usr/local/     localhost/
#backup /var/log/rsnapshot              localhost/
#backup /etc/passwd     localhost/
#backup /home/foo/My Documents/         localhost/
#backup /foo/bar/       localhost/      one_fs=1, rsync_short_args=-urltvpog
#backup_script  /usr/local/bin/backup_pgsql.sh  localhost/postgres/
# You must set linux_lvm_* parameters below before using lvm snapshots
#backup lvm://vg0/xen-home/     lvm-vg0/xen-home/

# EXAMPLE.COM
#backup_exec    /bin/date "+ backup of example.com started at %c"
#backup root@example.com:/home/ example.com/    +rsync_long_args=--bwlimit=16,exclude=core
#backup root@example.com:/etc/  example.com/    exclude=mtab,exclude=core
#backup_exec    ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql"
#backup root@example.com:/var/db/dump/  example.com/
#backup_exec    /bin/date "+ backup of example.com ended at %c"

# CVS.SOURCEFORGE.NET
#backup_script  /usr/local/bin/backup_rsnapshot_cvsroot.sh      rsnapshot.cvs.sourceforge.net/

# RSYNC.SAMBA.ORG
#backup rsync://rsync.samba.org/rsyncftp/       rsync.samba.org/rsyncftp/
</code>
++++

++++with no comments|
<code>
config_version  1.2

snapshot_root   /mnt/backup1
cmd_cp          /bin/cp

cmd_rm          /bin/rm

cmd_rsync       /usr/bin/rsync

cmd_logger      /usr/bin/logger

cmd_du          /usr/bin/du

cmd_rsnapshot_diff      /usr/bin/rsnapshot-diff

retain  daily   7
retain  weekly  4
retain  monthly 3

verbose         2

loglevel        3

logfile         /var/log/rsnapshot.log

lockfile        /var/run/rsnapshot.pid

rsync_short_args        -av
rsync_long_args --delete        --numeric-ids   --relative      --delete-excluded

exclude         /lost+found
exclude         snapraid.*
exclude         lost*
exclude         .Trash*
exclude         /proc/*
exclude         /run/*
exclude         /tmp/*
exclude         /VMs/*

backup  /media/disk1/KarlData   KarlData/
backup  /home/                  home/
</code>

**Remember! //!The rsnapshot configuration file separates with tabs not spaces!//**

++++

''/etc/rsnapshot2.conf''
To run locally: ''sudo rsnapshot -c /etc/rsnapshot2.conf daily''

''sudo vim /etc/rsnapshot2.conf''
++++with all comments|
<code rsnapshot>
#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
#                                               #
# PLEASE BE AWARE OF THE FOLLOWING RULE:        #
#                                               #
# This file requires tabs between elements      #
#                                               #
#################################################

#######################
# CONFIG FILE VERSION #
#######################

config_version  1.2

###########################
# SNAPSHOT ROOT DIRECTORY #
###########################

# All snapshots will be stored under this root directory.
#
# snapshot_root /var/cache/rsnapshot/ #default 
snapshot_root   /mnt/backup2

# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
#no_create_root 1

#################################
# EXTERNAL PROGRAM DEPENDENCIES #
#################################

# LINUX USERS:   Be sure to uncomment "cmd_cp". This gives you extra features.
# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
#
# See the README file or the man page for more details.
#
cmd_cp          /bin/cp

# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm          /bin/rm

# rsync must be enabled for anything to work. This is the only command that
# must be enabled.
#
cmd_rsync       /usr/bin/rsync

# Uncomment this to enable remote ssh backups over rsync.
#
#cmd_ssh        /usr/bin/ssh

# Comment this out to disable syslog support.
#
cmd_logger      /usr/bin/logger

# Uncomment this to specify the path to "du" for disk usage checks.
# If you have an older version of "du", you may also want to check the
# "du_args" parameter below.
#
cmd_du          /usr/bin/du

# Uncomment this to specify the path to rsnapshot-diff.
#
cmd_rsnapshot_diff      /usr/bin/rsnapshot-diff

# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
#
#cmd_preexec    /path/to/preexec/script

# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
#
#cmd_postexec   /path/to/postexec/script

# Paths to lvcreate, lvremove, mount and umount commands, for use with
# Linux LVMs.
#
#linux_lvm_cmd_lvcreate /sbin/lvcreate
#linux_lvm_cmd_lvremove /sbin/lvremove
#linux_lvm_cmd_mount    /bin/mount
#linux_lvm_cmd_umount   /bin/umount

#########################################
#     BACKUP LEVELS / INTERVALS         #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc.         #
#########################################
#retain alpha   6
#retain beta    7
#retain gamma   4
#retain delta   3

#retain hourly  6
retain  daily   7
retain  weekly  4
retain  monthly 3


############################################
#              GLOBAL OPTIONS              #
# All are optional, with sensible defaults #
############################################

# Verbose level, 1 through 5.
# 1     Quiet           Print fatal errors only
# 2     Default         Print errors and warnings only
# 3     Verbose         Show equivalent shell commands being executed
# 4     Extra Verbose   Show extra verbose information
# 5     Debug mode      Everything
#
verbose         2

# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
# If you want the rsync output, you have to set it to 4
#
loglevel        3

# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile         /var/log/rsnapshot.log

# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile        /var/run/rsnapshot.pid

# By default, rsnapshot check lockfile, check if PID is running
# and if not, consider lockfile as stale, then start
# Enabling this stop rsnapshot if PID in lockfile is not running
#
#stop_on_stale_lockfile         0

# Default rsync args. All rsync commands have at least these options set.
#
rsync_short_args        -av
rsync_long_args --delete        --numeric-ids   --relative      --delete-excluded

# ssh has no args passed by default, but you can specify some here.
#
#ssh_args       -p 22

# Default arguments for the "du" program (for disk space reporting).
# The GNU version of "du" is preferred. See the man page for more details.
# If your version of "du" doesn't support the -h flag, try -k flag instead.
#
du_args         -csh

# If this is enabled, rsync won't span filesystem partitions within a
# backup point. This essentially passes the -x option to rsync.
# The default is 0 (off).
#
#one_fs         0

# The include and exclude parameters, if enabled, simply get passed directly
# to rsync. If you have multiple include/exclude patterns, put each one on a
# separate line. Please look up the --include and --exclude options in the
# rsync man page for more details on how to specify file name patterns. 
# 
#include        ???
#include        ???
exclude         /lost+found
exclude         snapraid.*
exclude         lost*
exclude         .Trash*
exclude         /proc/*
exclude         /run/*
exclude         /tmp/*
exclude         /VMs/*

# The include_file and exclude_file parameters, if enabled, simply get
# passed directly to rsync. Please look up the --include-from and
# --exclude-from options in the rsync man page for more details.
#
#include_file   /path/to/include/file
#exclude_file   /path/to/exclude/file

# If your version of rsync supports --link-dest, consider enabling this.
# This is the best way to support special files (FIFOs, etc) cross-platform.
# The default is 0 (off).
#
#link_dest      0

# When sync_first is enabled, it changes the default behaviour of rsnapshot.
# Normally, when rsnapshot is called with its lowest interval
# (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest
# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync,
# and all interval calls simply rotate files. See the man page for more
# details. The default is 0 (off).
#
#sync_first     0

# If enabled, rsnapshot will move the oldest directory for each interval
# to [interval_name].delete, then it will remove the lockfile and delete
# that directory just before it exits. The default is 0 (off).
#
#use_lazy_deletes       0

# Number of rsync re-tries. If you experience any network problems or
# network card issues that tend to cause ssh to fail with errors like
# "Corrupted MAC on input", for example, set this to a non-zero value
# to have the rsync operation re-tried.
#
#rsync_numtries 0

# LVM parameters. Used to backup with creating lvm snapshot before backup
# and removing it after. This should ensure consistency of data in some special
# cases
#
# LVM snapshot(s) size (lvcreate --size option).
#
#linux_lvm_snapshotsize 100M

# Name to be used when creating the LVM logical volume snapshot(s).
#
#linux_lvm_snapshotname rsnapshot

# Path to the LVM Volume Groups.
#
#linux_lvm_vgpath       /dev

# Mount point to use to temporarily mount the snapshot(s).
#
#linux_lvm_mountpath    /path/to/mount/lvm/snapshot/during/backup

###############################
### BACKUP POINTS / SCRIPTS ###
###############################

# LOCALHOST
backup  /media/disk1/Media      Media/
#backup /home/          localhost/
#backup /home/          localhost/
#backup /etc/           localhost/
#backup /usr/local/     localhost/
#backup /var/log/rsnapshot              localhost/
#backup /etc/passwd     localhost/
#backup /home/foo/My Documents/         localhost/
#backup /foo/bar/       localhost/      one_fs=1, rsync_short_args=-urltvpog
#backup_script  /usr/local/bin/backup_pgsql.sh  localhost/postgres/
# You must set linux_lvm_* parameters below before using lvm snapshots
#backup lvm://vg0/xen-home/     lvm-vg0/xen-home/

# EXAMPLE.COM
#backup_exec    /bin/date "+ backup of example.com started at %c"
#backup root@example.com:/home/ example.com/    +rsync_long_args=--bwlimit=16,exclude=core
#backup root@example.com:/etc/  example.com/    exclude=mtab,exclude=core
#backup_exec    ssh root@example.com "mysqldump -A > /var/db/dump/mysql.sql"
#backup root@example.com:/var/db/dump/  example.com/
#backup_exec    /bin/date "+ backup of example.com ended at %c"

# CVS.SOURCEFORGE.NET
#backup_script  /usr/local/bin/backup_rsnapshot_cvsroot.sh      rsnapshot.cvs.sourceforge.net/

# RSYNC.SAMBA.ORG
#backup rsync://rsync.samba.org/rsyncftp/       rsync.samba.org/rsyncftp/
</code>
++++
++++with no comments|
<code>
config_version  1.2

snapshot_root   /mnt/backup2
cmd_cp          /bin/cp

cmd_rm          /bin/rm

cmd_rsync       /usr/bin/rsync

cmd_logger      /usr/bin/logger

cmd_du          /usr/bin/du

cmd_rsnapshot_diff      /usr/bin/rsnapshot-diff

retain  daily   7
retain  weekly  4
retain  monthly 3

verbose         2

loglevel        3

logfile         /var/log/rsnapshot.log

lockfile        /var/run/rsnapshot.pid

rsync_short_args        -av
rsync_long_args --delete        --numeric-ids   --relative      --delete-excluded

exclude         /lost+found
exclude         snapraid.*
exclude         lost*
exclude         .Trash*
exclude         /proc/*
exclude         /run/*
exclude         /tmp/*
exclude         /VMs/*

backup  /media/disk1/Media      Media/
</code>

**Remember! //!The rsnapshot configuration file separates with tabs not spaces!//**
++++

As these backup are made to nfs mounts it is very important the at nfs drive are mounted, or the backups will go into the local mount points and fill the local disks.
----

====String the Scripts Together into a Backup Master Script (Rsnapshot)====
++++Superseded, tl;dr;|
Do a manual run through first:
  - Start-up Back up Computer ''sudo bash ./Myscripts/ipmi_start_05_40.sh ; echo $?'' ...:... See [[https://wiki.kptree.net/doku.php?id=home_server:home_server_setup:other_services:ipmi#ipmi_start_script|IPMI startup script]] for description of this script.
  - Mount the back-up drives to the back-up server ...:... See [[https://wiki.kptree.net/doku.php?id=home_server:home_server_setup:other_services:back-up_server#nfs| NFS]]
    - Mount the first back-up drive ''sudo bash ~/Myscripts/mount_backup1.sh ; echo $?''
    - Mount the Second back-up drive ''sudo bash ~/Myscripts/mount_backup2.sh ; echo $?''
  - Perform the various backup
    - ''sudo rsnapshot -c /etc/rsnapshot1.conf daily''
    - ''sudo rsnapshot -c /etc/rsnapshot2.conf daily''
    - ''sudo rsync -av -u %%--%%delete /media/disk2/Media/ baumkp@192.168.5::all/media/Disk4/disk2 %%--%%exclude=snapraid.* %%--%%exclude=.Trash* %%--%%exclude=lost* >tmp.txt''
  - Unmount the backup drives on the backup servers
    - sudo bash ~/Myscripts/unmount_backup1.sh
    - sudo bash ~/Myscripts/unmount_backup2.sh
  - Turn off the remote back-up sever, see [[https://wiki.kptree.net/doku.php?id=home_server:home_server_setup:other_services:ipmi#ipmi_stop_script|IPMI Stop script]]
    - ''sudo bash ~/Myscripts/ipmi_stop_05_40.sh ; echo $?''


To use:
''sudo bash ~/Myscripts/server_backup.sh [<weekly> | monthly | quarterly]''
++++

''sudo vim ~/Myscripts/server_backup.sh''
++++ server_backup.sh code|
<code bash>
#!/bin/bash

if [ "$1" != "" ]; then
    case "$1" in
        weekly | monthly | quarterly)
               period="$1"
        ;;
        *)
               period="weekly"
        ;;
    esac
else
    period="weekly"
fi

strf='/var/log/backup.log'
str0='/home/baumkp'
str1='/Myscripts/'
ipmi_start='ipmi_start_05_40.sh'
ipmi_stop='ipmi_stop_05_40.sh'
# $rtn  is the saved global error code from last command (failed)
# $func1 is the glabal of function name that failed 
# $period is the rsnapshot period to be used: weekly, monthly, or quarterly : default is weekly 

function successtype 
{
# This function performs the success response
# $rtn is the inputed return
# $strf is the log file
# function input 1:
  # 0 : return on error with error status
  # 1 : (default & not 0) exit on error with error status

if [ "$1" != "" ]; then
    case "$1" in
        0)
               return_type="$1"
        ;;
        *)
               return_type=1
        ;;
    esac
else
    return_type=1
fi

echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"

if [ "$rtn" == "0" ]
  then
    echo "Successful"  >>"$strf"
    return "$rtn"
  else
    echo "Unsuccessful" "$rtn"  >>"$strf"
    if [ "$return_type" == "0" ]
      then
        return "$rtn"
      else
        exit "$rtn"
    fi
fi
}

#Script to Startup remote IPMI Server

echo "--------------------------------------------------"  >>"$strf"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "Start Backup Server" "$period"   >>"$strf"

func1="Remote Backup Server Started"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "$func1" | awk '{printf "%s  ", $0}'           >>"$strf"
cmd1="${str0}${str1}${ipmi_start}"
bash "$cmd1"
rtn=$?
#rtn=10  # for error on test
successtype

#Script to mount remote backup drive
func1="Mount /mntbackup1"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "$func1" | awk '{printf "%s  ", $0}'           >>"$strf"

if grep -qs "/mnt/backup1" /proc/mounts; then
    #Already mounted, log success
    echo "already mounted"   >>"$strf"
  else
    mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk1 /mnt/backup1
    rtn=$?
    successtype
  fi

function run_cmd
{
  # use global $strf for log file
  # use global $cmd1 for bash command to be run
  # use global $str3 for log descriptor string
  # use global $rtn to store function return value

  echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
  echo "${str3}" | awk '{printf "%s ", $0}'   >>"$strf"

  eval "${cmd1}"
  #echo "$cmd1" | awk '{printf "%s  ", $0}'           >>"$strf" # test script
  rtn=$?
  #rtn=10 # for error on test
  successtype 0
  return "$rtn"
}

#Script to run Rsnapshot Karl Data 1a
cmd1="rsync -a -u --delete '/media/disk1/KarlData/Karl Data 1' 'baumkp@192.168.1.5::all/media/Disk1/Karl Data 1/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk1/KarlData/Karl Data 1"
run_cmd
cmd1="rsnapshot -c /etc/rsnapshot1a.conf ${period}"
str3="/etc/rsnapshot1a"
#backup  /media/disk1/KarlData/Karl Data 1       KarlData/Karl Data 1
run_cmd


#Script to run Rsnapshot Karl Data 1b
cmd1="rsync -a -u --delete '/media/disk1/KarlData/Karl Data 2' 'baumkp@192.168.1.5::all/media/Disk1/Karl Data 2/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk1/KarlData/Karl Data 2"
run_cmd
cmd1="rsnapshot -c /etc/rsnapshot1b.conf ${period}"
str3="/etc/rsnapshot1b"
#backup  /media/disk1/KarlData/Karl Data 2       KarlData/Karl Data 2
run_cmd

#Script to run Rsnapshot Karl Data 1c
cmd1="rsync -a -u --delete '/home/' 'baumkp@192.168.1.5::all/media/Disk1/home/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /home/"
run_cmd
cmd1="rsnapshot -c /etc/rsnapshot1c.conf ${period}"
str3="/etc/rsnapshot1c"
#backup  /home/      home/
run_cmd

#Script to unmount NFS on /mnt/backup1
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
if grep -qs "/mnt/backup1" /proc/mounts; then
  /bin/umount /mnt/backup1
  if [ $? -eq 0 ]; then
    echo "Successfully unmounted /mnt/backup1"     >>"$strf"
  else
    echo "umount /mnt/backup1 failed"     >>"$strf"
  fi
else
  echo "/mnt/backup1 not actually mounted"     >>"$strf"
fi

#Script to mount remote backup drive
func1="Mount /mntbackup2"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "$func1" | awk '{printf "%s  ", $0}'           >>"$strf"

if grep -qs "/mnt/backup2" /proc/mounts; then
    #Already mounted, log success
    echo "already mounted"   >>"$strf"
  else
    mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk2 /mnt/backup2
    rtn=$?
    successtype
  fi

#Script to run Rsnapshot Karl Data 2
cmd1="rsync -a -u --delete '/media/disk1/Media' 'baumkp@192.168.1.5::all/media/Disk2/Media/.sync' --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk1/Media"
run_cmd
cmd1="/usr/bin/rsnapshot -c /etc/rsnapshot2.conf ${period}"
str3="rsnapshot2"
#backup  /media/disk1/Media      Media/
run_cmd

#Script to unmount NFS on /mnt/backup2
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
if grep -qs "/mnt/backup2" /proc/mounts; then
  /bin/umount /mnt/backup2
  if [ $? -eq 0 ]; then
    echo "Successfully unmounted /mnt/backup2"     >>"$strf"
  else
    echo "umount /mnt/backup2 failed"     >>"$strf"
  fi
else
  echo "/mnt/backup2 not actually mounted"     >>"$strf"
fi

#Script to mount remote Storage drive
func1="Mount /mnt/Sorage"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "$func1" | awk '{printf "%s  ", $0}'           >>"$strf"

if grep -qs "/mnt/Storage" /proc/mounts; then
    #Already mounted, log success
    echo "already mounted"   >>"$strf"
  else
    mount -t nfs -o rw,vers=4 192.168.1.5:/export/Disk4 /mnt/storage
    rtn=$?
    successtype
  fi

#rsync main media files to backup (rsync daemon is enabled service on backup server)
cmd1="rsync -a -u --delete /media/disk2/Media/ baumkp@192.168.1.5::all/media/Disk4/Media/.sync --exclude=snapraid.* --exclude=.Trash* --exclude=lost*"
str3="rsync /media/disk2/Media/"
run_cmd
cmd1="/usr/bin/rsnapshot -c /etc/rsnapshot4.conf ${period}"
str3="rsnapshot4"
#backup  /media/disk2/Media      Media/
#run_cmd # decided not to history backup this one.

#Script to unmount NFS on /mnt/storage
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
if grep -qs "/mnt/storage" /proc/mounts; then
  /bin/umount /mnt/storage
  if [ $? -eq 0 ]; then
     echo "Successfully unmounted /mnt/storage"     >>"$strf"
  else
     echo "umount /mnt/storage failed"     >>"$strf"
  fi
else
  echo "/mnt/storage not actually mounted"     >>"$strf"
fi

#Script to shutdown 

func1="Remote Backup Server Stop"
echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "$func1" | awk '{printf "%s  ", $0}'           >>"$strf"
cmd1="${str0}${str1}${ipmi_stop}"
bash "$cmd1"
rtn=$?
#rtn=10  # for error on test
successtype


echo $(date +"%Y-%m-%d %H:%M:%S") | awk '{printf "%s  ", $0}'           >>"$strf"
echo "Backup Server Script Ended"   >>"$strf"

exit 0
</code>
++++

----

<- home_server:home_server_setup:other_services:nut|Prev ^ home_server:home_server_setup:other_services:index|Start page ^ home_server:home_server_setup:other_services:back-up_server|Next ->