meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
linux:backup:duply [2016/11/27 18:31] – niziaklinux:backup:duply [2021/05/10 13:53] – niziak
Line 1: Line 1:
 +====== Duply ======
 +
 +====== Installation ======
 <code bash> <code bash>
 sudo apt-get install duply python-paramiko trickle sudo apt-get install duply python-paramiko trickle
 </code> </code>
  
 +====== Configuration ======
 Create backup profile: Create backup profile:
 <code bash>duply gitlab create</code> <code bash>duply gitlab create</code>
Line 27: Line 31:
 SOURCE='/' SOURCE='/'
 MAX_AGE=6M MAX_AGE=6M
 +MAX_FULL_BACKUPS=2
 +MAX_FULLS_WITH_INCRS=2
 +
 +MAX_FULLBKP_AGE=3M
 +DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
 +
  
 VOLSIZE=256 VOLSIZE=256
 DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE " DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
 +
 +VERBOSITY=4
 +TEMP_DIR=/tmp
  
 # Specify different id_rsa file: # Specify different id_rsa file:
 DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oIdentityFile='/root/.duply/gitlab/id_rsa' " DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oIdentityFile='/root/.duply/gitlab/id_rsa' "
 DUPL_PARAMS="$DUPL_PARAMS --no-compression " # Do not use GZip to compress files on remote system. DUPL_PARAMS="$DUPL_PARAMS --no-compression " # Do not use GZip to compress files on remote system.
-DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-carcfour128 -oCompression=no"  # light cipher for NAS+DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-carcfour128 "  # light cipher for NAS 
 +DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oCompression=no "  # disable ssh compression
 DUPL_PARAMS="$DUPL_PARAMS --asynchronous-upload " # uploads in background DUPL_PARAMS="$DUPL_PARAMS --asynchronous-upload " # uploads in background
 </file> </file>
Line 46: Line 60:
 </file> </file>
  
 +**Note:** duply dosn't make any cleanup or deletions during ''backup'' action. So destination storage can be full very quickly.
 +To perform maintenance of old backup accordign to ''MAX_AGE'', ''MAX_FULL_BACKUPS'', ''MAX_FULLS_WITH_INCRS'' and ''MAX_FULLBKP_AGE'' parameters it is need to call
 +duply with ''purge'' and ''cleanup'' commands. See cron script example below.
 +
 +Example options:
 +  * MAX_FULL_BACKUPS=2
 +  * MAX_FULLS_WITH_INCRS=1
 +Will keep 2 full backup sets, but only one with increments (last one).
 +
 +Sometimes it is good to check whether incremental backups are meaningful (it depends on type of data stored). If command
 +<code bash>duply gitlab status</code> shows that number of volume with each increment is similar to full backup, then making increments has no sense, ane time to keep increments
 +can be short e.g. MAX_FULLBKP_AGE=7D
 +
 +
 +====== Usage ======
 Start the backup Start the backup
 <code bash>sudo duply gitlab backup --progress</code> <code bash>sudo duply gitlab backup --progress</code>
Line 53: Line 82:
  
 duply gitlab verify  # long operation duply gitlab verify  # long operation
 +</code>
 +
 +===== cron script =====
 +
 +<file bash>
 +#!/bin/bash -ue
 +set -o pipefail
 +trap "banner error; echo LINE: $LINENO" ERR
 +
 +duply gitlab backup
 +duply gitlab purge --force # list outdated backup archives and delete them
 +duply gitlab-to-grinnux purgeIncr --force
 +duply gitlab-to-grinnux purgeFull --force
 +duply gitlab cleanup --extra-clean --force > /dev/null # list broken backup files and delete them
 +banner ALL OK
 +</file>
 +
 +===== shell function =====
 +<code bash>
 +#!/bin/bash -ueE
 +set -o pipefail
 +trap "banner error; echo LINE: $LINENO" ERR
 +
 +run_duply() {
 +    echo "====================================================="
 +    duply ${1} backup
 +    echo "====================================================="
 +    duply ${1} cleanup --extra-clean --force
 +    duply ${1} purge --force
 +    duply ${1} purgeIncr --force
 +    duply ${1} purgeFull --force
 +    echo "====================================================="
 +    duply ${1} cleanup --extra-clean --force > /dev/null
 +    echo "====================================================="    
 +    banner ${1} OK
 +}
 </code> </code>
  
Line 99: Line 164:
  
 ====== Issues ====== ====== Issues ======
 +
 +==== no acceptable kex algorithm ====
 ssh: Exception: Incompatible ssh peer (no acceptable kex algorithm) ssh: Exception: Incompatible ssh peer (no acceptable kex algorithm)
  
Line 116: Line 183:
 </code> </code>
  
 +====  can't be deleted ====
 +<code bash> duply mybackup purge --force</code>
 +<code>
 +Last full backup date: Wed May 24 01:11:54 2017
 +There are backup set(s) at time(s):
 +Thu Nov 24 01:05:26 2016
 +Fri Nov 25 01:09:43 2016
 +Sat Nov 26 01:10:50 2016
 +Which can't be deleted because newer sets depend on them.
 +No old backup sets found, nothing deleted.
 +</code>
  
 +Solution is to run:
 +<code bash> duply mybackup purgeIncr --force</code>
 +<code bash> duply mybackup purgeFull --force</code>