meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
linux:backup:duply [2016/11/27 18:31] – niziaklinux:backup:duply [2021/05/10 13:57] (current) – niziak
Line 1: Line 1:
 +====== Duply ======
 +
 +====== Installation ======
 <code bash> <code bash>
 sudo apt-get install duply python-paramiko trickle sudo apt-get install duply python-paramiko trickle
 </code> </code>
  
 +====== Configuration ======
 Create backup profile: Create backup profile:
 <code bash>duply gitlab create</code> <code bash>duply gitlab create</code>
 Profile file ''$HOME/.duply/gitlab/conf'' was created. Profile file ''$HOME/.duply/gitlab/conf'' was created.
 +
 +  IMPORTANT
 +       Copy the whole profile folder after the first backup to a safe place.  It contains everything needed to restore your backups. You will need it if you have to restore the backup from another system
 +       (e.g. after a system crash). Keep access to these files restricted as they contain all information (gpg data, ftp data) to access and modify your backups.
 +
 +       Repeat this step after all configuration changes. Some configuration options are crucial for restoration.
 +
  
 Generate random password: Generate random password:
Line 27: Line 38:
 SOURCE='/' SOURCE='/'
 MAX_AGE=6M MAX_AGE=6M
 +MAX_FULL_BACKUPS=2
 +MAX_FULLS_WITH_INCRS=2
 +
 +MAX_FULLBKP_AGE=3M
 +DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
 +
  
 VOLSIZE=256 VOLSIZE=256
 DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE " DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
 +
 +VERBOSITY=4
 +TEMP_DIR=/tmp
  
 # Specify different id_rsa file: # Specify different id_rsa file:
 DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oIdentityFile='/root/.duply/gitlab/id_rsa' " DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oIdentityFile='/root/.duply/gitlab/id_rsa' "
 DUPL_PARAMS="$DUPL_PARAMS --no-compression " # Do not use GZip to compress files on remote system. DUPL_PARAMS="$DUPL_PARAMS --no-compression " # Do not use GZip to compress files on remote system.
-DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-carcfour128 -oCompression=no"  # light cipher for NAS+DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-carcfour128 "  # light cipher for NAS 
 +DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oCompression=no "  # disable ssh compression
 DUPL_PARAMS="$DUPL_PARAMS --asynchronous-upload " # uploads in background DUPL_PARAMS="$DUPL_PARAMS --asynchronous-upload " # uploads in background
 </file> </file>
Line 46: Line 67:
 </file> </file>
  
 +**Note:** duply dosn't make any cleanup or deletions during ''backup'' action. So destination storage can be full very quickly.
 +To perform maintenance of old backup accordign to ''MAX_AGE'', ''MAX_FULL_BACKUPS'', ''MAX_FULLS_WITH_INCRS'' and ''MAX_FULLBKP_AGE'' parameters it is need to call
 +duply with ''purge'' and ''cleanup'' commands. See cron script example below.
 +
 +Example options:
 +  * MAX_FULL_BACKUPS=2
 +  * MAX_FULLS_WITH_INCRS=1
 +Will keep 2 full backup sets, but only one with increments (last one).
 +
 +Sometimes it is good to check whether incremental backups are meaningful (it depends on type of data stored). If command
 +<code bash>duply gitlab status</code> shows that number of volume with each increment is similar to full backup, then making increments has no sense, ane time to keep increments
 +can be short e.g. MAX_FULLBKP_AGE=7D
 +
 +
 +====== Usage ======
 Start the backup Start the backup
 <code bash>sudo duply gitlab backup --progress</code> <code bash>sudo duply gitlab backup --progress</code>
Line 53: Line 89:
  
 duply gitlab verify  # long operation duply gitlab verify  # long operation
 +</code>
 +
 +===== cron script =====
 +
 +<file bash>
 +#!/bin/bash -ue
 +set -o pipefail
 +trap "banner error; echo LINE: $LINENO" ERR
 +
 +duply gitlab backup
 +duply gitlab purge --force # list outdated backup archives and delete them
 +duply gitlab-to-grinnux purgeIncr --force
 +duply gitlab-to-grinnux purgeFull --force
 +duply gitlab cleanup --extra-clean --force > /dev/null # list broken backup files and delete them
 +banner ALL OK
 +</file>
 +
 +===== shell function =====
 +<code bash>
 +#!/bin/bash -ueE
 +set -o pipefail
 +trap "banner error; echo LINE: $LINENO" ERR
 +
 +run_duply() {
 +    echo "====================================================="
 +    duply ${1} backup
 +    echo "====================================================="
 +    duply ${1} cleanup --extra-clean --force
 +    duply ${1} purge --force
 +    duply ${1} purgeIncr --force
 +    duply ${1} purgeFull --force
 +    echo "====================================================="
 +    duply ${1} cleanup --extra-clean --force > /dev/null
 +    echo "====================================================="    
 +    banner ${1} OK
 +}
 </code> </code>
  
Line 99: Line 171:
  
 ====== Issues ====== ====== Issues ======
 +
 +==== no acceptable kex algorithm ====
 ssh: Exception: Incompatible ssh peer (no acceptable kex algorithm) ssh: Exception: Incompatible ssh peer (no acceptable kex algorithm)
  
Line 116: Line 190:
 </code> </code>
  
 +====  can't be deleted ====
 +<code bash> duply mybackup purge --force</code>
 +<code>
 +Last full backup date: Wed May 24 01:11:54 2017
 +There are backup set(s) at time(s):
 +Thu Nov 24 01:05:26 2016
 +Fri Nov 25 01:09:43 2016
 +Sat Nov 26 01:10:50 2016
 +Which can't be deleted because newer sets depend on them.
 +No old backup sets found, nothing deleted.
 +</code>
  
 +Solution is to run:
 +<code bash> duply mybackup purgeIncr --force</code>
 +<code bash> duply mybackup purgeFull --force</code>