meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
linux:backup:duply [2016/05/04 12:31] – niziaklinux:backup:duply [2021/05/10 13:53] – niziak
Line 1: Line 1:
 +====== Duply ======
 +
 +====== Installation ======
 <code bash> <code bash>
-sudo apt-get install duply python-paramiko+sudo apt-get install duply python-paramiko trickle
 </code> </code>
  
 +====== Configuration ======
 Create backup profile: Create backup profile:
 <code bash>duply gitlab create</code> <code bash>duply gitlab create</code>
Line 10: Line 14:
 <code bash>openssl rand -base64 20</code> <code bash>openssl rand -base64 20</code>
  
-<file | ~/.duply/gitlab/conf>+<file bash | ~/.duply/gitlab/conf>
 #GPG_KEY= #GPG_KEY=
 GPG_PW='<generated passsword>' GPG_PW='<generated passsword>'
Line 16: Line 20:
  
 Configure backup section: Configure backup section:
-<file | ~/.duply/gitlab/conf> +<file bash | ~/.duply/gitlab/conf> 
-TARGET='scp://gitlabbackup@192.168.0.230//mnt/backup/gitlabbackup'+# Paramiko SSH is very CPU consuming 
 +#TARGET='scp://gitlabbackup@192.168.0.230//mnt/backup/gitlabbackup' 
 + 
 +TARGET='sftp://gitlabbackup@192.168.0.230//mnt/backup/gitlabbackup' 
 + 
 +# Limit network speed 
 +DUPL_PRECMD="trickle -s -u 1500 -d 256" 
 SOURCE='/' SOURCE='/'
 MAX_AGE=6M MAX_AGE=6M
 +MAX_FULL_BACKUPS=2
 +MAX_FULLS_WITH_INCRS=2
 +
 +MAX_FULLBKP_AGE=3M
 +DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
 +
  
 VOLSIZE=256 VOLSIZE=256
 DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE " DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
 +
 +VERBOSITY=4
 +TEMP_DIR=/tmp
  
 # Specify different id_rsa file: # Specify different id_rsa file:
 DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oIdentityFile='/root/.duply/gitlab/id_rsa' " DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oIdentityFile='/root/.duply/gitlab/id_rsa' "
 +DUPL_PARAMS="$DUPL_PARAMS --no-compression " # Do not use GZip to compress files on remote system.
 +DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-carcfour128 "  # light cipher for NAS
 +DUPL_PARAMS="$DUPL_PARAMS --ssh-options=-oCompression=no "  # disable ssh compression
 +DUPL_PARAMS="$DUPL_PARAMS --asynchronous-upload " # uploads in background
 </file> </file>
  
Line 36: Line 60:
 </file> </file>
  
 +**Note:** duply dosn't make any cleanup or deletions during ''backup'' action. So destination storage can be full very quickly.
 +To perform maintenance of old backup accordign to ''MAX_AGE'', ''MAX_FULL_BACKUPS'', ''MAX_FULLS_WITH_INCRS'' and ''MAX_FULLBKP_AGE'' parameters it is need to call
 +duply with ''purge'' and ''cleanup'' commands. See cron script example below.
 +
 +Example options:
 +  * MAX_FULL_BACKUPS=2
 +  * MAX_FULLS_WITH_INCRS=1
 +Will keep 2 full backup sets, but only one with increments (last one).
 +
 +Sometimes it is good to check whether incremental backups are meaningful (it depends on type of data stored). If command
 +<code bash>duply gitlab status</code> shows that number of volume with each increment is similar to full backup, then making increments has no sense, ane time to keep increments
 +can be short e.g. MAX_FULLBKP_AGE=7D
 +
 +
 +====== Usage ======
 Start the backup Start the backup
 <code bash>sudo duply gitlab backup --progress</code> <code bash>sudo duply gitlab backup --progress</code>
Line 44: Line 83:
 duply gitlab verify  # long operation duply gitlab verify  # long operation
 </code> </code>
 +
 +===== cron script =====
 +
 +<file bash>
 +#!/bin/bash -ue
 +set -o pipefail
 +trap "banner error; echo LINE: $LINENO" ERR
 +
 +duply gitlab backup
 +duply gitlab purge --force # list outdated backup archives and delete them
 +duply gitlab-to-grinnux purgeIncr --force
 +duply gitlab-to-grinnux purgeFull --force
 +duply gitlab cleanup --extra-clean --force > /dev/null # list broken backup files and delete them
 +banner ALL OK
 +</file>
 +
 +===== shell function =====
 +<code bash>
 +#!/bin/bash -ueE
 +set -o pipefail
 +trap "banner error; echo LINE: $LINENO" ERR
 +
 +run_duply() {
 +    echo "====================================================="
 +    duply ${1} backup
 +    echo "====================================================="
 +    duply ${1} cleanup --extra-clean --force
 +    duply ${1} purge --force
 +    duply ${1} purgeIncr --force
 +    duply ${1} purgeFull --force
 +    echo "====================================================="
 +    duply ${1} cleanup --extra-clean --force > /dev/null
 +    echo "====================================================="    
 +    banner ${1} OK
 +}
 +</code>
 +
 +====== SFTP and rbash ======
 +<file | /etc/passwd>
 +mybackup:x:1002:1002:Backup,,,:/home/mybackup:/bin/rbash
 +</file>
 +
 +<file | /etc/ssh/sshd_config>
 +Subsystem sftp internal-sftp
 +</file>
 +
 +====== Verbosity ======
 +
 +==== VERBOSITY=5 ====
 +Lots of info
 +<code>
 +Ignoring incremental Backupset (start_time: Fri Aug  5 13:49:11 2016; needed: Fri Jun 17 14:20:07 2016)
 +Ignoring incremental Backupset (start_time: Fri Aug  5 13:49:11 2016; needed: Thu Jul 21 14:33:52 2016)
 +Added incremental Backupset (start_time: Fri Aug  5 13:49:11 2016 / end_time: Wed Aug 10 14:54:49 2016)
 +Ignoring incremental Backupset (start_time: Wed Aug 10 14:54:49 2016; needed: Fri Jun 17 14:20:07 2016)
 +</code>
 +
 +==== VERBOSITY=4 ====
 +Usefull report:
 +<code>
 +-------------[ Backup Statistics ]--------------
 +StartTime 1479517583.39 (Sat Nov 19 01:06:23 2016)
 +EndTime 1479517751.96 (Sat Nov 19 01:09:11 2016)
 +ElapsedTime 168.57 (2 minutes 48.57 seconds)
 +SourceFiles 41
 +SourceFileSize 13621422991 (12.7 GB)
 +NewFiles 1
 +NewFileSize 4096 (4.00 KB)
 +DeletedFiles 0
 +ChangedFiles 1
 +ChangedFileSize 13621360640 (12.7 GB)
 +ChangedDeltaSize 0 (0 bytes)
 +DeltaEntries 2
 +RawDeltaSize 6101758 (5.82 MB)
 +TotalDestinationSizeChange 5214178 (4.97 MB)
 +Errors 0
 +-------------------------------------------------
 +</code>
 +
 +
 +====== Issues ======
 +
 +==== no acceptable kex algorithm ====
 +ssh: Exception: Incompatible ssh peer (no acceptable kex algorithm)
 +
 +Python paramiko module needs upgrade
 +
 +<code bash>
 +apt-get install python-pip python-dev python-cffi libffi-dev build-essential
 +pip install --ugprade cffi
 +pip install pycparser==2.13
 +pip install --ugprade cryptography
 +</code>
 +To solve error "AssertionError: sorry, but this version only supports 100 named groups"
 +please install <code bash>pip install pycparser==2.13</code>
 +
 +<code bash>
 +pip install --ugprade paramiko
 +</code>
 +
 +====  can't be deleted ====
 +<code bash> duply mybackup purge --force</code>
 +<code>
 +Last full backup date: Wed May 24 01:11:54 2017
 +There are backup set(s) at time(s):
 +Thu Nov 24 01:05:26 2016
 +Fri Nov 25 01:09:43 2016
 +Sat Nov 26 01:10:50 2016
 +Which can't be deleted because newer sets depend on them.
 +No old backup sets found, nothing deleted.
 +</code>
 +
 +Solution is to run:
 +<code bash> duply mybackup purgeIncr --force</code>
 +<code bash> duply mybackup purgeFull --force</code>