meta data for this page
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
linux:backup:duply [2016/09/14 10:50] – niziak | linux:backup:duply [2021/05/10 11:57] (current) – niziak | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Duply ====== | ||
+ | |||
+ | ====== Installation ====== | ||
<code bash> | <code bash> | ||
sudo apt-get install duply python-paramiko trickle | sudo apt-get install duply python-paramiko trickle | ||
</ | </ | ||
+ | ====== Configuration ====== | ||
Create backup profile: | Create backup profile: | ||
<code bash> | <code bash> | ||
Profile file '' | Profile file '' | ||
+ | |||
+ | IMPORTANT | ||
+ | Copy the whole profile folder after the first backup to a safe place. | ||
+ | (e.g. after a system crash). Keep access to these files restricted as they contain all information (gpg data, ftp data) to access and modify your backups. | ||
+ | |||
+ | | ||
+ | |||
Generate random password: | Generate random password: | ||
<code bash> | <code bash> | ||
- | <file | ~/ | + | < |
#GPG_KEY= | #GPG_KEY= | ||
GPG_PW='< | GPG_PW='< | ||
Line 16: | Line 27: | ||
Configure backup section: | Configure backup section: | ||
- | <file | ~/ | + | < |
# Paramiko SSH is very CPU consuming | # Paramiko SSH is very CPU consuming | ||
# | # | ||
Line 27: | Line 38: | ||
SOURCE='/' | SOURCE='/' | ||
MAX_AGE=6M | MAX_AGE=6M | ||
+ | MAX_FULL_BACKUPS=2 | ||
+ | MAX_FULLS_WITH_INCRS=2 | ||
+ | |||
+ | MAX_FULLBKP_AGE=3M | ||
+ | DUPL_PARAMS=" | ||
+ | |||
VOLSIZE=256 | VOLSIZE=256 | ||
DUPL_PARAMS=" | DUPL_PARAMS=" | ||
+ | |||
+ | VERBOSITY=4 | ||
+ | TEMP_DIR=/ | ||
# Specify different id_rsa file: | # Specify different id_rsa file: | ||
DUPL_PARAMS=" | DUPL_PARAMS=" | ||
- | DUPL_PARAMS=" | + | DUPL_PARAMS=" |
+ | DUPL_PARAMS=" | ||
+ | DUPL_PARAMS=" | ||
+ | DUPL_PARAMS=" | ||
</ | </ | ||
Line 44: | Line 67: | ||
</ | </ | ||
+ | **Note:** duply dosn't make any cleanup or deletions during '' | ||
+ | To perform maintenance of old backup accordign to '' | ||
+ | duply with '' | ||
+ | |||
+ | Example options: | ||
+ | * MAX_FULL_BACKUPS=2 | ||
+ | * MAX_FULLS_WITH_INCRS=1 | ||
+ | Will keep 2 full backup sets, but only one with increments (last one). | ||
+ | |||
+ | Sometimes it is good to check whether incremental backups are meaningful (it depends on type of data stored). If command | ||
+ | <code bash> | ||
+ | can be short e.g. MAX_FULLBKP_AGE=7D | ||
+ | |||
+ | |||
+ | ====== Usage ====== | ||
Start the backup | Start the backup | ||
<code bash> | <code bash> | ||
Line 51: | Line 89: | ||
duply gitlab verify | duply gitlab verify | ||
+ | </ | ||
+ | |||
+ | ===== cron script ===== | ||
+ | |||
+ | <file bash> | ||
+ | #!/bin/bash -ue | ||
+ | set -o pipefail | ||
+ | trap " | ||
+ | |||
+ | duply gitlab backup | ||
+ | duply gitlab purge --force # list outdated backup archives and delete them | ||
+ | duply gitlab-to-grinnux purgeIncr --force | ||
+ | duply gitlab-to-grinnux purgeFull --force | ||
+ | duply gitlab cleanup --extra-clean --force > /dev/null # list broken backup files and delete them | ||
+ | banner ALL OK | ||
+ | </ | ||
+ | |||
+ | ===== shell function ===== | ||
+ | <code bash> | ||
+ | #!/bin/bash -ueE | ||
+ | set -o pipefail | ||
+ | trap " | ||
+ | |||
+ | run_duply() { | ||
+ | echo " | ||
+ | duply ${1} backup | ||
+ | echo " | ||
+ | duply ${1} cleanup --extra-clean --force | ||
+ | duply ${1} purge --force | ||
+ | duply ${1} purgeIncr --force | ||
+ | duply ${1} purgeFull --force | ||
+ | echo " | ||
+ | duply ${1} cleanup --extra-clean --force > /dev/null | ||
+ | echo " | ||
+ | banner ${1} OK | ||
+ | } | ||
</ | </ | ||
Line 62: | Line 136: | ||
</ | </ | ||
+ | ====== Verbosity ====== | ||
+ | |||
+ | ==== VERBOSITY=5 ==== | ||
+ | Lots of info | ||
+ | < | ||
+ | Ignoring incremental Backupset (start_time: | ||
+ | Ignoring incremental Backupset (start_time: | ||
+ | Added incremental Backupset (start_time: | ||
+ | Ignoring incremental Backupset (start_time: | ||
+ | </ | ||
+ | |||
+ | ==== VERBOSITY=4 ==== | ||
+ | Usefull report: | ||
+ | < | ||
+ | -------------[ Backup Statistics ]-------------- | ||
+ | StartTime 1479517583.39 (Sat Nov 19 01:06:23 2016) | ||
+ | EndTime 1479517751.96 (Sat Nov 19 01:09:11 2016) | ||
+ | ElapsedTime 168.57 (2 minutes 48.57 seconds) | ||
+ | SourceFiles 41 | ||
+ | SourceFileSize 13621422991 (12.7 GB) | ||
+ | NewFiles 1 | ||
+ | NewFileSize 4096 (4.00 KB) | ||
+ | DeletedFiles 0 | ||
+ | ChangedFiles 1 | ||
+ | ChangedFileSize 13621360640 (12.7 GB) | ||
+ | ChangedDeltaSize 0 (0 bytes) | ||
+ | DeltaEntries 2 | ||
+ | RawDeltaSize 6101758 (5.82 MB) | ||
+ | TotalDestinationSizeChange 5214178 (4.97 MB) | ||
+ | Errors 0 | ||
+ | ------------------------------------------------- | ||
+ | </ | ||
+ | |||
+ | |||
+ | ====== Issues ====== | ||
+ | |||
+ | ==== no acceptable kex algorithm ==== | ||
+ | ssh: Exception: Incompatible ssh peer (no acceptable kex algorithm) | ||
+ | |||
+ | Python paramiko module needs upgrade | ||
+ | |||
+ | <code bash> | ||
+ | apt-get install python-pip python-dev python-cffi libffi-dev build-essential | ||
+ | pip install --ugprade cffi | ||
+ | pip install pycparser==2.13 | ||
+ | pip install --ugprade cryptography | ||
+ | </ | ||
+ | To solve error " | ||
+ | please install <code bash>pip install pycparser==2.13</ | ||
+ | |||
+ | <code bash> | ||
+ | pip install --ugprade paramiko | ||
+ | </ | ||
+ | |||
+ | ==== can't be deleted ==== | ||
+ | <code bash> duply mybackup purge --force</ | ||
+ | < | ||
+ | Last full backup date: Wed May 24 01:11:54 2017 | ||
+ | There are backup set(s) at time(s): | ||
+ | Thu Nov 24 01:05:26 2016 | ||
+ | Fri Nov 25 01:09:43 2016 | ||
+ | Sat Nov 26 01:10:50 2016 | ||
+ | Which can't be deleted because newer sets depend on them. | ||
+ | No old backup sets found, nothing deleted. | ||
+ | </ | ||
+ | |||
+ | Solution is to run: | ||
+ | <code bash> duply mybackup purgeIncr --force</ | ||
+ | <code bash> duply mybackup purgeFull --force</ | ||