Duplicity and OVH's Object Storage

Keeping backups is an absolute must, and after spotting cheap object storage from OVH.com available for 0.01€/GB/Month providing great connectivity, availability and 3x replication of data, I just had to store my backups there.

The tool to use is Duplicity, it makes incremental (only uploading files that have been added, modified) and encrypted (using gpg or a symetrical key) backups of the folders of your choice and can store them almost anywhere (local storage, FTP, SSH, WebDav, rsync, Amazon S3 and most importantly OpenSwift object storage).

Step 1: Install Duplicity

I strongly recommend grabbing the latest version of Duplicity from the developer's PPA, the version found in Debian's repos is slightly old (v0.6.18) and I experienced trouble using it with Swift.

First off add the repository:
sudo add-apt-repository ppa:duplicity-team/ppa

Now update package lists and install:
sudo apt-get update && sudo apt-get install duplicity

A quick duplicity --version gives duplicity 0.7.05, we're up to date !

Step 2: Install all the Swift magic

Install python-pip:
sudo apt-get install python-pip

Then install the swiftclient and keystoneclient pip modules:
sudo pip install python-swiftclient python-keystoneclient

Step 3: The backup script

vi backup.sh

#!/bin/bash

src="/home/cuonic"
dest="swift://containername"
export PASSPHRASE="MyT0PS3CR3TP@$$PHR@$3"

export SWIFT_USERNAME="username"
export SWIFT_PASSWORD="password"
export SWIFT_AUTHURL="https://auth.cloud.ovh.net/v2.0/"
export SWIFT_AUTHVERSION="2"
export SWIFT_TENANTNAME="tenantname"
export SWIFT_REGIONNAME="GRA1"

/usr/bin/duplicity --verbosity notice --progress --asynchronous-upload --cf-backend swift --volsize 100 "$src" "$dest"

unset PASSPHRASE SWIFT_USERNAME SWIFT_PASSWORD SWIFT_AUTHURL SWIFT_AUTHVERSION SWIFT_TENANTNAME  SWIFT_REGIONNAME

Explications:

  • src is the directory you want to backup (in this case it's my entire home directory)
  • dest is where to backup to, you need to replace containername with the name of your Object storage container.
  • PASSPHRASE is the symetric encryption key that will be used to encrypt your backups, you can also use GPG keys but that isn't covered here.
  • You obviously need to replace SWIFT_USERNAME and SWIFT_PASSWORD with your own credentials obtained via the OVH Cloud Control Panel.
  • SWIFT_TENANTNAME can be found in the OVH Horizon Control Panel as the projet ID displayed at the top of the page

TenantName

  • SWIFT_REGIONNAME is where you want your data to be stored geographically (3 options with OVH: GRA1, BHS1, SBG1)

Now save the file with :x

Change the permission of the file so that only you can read it's contents and execute it:

chmod 700 backup.sh

Step 4: Backup

Execute the script: ./backup.sh

Be patient, depending on the overall file size, your connection speed and other factors this could take ages, and if you're on an ADSL connection you won't be able to use the internet much, so watch a few films and wait for the following type of output:

--------------[ Backup Statistics ]--------------
StartTime 1447941101.02 (Thu Nov 19 14:51:41 2015)
EndTime 1447941795.38 (Thu Nov 19 15:03:15 2015)
ElapsedTime 694.36 (11 minutes 34.36 seconds)
SourceFiles 29747
SourceFileSize 3237726252 (3.02 GB)
NewFiles 29747
NewFileSize 3237726252 (3.02 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 29747
RawDeltaSize 3218821765 (3.00 GB)
TotalDestinationSizeChange 2649932101 (2.47 GB)
Errors 0
-------------------------------------------------

And you're done ! You can either execute backup.sh manually whenever you feel the need to backup or add it as a cronjob to backup daily, weekly or monthly.