offsite / on-the-fly encrypted backups / gocryptfs

Written by
Walter Doekes
Published on 2020-10-23

Earlier, I wrote about using encfs to do on-the-fly encrypted backups (using encfs). The idea was that you grant ssh+rsync access to a backup system, but that that system does not know what it is backing up. This provides a layer of security between your backup provider and your private data.

That scheme works like this:

there is a remote system doing periodic incremental rsync backups, like a PlanB Backup server;
you grant ssh+rsync access to that system;
but only to a specific path;
on that path, you mount an encrypted view of your filesystem — a.k.a. reverse encryption;
the backup server only sees encrypted files, but it will still benefit from the incremental nature of rsync when creating backups.

The issue came up again, so here’s a little update of the current state of things:

encfs has been brought to life a little after it appeared inactive for a while. Current version in Ubuntu/Focal is 1.9.5, released April 2018.
gocryptfs has joined the competition in 2016. It also has an Ubuntu/Focal package, now at 1.7.1, released October 2019.
It appears those are the only two well-known methods to create reverse encrypted mounts.

Here we’ll explore gocryptfs.

gocryptfs

gocryptfs sports the reverse option we’re looking for. It’s written in golang. It has been audited in 2017, although the audit did not cover the reverse mode specifically (the AES-SIV mode).

Version 1.7 lets you choose from the following encryption ciphers:

$ gocryptfs -speed
AES-GCM-256-OpenSSL    772.22 MB/s
AES-GCM-256-Go        1452.66 MB/s  (selected in auto mode)
AES-SIV-512-Go         199.04 MB/s

For reverse encryption, we’re forced to use the AES-SIV-512-Go cipher (see -aessiv below), described in RFC 5297. Since the reverse encryption doesn’t keep any state, it cannot store IVs (or nonces). The Synthetic Initialization Vector (SIV) block cipher mode produces a deterministic IV. (With a random IV, the encrypted file would keep changing.) This comes with a speed penalty.

Initializing the gocrypt reverse config

Create a configuration directory, password and config file:

# umask 0077

# mkdir /.nobackup

# dd if=/dev/random bs=32 count=1 |
    base64 >/.nobackup/gocryptfs.reverse.rootfs.pass
1+0 records in
1+0 records out
32 bytes copied, 5.0439e-05 s, 634 kB/s

Here we choose to enable -plaintextnames. This means that filenames will be readable. That is less secure, but a lot more convenient when asked to restore a single file.

# gocryptfs -reverse -aessiv -plaintextnames \
    -config /.nobackup/gocryptfs.reverse.rootfs.conf \
    -passfile /.nobackup/gocryptfs.reverse.rootfs.pass \
    -init /.nobackup/  # path is irrelevant

Using config file at custom location /.nobackup/gocryptfs.reverse.rootfs.conf
Choose a password for protecting your files.
passfile: reading from file "/.nobackup/gocryptfs.reverse.rootfs.pass"

Your master key is:

    d9504720-c98856e9-860d56bc-2f40eebd-
    fefb9b2a-aac5cb1d-46f81fa3-abe2aee2

If the gocryptfs.conf file becomes corrupted or you ever forget your password,
there is only one hope for recovery: The master key. Print it to a piece of
paper and store it in a drawer. This message is only printed once.
The gocryptfs-reverse filesystem has been created successfully.
You can now mount it using: gocryptfs -reverse /.nobackup MOUNTPOINT

The gocryptfs -init call outputs a master key and writes a configuration file. You can save the key, but if you keep the /.nobackup/gocryptfs.reverse.rootfs.pass password and the /.nobackup/gocryptfs.reverse.rootfs.conf configuration backed up somewhere, you should be safe too.

The configuration file it has written should look somewhat like this:

# cat /.nobackup/gocryptfs.reverse.rootfs.conf
{
  "Creator": "gocryptfs 1.7.1",
  "EncryptedKey": "GhBnT...",
  "ScryptObject": {
    "Salt": "Oyq5J...",
    "N": 65536,
    "R": 8,
    "P": 1,
    "KeyLen": 32
  },
  "Version": 2,
  "FeatureFlags": [
    "GCMIV128",
    "HKDF",
    "PlaintextNames",
    "AESSIV"
  ]
}

Testing the mount point

Let’s do the initial mount:

# mkdir -p /home/remotebackup/rootfs

Don’t forget the specify -reverse when doing the mount. (It will imply readonly, -ro.)

Here we can exclude some things that should not be backed up, like the /.nobackup/ directory, or the mount point itself (we don’t want recursion from the encrypted filesystem into itself, which caused trouble for us with encfs). Setting the owner using -force_owner is useful when a non-root user will be doing the rsync backups.

# gocryptfs -reverse \
  -config /.nobackup/gocryptfs.reverse.rootfs.conf \
  -passfile /.nobackup/gocryptfs.reverse.rootfs.pass \
  -exclude .nobackup/ \
  -exclude .home/remotebackup/rootfs \
  -force_owner 1000:1000 \
  -fsname gcryptfs-reverse-/
  / /home/remotebackup/rootfs

Using config file at custom location /.nobackup/gocryptfs.reverse.rootfs.conf
passfile: reading from file "/.nobackup/gocryptfs.reverse.rootfs.pass"
Decrypting master key
Filesystem mounted and ready.

Setting up fstab

Of course you want the filesystem to be mounted automatically. Setting up /etc/fstab is a breeze because the gcrypt fuse-mount binary takes its options in a mount-compatible fashion (using -o):

# This should be a single line in fstab. Be sure to join the line after
# the commas without any whitespace.
/ /home/remotebackup/rootfs fuse.gocryptfs
    reverse,config=/.nobackup/gocryptfs.reverse.rootfs.conf,
    passfile=/.nobackup/gocryptfs.reverse.rootfs.pass,
    exclude=.nobackup,exclude=home/remotebackup/rootfs,
    force_owner=1000:1000,fsname=gcryptfs-reverse-/ 0 0

With that line in fstab, it’s a matter of mounting the destination:

# mount /home/remotebackup/rootfs

# mount | grep /remotebackup/
/ on /home/remotebackup/rootfs type fuse.gocryptfs-reverse (ro,nosuid,nodev,relatime,user_id=0,group_id=0,max_read=131072)

# df -h | grep -E '/$|/rootfs$'
/dev/mapper/ubuntu--vg-root  106G   98G  3.2G  97% /
gcryptfs-reverse-/           106G   98G  3.2G  97% /home/remotebackup/rootfs

Nice and easy.

Test that we can decrypt single files

Since we’re using this for backups, chances are we’ll want to decrypt single files at one point. (There was a reason we were using -plaintextnames.)

Is it possible to do so without syncing the entire filesystem?

Yes it is. See this example:

# mkdir /home/remotebackup/test-{encrypted,decrypted}

/home/walter/example.c is a human readable file.

# hd /home/walter/example.c
00000000  69 6e 74 20 66 75 6e 63  28 29 20 7b 0a 20 20 20  |int func() {.   |
...

The version in /home/remotebackup is not, obviously:

# hd /home/remotebackup/rootfs/home/walter/example.c
00000000  00 02 b1 df 38 5c 1b 2e  34 3c 71 e7 e7 02 df 45  |....8\..4<q....E|
...

Copy to the temporary directory, as if we’re restoring a single file:

# cp /home/remotebackup/rootfs/home/walter/example.c \
    /home/remotebackup/test-encrypted/example.c

Forward-encrypt this temporary directory:

# gocryptfs \
  -config /.nobackup/gocryptfs.reverse.rootfs.conf \
  -passfile /.nobackup/gocryptfs.reverse.rootfs.pass \
  /home/remotebackup/test-encrypted/ \
  /home/remotebackup/test-decrypted/

Or, alternately, by using only the stored master-key we saved at the initialization step and the appropriate configuration (in this case -aessiv and -plaintextnames).

# gocryptfs -aessiv -plaintextnames \
    -masterkey=d9504720-c98856e9-860d56bc-2f40eebd-fefb9b2a-aac5cb1d-46f81fa3-abe2aee2 \
    /home/remotebackup/test-encrypted/ \
    /home/remotebackup/test-decrypted/

Using explicit master key.
THE MASTER KEY IS VISIBLE VIA "ps ax" AND MAY BE STORED IN YOUR SHELL HISTORY!
ONLY USE THIS MODE FOR EMERGENCIES
Filesystem mounted and ready.

If the hashes of /home/walter/example.c and /home/remotebackup/test-decrypted/example.c are equal, then it really is that easy:

# md5sum /home/walter/example.c \
    /home/remotebackup/rootfs/home/walter/example.c \
    /home/remotebackup/test-encrypted/example.c \
    /home/remotebackup/test-decrypted/example.c
8b2399d85114f8e5f6ad10239afad345  /home/walter/example.c
d3962161b8fbc819b75325ce3c4c7805  /home/remotebackup/rootfs/home/walter/example.c
d3962161b8fbc819b75325ce3c4c7805  /home/remotebackup/test-encrypted/example.c
8b2399d85114f8e5f6ad10239afad345  /home/remotebackup/test-decrypted/example.c

The hashes match. All is good!

Conclusion

It looks like gocryptfs is a good candidate to support encrypted backups through a reverse encrypted mount. Possibly the recent encfs works equally well, but gocryptfs appears to be more actively maintained at the time of writing. And it works out of the box.

Securing ssh+rsync access for the remotebackup user is beyond the scope of this article. But it shouldn’t be too hard to chroot it to its own home directory, denying it access to any unencrypted files.

Of course, an alternate solution is using a snapshotting filesystem like ZFS with local encryption, and backing up raw (encrypted) snapshots. But that’s definitely something for another day. In the mean time, you can check out planb-zfssync, if you want to go that route.

Disclaimer

As I am no encryption expert, I did not audit the encryption methods in gocryptfs. As always, use common sense, and be sure to read the relevant security bulletins.

offsite / on-the-fly encrypted backups / gocryptfs