[bareos-users] zfs send/receive and Bareos

[Jan Behrend]

Hello list,

we have quite a few file systems will terrible iNode/Data ratio, i.e.
millions of files in a few terabytes of data.

This obviously takes ages for _any_ file based backup system so scan
for changes files.

Some solutions to this problem could be:

1) Create a list of changed files and feed that to Bareos, so it does
not need to produce this list itself. It would still need to access
the changed files and produce a corresponding I/O load on the system,
but nothing compared to a whole file system scan.

2) Since most of these file systems are on ZFS anyway, one could use
the send/receive feature to lose the need to access single files all
together. Question is: How can I "catch" the ZFS send datastream and
put it into Bareos and its storage, without dumping the stream into a
file first and subsequently have it being picked up by Baroes.

Thanks in advance for any insight!

Cheers, Jan
--
MAX-PLANCK-INSTITUT fuer Radioastronomie
Jan Behrend - Rechenzentrum
----------------------------------------
Auf dem Huegel 69, D-53121 Bonn                                  
Tel: [phone]
http://www.mpifr-bonn.mpg.de

[Damiano Verzulli]

Post by Jan Behrend
Hello list,
we have quite a few file systems will terrible iNode/Data ratio, i.e.
millions of files in a few terabytes of data.
This obviously takes ages for _any_ file based backup system so scan
for changes files.

Hi Jan,

I spent some time, lately, trying to optimize our very-similar-to-yours
backup: 6.2TB of data related to 24M files (actually, the whole maildirs of
our mail server infrastructure).

With following hardware setup:

- SD running on an ad-hoc server and driving a SAS-based auto-changer
and LTO-6 drives;
- FD running on a different server and hosting the whole set of tiny
files;
- the two servers (FD and SD) connected via a 10GEth link

and _A_LOT_ of tuning at various level (from the disk/RAID configuration,
up to file system, up to OS, up to BareOS components)
we've been able to reach a transfer rate of about 95 MB/s over a 18 hours
long timeframe [see details in the attached *Job_181.log* file]

With some additional tuning we've been able to slightly increase the number
(+/- 104 MB/s over a 9 hour long backup - 3.5TB for 13.7M files; a "subset"
of the above - details in the *Job_643.log* attached file)

We even tried to maximize the SAS bandwidth (we have three LTO drives,
connected on dedicated SAS channels) by running concurrent backups: our
maildirs are hosted on three different servers, so we tryied to run three
concurrent jobs towards the three drives. We easily crossed the 160MB/s
with peaks around 180/200 MB/s (the total for the three backups).
(*I guess you got the idea: should you need details, don't hesitate to ask*)

Please note that we're *NOT* using "spool" (we did some experiment and
decided it's not a perfect suite for our needs).

Post by Jan Behrend
1) Create a list of changed files and feed that to Bareos, so it does
not need to produce this list itself. It would still need to access
the changed files and produce a corresponding I/O load on the system,

Our best friend has been :

vm.vfs_cache_pressure = 0

on the FD side. Basically it tell the OS to ALWAYS KEEP in RAM, cached, the
whole set of DENTRYies. Basically saving the OS from the continuosly
directory lookup, retrieving the list of files.
This, obviously, has an impact in terms of RAM but... we don't have problem
with this (it's a "dedicated" FD with 128G of RAM).


Nevertheless....

Post by Jan Behrend
but nothing compared to a whole file system scan.

....we did some test backupping a 400G single file and... this easily
outperformed the maildir ones (we easily had the three drives running at
160MB/s each one, concurrently).



2) Since most of these file systems are on ZFS.....
due to really nice snapshot support, we started with OpenZFS hosting the
FD. Unfortunately, even if with 128GB of RAM and with compression enabled,
we registered _VERY_BAD_ reading performances. Our life changed
significantly switching from ZFS to MDADM (yes! I know they are *not*
comparable. But our first priority was "backup-time", not "snapshot
support") :-)

Currently I'm trying to find the way to "combine" the (reading) speed of
MDADM with the snapshot support of ZFS.... But this is slightly off-topic


As for your last point:

,,,anyway, one could use

Post by Jan Behrend
the send/receive feature to lose the need to access single files all
together. Question is: How can I "catch" the ZFS send datastream and
put it into Bareos and its storage, without dumping the stream into a
file first and subsequently have it being picked up by Baroes.

I'm not an expert about "BareOS plugin", but what you're imaging is
somewhat similar to what the VMware plugin achieve for snapshot/CBT backup
of VMware VMs. So this sounds definitely possible but... I've never heard
of someone being working on it. Sorry!

Hope this help!

Cheers,
DV

P.S.: would you mind sharing some technical details about "yours" numbers,
expecially in terms of speed? It's really hard to get "real numbers" to
compare with. I would really appreciate! Thanks

[Jan Behrend]

Hi Damiano,

thanks for your reply!

Post by Damiano Verzulli
P.S.: would you mind sharing some technical details about "yours"
numbers, expecially in terms of speed? It's really hard to get "real
numbers" to compare with. I would really appreciate! Thanks

Both our setups indeed seem to be similar:

- 10GbE dedicated backup network with 10GbE/1GbE clients
- LTO6 tape library
- Rather old spinning disk storage (~400 MiB/s)

e.g. file system:

* 1.2T data with 2.2M inodes (544 KiB/file)
I've never actually taken the time on this one but it runs 48 hours
straight at least.

* 544G data with 3.8M inodes (145 KiB/file)
* 529G data with 4.2M inodes (127 KiB/file)
This is are home directories ...

* 14T data with 166M inodes (82 KiB/file)
I won't even start to back this one up ...

Either you invest on heavy duty I/O in your backup infrastructure or
you try to get around it all together.

We're using hourly ZFS snapshots for file based backups.
Bareos is only (mostly) needed for desaster recovery, hence my inquiry.

Cheers, Jan
--
Anfragen bitte an it-***@mpifr-bonn.mpg.de
Please send requests to it-***@mpifr-bonn.mpg.de
----------------------------------------
Jan Behrend - Rechenzentrum
----------------------------------------
https://itwiki.mpifr-bonn.mpg.de
Tel: 359