[bareos-users] Multithreading in bareos-fd compression

Post by d***@i2tic.com
Is there any compression method with Multithreading support?.
I want to backup an SQL file of +300GB, and the problem is that GZIP and
LZ4HC at least are single threaded.

Daniel Carrasco

2018-09-21 17:01:59 UTC

Thanks for your response.

Yeah, that's what I thought for the SQL backup file on that machine, but
I've a MsSQL data file that is backup using VSS because is in use, and that
option is not valid.

I also think that don't use the full power is good to don't slowdown the
other services, but also having an option to select how many cores to use
is a good idea. For example, this server is the most of time at less than
50% of CPU, and at night is just waiting on less than 5%, so you can use
the 40% at day and 90% at night without problem and will work at x10 and
x22 the single core speed.

Greetings.

Post by d***@i2tic.com
Is there any compression method with Multithreading support?.
I want to backup an SQL file of +300GB, and the problem is that GZIP and
LZ4HC at least are single threaded.

You can hack it up by having a ClientRunBeforeJob script that uses pbzip2 (or
another tool) to compress the file before the backup. You may want a
ClientRunAfterJob script that then deletes the file after the backup.
We do something kinda like this, and we built a specific backup job just for
the thing we needed to backup. So we have the general backup for that system
that then specifically ignores the directory, and then another job comes along
and does just that one directory.
Checkout the Catalog backup in the Bareos docs, it has a lot of the same features.
I personally like that the Bareos FD only uses one core to do compression.
Yes, it is slower, but it controls the impact that backups have on systems.
--
You received this message because you are subscribed to a topic in the
Google Groups "bareos-users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/bareos-users/ihBUmKx5eAE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

Eric Browning

2018-09-21 18:51:31 UTC

Why not use a task to backup the database to the local drive first then the
script to compress it before backing it up with bareos and then deleting
the backup on the local drive.

Post by Daniel Carrasco
Thanks for your response.
Yeah, that's what I thought for the SQL backup file on that machine, but
I've a MsSQL data file that is backup using VSS because is in use, and that
option is not valid.
I also think that don't use the full power is good to don't slowdown the
other services, but also having an option to select how many cores to use
is a good idea. For example, this server is the most of time at less than
50% of CPU, and at night is just waiting on less than 5%, so you can use
the 40% at day and 90% at night without problem and will work at x10 and
x22 the single core speed.
Greetings.

Post by d***@i2tic.com
Is there any compression method with Multithreading support?.
I want to backup an SQL file of +300GB, and the problem is that GZIP

and

Post by d***@i2tic.com
LZ4HC at least are single threaded.

You can hack it up by having a ClientRunBeforeJob script that uses pbzip2 (or
another tool) to compress the file before the backup. You may want a
ClientRunAfterJob script that then deletes the file after the backup.
We do something kinda like this, and we built a specific backup job just for
the thing we needed to backup. So we have the general backup for that system
that then specifically ignores the directory, and then another job comes along
and does just that one directory.
Checkout the Catalog backup in the Bareos docs, it has a lot of the same features.
I personally like that the Bareos FD only uses one core to do
compression.
Yes, it is slower, but it controls the impact that backups have on systems.
--
You received this message because you are subscribed to a topic in the
Google Groups "bareos-users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/bareos-users/ihBUmKx5eAE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
Eric Browning
Systems Administrator
801-984-7623

Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary

Twitter: @SCCMrB

Daniel Carrasco

2018-09-22 07:01:47 UTC

Hello,

Thanks!!, but I've already have it, and what I thought is exactly this,
compress with pigz before the backup and copy the file to several sites
(bareos, other server...).
My problem comes because I want to also backup the raw file in bareos to be
able to recover the file directly without import a dump, because the dump
takes too much to be recovered, and raw file less than 30m.

Greetings!

El vie., 21 sept. 2018 20:51, Eric Browning <

Post by Eric Browning
Why not use a task to backup the database to the local drive first then
the script to compress it before backing it up with bareos and then
deleting the backup on the local drive.

Post by d***@i2tic.com
Is there any compression method with Multithreading support?.
I want to backup an SQL file of +300GB, and the problem is that GZIP

and

Post by d***@i2tic.com
LZ4HC at least are single threaded.

--
Eric Browning
Systems Administrator
801-984-7623
Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary

Bruno Friedmann

2018-09-22 13:26:22 UTC

There's perhaps another tricks that could help but need testing
especially the restore and how mssql run afterward.

Create a fileset only for your database and use the sparse = yes attribute
Because most of those db files are just fill up with empty space.

check and see how much real space is used (seems 30m) with sparse activated
you will have to compress (perhaps even not) a very small part of those
300Go file

Work well with almost empty file for virtual machine, so could be also
a way with db files.

Once again, test, test, test before using it for production.

Post by d***@i2tic.com
Hello,
Thanks!!, but I've already have it, and what I thought is exactly this,
compress with pigz before the backup and copy the file to several sites
(bareos, other server...).
My problem comes because I want to also backup the raw file in bareos to be
able to recover the file directly without import a dump, because the dump
takes too much to be recovered, and raw file less than 30m.
Greetings!
El vie., 21 sept. 2018 20:51, Eric Browning <

Post by d***@i2tic.com
Is there any compression method with Multithreading support?.
I want to backup an SQL file of +300GB, and the problem is that GZIP

and

Post by d***@i2tic.com
LZ4HC at least are single threaded.

--
Eric Browning
Systems Administrator
801-984-7623
Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary

--
Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch
Bareos Partner, openSUSE Member, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

openSUSE Tumbleweed
Linux 4.16.9-1-default x86_64 GNU/Linux, nvidia: 390.59
Qt: 5.10.0, KDE Frameworks: 5.46.0, Plasma: 5.12.5, kmail2 5.8.1

Daniel Carrasco

2018-09-24 07:50:24 UTC

Finally I've tried LZ4 and the size is the double, but at least it takes
only 35m to be completed and compression ratio is 80%.

I'll try to do some tests to see if backup is right restoring to a test
server.

Thanks, and greetings!.

El sÃ¡b., 22 sept. 2018 a las 15:26, Bruno Friedmann (<

Post by Bruno Friedmann
There's perhaps another tricks that could help but need testing
especially the restore and how mssql run afterward.
Create a fileset only for your database and use the sparse = yes attribute
Because most of those db files are just fill up with empty space.
check and see how much real space is used (seems 30m) with sparse activated
you will have to compress (perhaps even not) a very small part of those
300Go file
Work well with almost empty file for virtual machine, so could be also
a way with db files.
Once again, test, test, test before using it for production.

Post by d***@i2tic.com
able to recover the file directly without import a dump, because the dump
takes too much to be recovered, and raw file less than 30m.
Greetings!
El vie., 21 sept. 2018 20:51, Eric Browning <

Post by Daniel Carrasco
Thanks for your response.
Yeah, that's what I thought for the SQL backup file on that machine,

but

Post by Daniel Carrasco
I've a MsSQL data file that is backup using VSS because is in use, and that
option is not valid.
I also think that don't use the full power is good to don't slowdown

the

Post by Daniel Carrasco
other services, but also having an option to select how many cores to

use

Post by Daniel Carrasco
is a good idea. For example, this server is the most of time at less

than

Post by Daniel Carrasco
50% of CPU, and at night is just waiting on less than 5%, so you can

use

Post by Daniel Carrasco
the 40% at day and 90% at night without problem and will work at x10

and

Post by Daniel Carrasco
x22 the single core speed.
Greetings.

Post by d***@i2tic.com
Is there any compression method with Multithreading support?.
I want to backup an SQL file of +300GB, and the problem is that

GZIP

Post by Douglas K. Rand
and

Post by d***@i2tic.com
LZ4HC at least are single threaded.

just

Post by Douglas K. Rand
for
the thing we needed to backup. So we have the general backup for that system
that then specifically ignores the directory, and then another job

comes

Post by Douglas K. Rand
along
and does just that one directory.
Checkout the Catalog backup in the Bareos docs, it has a lot of the

same

Post by Douglas K. Rand
features.
I personally like that the Bareos FD only uses one core to do compression.
Yes, it is slower, but it controls the impact that backups have on systems.
--
You received this message because you are subscribed to a topic in

the

Post by Douglas K. Rand
Google Groups "bareos-users" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/bareos-users/ihBUmKx5eAE/unsubscribe.

Post by Douglas K. Rand
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google

Groups

Post by Daniel Carrasco
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it,

send an