[bareos-users] Bareos Wild and Regex

[Dakota Pilot]

I've used wildcards and regex for years and thought I understood them until I encountered Bareos version of wildcards/regex!! Any help is appreciated since I've been banging my head on this wall for some time.

I want to match a directory named cache2 under C:/users/ for all users and exclude it no matter how deep in the hierarchy it is. This is on a Windows system so I tried these expressions for WildDir

*/cache2
[A-Z]:/*/cache2

Both work fine in bwild if given a file like this.

c:/tmp/
c:/xyz/tmp
d:/tmp
d:/x/y/z/tmp
c:/users/brett/appdata
/home
/home/lost+found
/home/brett
C:/cache2
D:/somedir/cache2
E:/somedir/anotherdir/cache2

However, when I put this into a WildDir in a fileset and run "estimate job=jobname listing the directory is not excluded unless it is in the top level, that is c:/Users/username/cache2.

I also tried RegExDir which as I understand it is supposed to match the regex in a filename which may contain the full path. If I use this regex in bregex it works and finds the directories with cache2 in the name.

[A-Z]:/.*/cache2

However, when added to a fileset it does not work. I'm including the fileset conf file so what is wrong. Given that bwild and bregex both say the expression should work it must be something in the file set. As shown in the fileset I have this same thing working for other directories but I had to create expressions which first define the directory path and then one to not backup the files. There has to be a better way to do it and hopefully with wildcards/regex I can simply say if this match is found exclude it.

Which is better to use - RegEx or wildcards. I read somewhere in the docs that RegEx can get expensive to what are the guidelines for using it.

Here is the fileset.

BOF bareos-dir.d/fileset/WinWS.conf

FileSet {
# File set for Windows Workstations. Excludes all directories and
# then backups up only the desired ones.

Name = "WinWS"

Enable VSS = yes

# Included options
Include {
# Backup all users
File = "C:/Users"

# Other files.
File = "S:/Programs/Steam/userdata"
File = "S:/bin"
File = "S:/Games/XPlane-11"
File = "C:/ProgramData"
File = "S:/Projects"

Options {
Signature = MD5
AutoExclude = yes
Drive Type = fixed
IgnoreCase = yes
Compression = GZIP

# Do selected folders and files for users.
# Documents folder.

RegExDir = "[A-Z]:/Users/"
Wild = "[A-Z]:/Users/*"

# Steam files
RegExDir = "[A-Z]:/Programs/Steam/userdata"
Wild = "[A-Z]:/Programs/Steam/userdata/*"

# Other files
RegExDir = "[A-Z]:/bin"
Wild = "[A-Z]:/bin/*"

# X-Plane
WildDir = "[A-Z]:/Games/XPlane-11/Aircraft"
Wild = "[A-Z]:/Games/XPlane-11/Aircraft/*"
WildDir = "[A-Z]:/Games/XPlane-11/Custom*"
Wild = "[A-Z]:/Games/XPlane-11/Custom*/*"
WildDir = "[A-Z]:/Games/XPlane-11/Output"
WildDir = "[A-Z]:/Games/XPlane-11/Output/preferences"
Wild = "[A-Z]:/Games/XPlane-11/Output/preferences/*"
WildDir = "[A-Z]:/Games/XPlane-11/Resources"
WildDir = "[A-Z]:/Games/XPlane-11/Resources/plugins"
Wild = "[A-Z]:/Games/XPlane-11/Resources/plugins/*"

# ProgramData Folder
WildDir = "[A-Z]:/ProgramData/Bareos"
Wild = "[A-Z]:/ProgramData/Bareos/*"

# Projects Directory
WildDir = "[A-Z]:/Projects"
Wild = "[A-Z]:/Projects/*"
}


# Excluded
Options {
Exclude = yes
# Exclude system files.
WildFile = "[A-Z]:/pagefile.sys"
WildDir = "[A-Z]:/RECYCLER"
WildDir = "[A-Z]:/$RECYCLE.BIN"
WildDir = "[A-Z]:/System Volume Information"

# Exclude other items
RegExDir = "[A-Z]:/Games/XPlane-11"
Wild = "[A-Z]:/Games/XPlane-11/*"
RegExDir = "[A-Z]:/ProgramData"
Wild = "[A-Z]:/ProgramData/*"
RegExDir = "[A-Z]:/Projects"
WildDir = "[A-Z[:/*/cache2"
}
}
}


EOF

[Dakota Pilot]

There is a typo in the last WildDir line where there is a [ instead of ] but that is fixed in the file and makes no difference.

[r***@gmail.com]

hello!
I do not know if this will help.
but I use this configuration.

I use 2 different include options.
1st enables compression and exclude compressed formats.
2nd disabled compressing, backups only compressed formats.

I've spent a lot of time to figure out trick to speed up backing up files.

Post by Dakota Pilot
I've used wildcards and regex for years and thought I understood them until I encountered Bareos version of wildcards/regex!! Any help is appreciated since I've been banging my head on this wall for some time.
I want to match a directory named cache2 under C:/users/ for all users and exclude it no matter how deep in the hierarchy it is. This is on a Windows system so I tried these expressions for WildDir
*/cache2
[A-Z]:/*/cache2
Both work fine in bwild if given a file like this.
c:/tmp/
c:/xyz/tmp
d:/tmp
d:/x/y/z/tmp
c:/users/brett/appdata
/home
/home/lost+found
/home/brett
C:/cache2
D:/somedir/cache2
E:/somedir/anotherdir/cache2
However, when I put this into a WildDir in a fileset and run "estimate job=jobname listing the directory is not excluded unless it is in the top level, that is c:/Users/username/cache2.
I also tried RegExDir which as I understand it is supposed to match the regex in a filename which may contain the full path. If I use this regex in bregex it works and finds the directories with cache2 in the name.
[A-Z]:/.*/cache2
However, when added to a fileset it does not work. I'm including the fileset conf file so what is wrong. Given that bwild and bregex both say the expression should work it must be something in the file set. As shown in the fileset I have this same thing working for other directories but I had to create expressions which first define the directory path and then one to not backup the files. There has to be a better way to do it and hopefully with wildcards/regex I can simply say if this match is found exclude it.
Which is better to use - RegEx or wildcards. I read somewhere in the docs that RegEx can get expensive to what are the guidelines for using it.
Here is the fileset.
BOF bareos-dir.d/fileset/WinWS.conf
FileSet {
# File set for Windows Workstations. Excludes all directories and
# then backups up only the desired ones.
Name = "WinWS"
Enable VSS = yes
# Included options
Include {
# Backup all users
File = "C:/Users"
# Other files.
File = "S:/Programs/Steam/userdata"
File = "S:/bin"
File = "S:/Games/XPlane-11"
File = "C:/ProgramData"
File = "S:/Projects"
Options {
Signature = MD5
AutoExclude = yes
Drive Type = fixed
IgnoreCase = yes
Compression = GZIP
# Do selected folders and files for users.
# Documents folder.
RegExDir = "[A-Z]:/Users/"
Wild = "[A-Z]:/Users/*"
# Steam files
RegExDir = "[A-Z]:/Programs/Steam/userdata"
Wild = "[A-Z]:/Programs/Steam/userdata/*"
# Other files
RegExDir = "[A-Z]:/bin"
Wild = "[A-Z]:/bin/*"
# X-Plane
WildDir = "[A-Z]:/Games/XPlane-11/Aircraft"
Wild = "[A-Z]:/Games/XPlane-11/Aircraft/*"
WildDir = "[A-Z]:/Games/XPlane-11/Custom*"
Wild = "[A-Z]:/Games/XPlane-11/Custom*/*"
WildDir = "[A-Z]:/Games/XPlane-11/Output"
WildDir = "[A-Z]:/Games/XPlane-11/Output/preferences"
Wild = "[A-Z]:/Games/XPlane-11/Output/preferences/*"
WildDir = "[A-Z]:/Games/XPlane-11/Resources"
WildDir = "[A-Z]:/Games/XPlane-11/Resources/plugins"
Wild = "[A-Z]:/Games/XPlane-11/Resources/plugins/*"
# ProgramData Folder
WildDir = "[A-Z]:/ProgramData/Bareos"
Wild = "[A-Z]:/ProgramData/Bareos/*"
# Projects Directory
WildDir = "[A-Z]:/Projects"
Wild = "[A-Z]:/Projects/*"
}
# Excluded
Options {
Exclude = yes
# Exclude system files.
WildFile = "[A-Z]:/pagefile.sys"
WildDir = "[A-Z]:/RECYCLER"
WildDir = "[A-Z]:/$RECYCLE.BIN"
WildDir = "[A-Z]:/System Volume Information"
# Exclude other items
RegExDir = "[A-Z]:/Games/XPlane-11"
Wild = "[A-Z]:/Games/XPlane-11/*"
RegExDir = "[A-Z]:/ProgramData"
Wild = "[A-Z]:/ProgramData/*"
RegExDir = "[A-Z]:/Projects"
WildDir = "[A-Z[:/*/cache2"
}
}
}
EOF

[Dakota Pilot]

Interesting. I may split it into two includes with one having exclude = yes in it. I think it's Bareos acting wonky since bregex and bwild do the selections perfectly no matter what depth the directory is. So something must be happening when it gets parsed.

I notice you put your Files = after the includes. Any reason since I assume the config engine doesn't care.

[Dakota Pilot]

No luck. If I put a separate include with it's options Bareos still will not exclude the directory with a WildDir or RegExDir. Given that bregex and bwild work just fine I think there is something broken somewhere in my config but I have no idea where. Bareos just won't exclude that directory.

[Dakota Pilot]

I did some more testing and this RegEx works great in bregx on a test file which contains the directory path that I'm matching.

C:/Users/Administrator/AppData/Local/Mozilla/Firefox/Profiles/44beavkz.default/cache2/entries/F8AC72083E334F70A553AE68455FBDF0E65C5221


I want to find /cache2 and if it's in the path exclude it. Bregex works fine and matches the lines in the test file. Bwild works on it's wild equivalent, too.

However, when I put this into the fileset it fails according to estimate job=jobname listing because this path shows up in the listing. I run the command from the command line piping it to bconsole and then to less. I also ran the actual job and still the file is included.

[Dakota Pilot]

Forgot to include the regex and wild expressions.

The regex that works with bregex is

.*/cache2

and with bwild I can do

*/cache2/*

Both match but when put in the Exclude options section - which exclude other directories and files just fine - the directories are still included.