[bareos-users] Issues getting fd-plugins/mysql-python/ to work. Can anyone provide a complete client and fileset cf

[Sean Sullivan]

I am trying to monkeypatch the mysql plugin to instead sudo up to postgres and run pg_dump instead. I just did a few things to make it work so far:

1.) cp /usr/lib/bareos/plugins/BareosFd{MySQL,Postgres}SQLclass.py; cp /usr/lib/bareos/plugins/bareos-fd-{mysql,postgresql}.py
2.) sed -i 's/mysql/psql/g' to replace all mentions of mysql in both files.
3.) edit BareosFDPostgresSQLclass.py to replace options and mentions of mysql with psql.
4.) I set the psql and pg_dump commands to have sudo -u postgres in front of them so I wouldn't have to deal with authentication right now. Once I can get the basic plugin working I will try to add authentication options.

Currently the bareos-fd client restarts without issue and I can create a job using the new fileset and plugin:

/etc/bareos/client/myself.conf::
Client {
Name = ss-psql-bareos-test-02-fd
Maximum Concurrent Jobs = 20

# remove comment from "Plugin Directory" to load plugins from specified directory.
# if "Plugin Names" is defined, only the specified plugins will be loaded,
# otherwise all storage plugins (*-fd.so) from the "Plugin Directory".
#
Plugin Directory = /usr/lib/bareos/plugins
# Plugin Names = ""

# if compatible is set to yes, we are compatible with bacula
# if set to no, new bareos features are enabled which is the default
# compatible = yes
}

/etc/bareos/bareos-dir.d/fileset/PostgresPGDump.conf::
FileSet {
Name = "PostgresPGDump"
Description = "Backup all databases via pgdump and then stream to director"
Include {
Options {
compression=GZIP
signature = MD5
}
Plugin = "python:module_path=/usr/lib/bareos/plugins:module_name=bareos-fd-postgresql"
}
}


In bareos I can create a new job and that works. However, when I try to get an estimate for the job, the client seems to go into an infinite loop::

*configure add job name=ss-psql-bareos-test-02-psql-test-job client=ss-psql-bareos-test-02 jobdefs=PostgresPGDump
Created resource config file "/etc/bareos/bareos-dir.d/job/ss-psql-bareos-test-02-psql-test-job.conf":
Job {
Name = ss-psql-bareos-test-02-psql-test-job
Client = ss-psql-bareos-test-02
JobDefs = PostgresPGDump
}


*estimate listing job=ss-psql-bareos-test-02-psql-test-job
Using Catalog "MyCatalog"
Connecting to Client ss-psql-bareos-test-02 at ss-psql-bareos-test-02:9102
python-fd: Starting backup of /_pgsqlbackups_/postgres.sql
-rwx------ 0 root root 26311192 2018-01-24 20:24:44 /_pgsqlbackups_/postgres.sql


From here it will just sit like this for hours. The databases are small (it's just the dellstore2 test database). I tried to enable debugging and tracelog but I don't see anything in /var/log/bareos/. I did find a trace of an old attempt in /var/lib/bareos/ with the following inside:

ss-psql-bareos-test-02-fd (100): python-fd.c:2741-0 python-fd: start_backup called
ss-psql-bareos-test-02-fd (100): python-fd.c:2741-0 python-fd: Dumper: 'sudo -u postgres pg_dump --format=p --blobs -- --clean dellstore'
ss-psql-bareos-test-02-fd (150): attr.c:311-0 -rwx------ 0 root root 24934936 2018-01-24 00:45:19 /_pgsqlbackups_/dellstore.sql
ss-psql-bareos-test-02-fd (150): fd_plugins.c:992-0 startBackup returned type=3, fname=/_pgsqlbackups_/dellstore.sql
ss-psql-bareos-test-02-fd (100): python-fd.c:2741-0 python-fd: end_backup_file() entry point in Python called. Returncode: 1
ss-psql-bareos-test-02-fd (200): runscript.c:149-0 runscript: running all RUNSCRIPT object (ClientAfterJob) JobStatus=R
ss-psql-bareos-test-02-fd (150): fd_plugins.c:399-0 plugin_ctx=7fa2fc02f4d8 JobId=0
ss-psql-bareos-test-02-fd (150): fd_plugins.c:201-0 name=<NULL> len=0 plugin=python-fd.so plen=6
ss-psql-bareos-test-02-fd (100): python-fd.c:2741-0 python-fd: handle_plugin_event called with bEventJobEnd
ss-psql-bareos-test-02-fd (150): fd_plugins.c:201-0 name=<NULL> len=0 plugin=bpipe-fd.so plen=5
ss-psql-bareos-test-02-fd (150): fd_plugins.c:239-0 Event 2 disabled for this plugin.
ss-psql-bareos-test-02-fd (150): fd_plugins.c:1831-0 Free instance fd-plugin_ctx_list=7fa2fc02f4d8 JobId=0

I have since removed the extra -- but now I can't seem to generate another trace and the process just seems to hang::

root 5793 0.0 0.2 371072 13284 ? Ssl 18:48 0:00 /usr/sbin/bareos-fd
root 5837 0.0 0.0 4512 792 ? S 18:49 0:00 \_ /bin/sh -c sudo -u postgres /usr/bin/pg_dump --format=p --blobs --clean postgres
root 5838 0.0 0.0 55560 3856 ? S 18:49 0:00 | \_ sudo -u postgres /usr/bin/pg_dump --format=p --blobs --clean postgres
postgres 5839 0.0 0.1 99468 8252 ? S 18:49 0:00 | \_ /usr/lib/postgresql/9.6/bin/pg_dump --format=p --blobs --clean postgres
root 6481 0.0 0.0 4512 760 ? S 20:24 0:00 \_ /bin/sh -c sudo -u postgres /usr/bin/pg_dump --format=p --blobs --clean postgres
root 6482 0.0 0.0 55560 3940 ? S 20:24 0:00 \_ sudo -u postgres /usr/bin/pg_dump --format=p --blobs --clean postgres
postgres 6483 0.0 0.1 99468 8324 ? S 20:24 0:00 \_ /usr/lib/postgresql/9.6/bin/pg_dump --format=p --blobs --clean postgres


Can someone point me in the right direction in terms of troubleshooting this or what I would need to get the mysql plugin working? Do I need to specify a file like in the example? I don't want to backup all of /etc/ like the example mentions.

Thanks in advance for all of your help!!!

[Sean Sullivan]

So a small update. I must have restarted the bareos-fd client after enabling debug mode as running the setdebug command again seems to create a new one. Nothing is in it outside of the (my guess) init of the bareos-fd instance though.

Good news is it _seems_ to create a backup if you run the job::

Build OS: x86_64-pc-linux-gnu ubuntu Ubuntu 16.04 LTS
JobId: 23
Job: ss-psql-bareos-test-02-psql-test-job.2018-01-24_20.58.28_37
Backup Level: Full (upgraded from Incremental)
Client: "ss-psql-bareos-test-02" 17.2.4 (21Sep17) x86_64-pc-linux-gnu,ubuntu,Ubuntu 16
.04 LTS,xUbuntu_16.04,x86_64
FileSet: "PostgresPGDump" 2018-01-23 23:59:25
Pool: "Full" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Job resource)
Scheduled time: 24-Jan-2018 20:58:21
Start time: 24-Jan-2018 20:58:31
End time: 24-Jan-2018 20:58:33
Elapsed time: 2 secs
Priority: 10
FD Files Written: 2
SD Files Written: 2
FD Bytes Written: 4,605,140 (4.605 MB)
SD Bytes Written: 4,605,527 (4.605 MB)
Rate: 2302.6 KB/s
Software Compression: 65.5 % (gzip)
VSS: no
Encryption: no
Accurate: no
Volume name(s): Full-0001
Volume Session Id: 13
Volume Session Time: 1516308262
Last Volume Bytes: 39,085,090 (39.08 MB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK


Choose a query (1-20): 12
Enter JobId: 23
+------------------+---------------+
| path | name |
+------------------+---------------+
| /_pgsqlbackups_/ | dellstore.sql |
| /_pgsqlbackups_/ | postgres.sql |
+------------------+---------------+


---------------

So I have 2 questions.

1.) Does this mean the mysql plugin will also hang on the estimate? (i'm setting up a vm with mysql to test this on now but thought I would ask)
2.) How do I prevent my plugin from hanging on trying to estimate? I am assuming I am missing a function.

[Sean Sullivan]

Here is a link to the fork for the postgres plugin that I have currently if anyone wants it. Hopefully either I can fix it or someone else can:

https://github.com/seapasulli/bareos-contrib/tree/master/fd-plugins/postgresql-python

[Sean Sullivan]

Link to what I have so far. I successfully backed up and restored the .sql dumps to the server so I'm assuming it's working sans the estimate part.

https://github.com/lookcrabs/bareos-contrib/tree/master/fd-plugins/postgresql-python