[BackupPC-devel] BackupPC 4.0.0alpha1 released

[Craig Barratt]

BackupPC community,

I'm pleased to announce that BackupPC 4.0.0alpha1 has been released on
SourceForge at:

https://sourceforge.net/projects/backuppc/files/backuppc-beta/4.0.0alpha1/


BackupPC 4.0.0alpha1 contains a few bug fixes since BackupPC 4.0.0alpha0
was released just over a week ago. If you are testing BackupPC 4.0.0alpha0,
I recommend you upgrade.

Each of the three packages in the release has been updated. You should
upgrade all three packages:

- BackupPC-4.0.0alpha1.tar.gz: the usual BackupPC release tar ball.
- BackupPC-XS-0.10.tar.gz: a perl XS module with C code that replaces
several BackupPC perl libraries for improved performance.
- rsync-bpc-3.0.9.1.tar.gz: a modified rsync that runs on the server that
has a shim layer that interfaces directly to the BackupPC file system.

As before, I do not recommend using this release in any production
environment. Having some people try it out in a sandbox in their
environment would be very helpful

I'm attaching some short notes on the build/install steps for each package.

Craig

BackupPC-XS-0.10.tar.gz:

tar zxvf BackupPC-XS-0.10.tar.gz
cd BackupPC-XS-0.10
perl Makefile.PL
make
make test
make install

rsync-bpc-3.0.9.1.tar.gz:

tar zxvf rsync-bpc-3.0.9.1.tar.gz
cd rsync-bpc-3.0.9.1
./configure.sh
make
make install

BackupPC-4.0.0alpha1.tar.gz:

tar zxvf BackupPC-4.0.0alpha1.tar.gz
cd BackupPC-4.0.0alpha1
./configure.pl

The last step for each will need to be run as a privileged user.

If you want to install rsync_bpc in /usr/local/bin (default might be
/usr/bin), then you should add the --prefix option to configure.sh:

./configure.sh --prefix=/usr/local

[b***@kosowsky.org]

Post by Craig Barratt
BackupPC community,
I'm pleased to announce that BackupPC 4.0.0alpha1 has been released on

Awesome...
It would be great to hear feedback from early adopters... both to
understand any potential issues and to build confidence and excitement
from real world users... Maybe just keep such comments on the devel
list if this is not interesting to all users.

[Tomasz Chmielewski]

Post by Craig Barratt
I'm pleased to announce that BackupPC 4.0.0alpha1 has been released
https://sourceforge.net/projects/backuppc/files/backuppc-beta/4.0.0alpha1/
BackupPC 4.0.0alpha1 contains a few bug fixes since BackupPC
4.0.0alpha0 was released just over a week ago. If you are testing
BackupPC 4.0.0alpha0, I recommend you upgrade.

Hi,

I was wondering if there are any unimplemented (but planned) features or
known bugs for BackupPC 4.0.0 (thinking of general stuff needing fixing
before releasing 4.0.0).

[Craig Barratt]

Post by Tomasz Chmielewski
I was wondering if there are any unimplemented (but planned) features or
known bugs for BackupPC 4.0.0 (thinking of general stuff needing fixing
before releasing 4.0.0).

I'm not aware of any current bugs, but that demonstrates the need for more
testing rather than readiness for a production release.

There are a few features I'd like to add for 4.0.0:
- parallelize the V4 pool reference count updates and scan, which is run
each night (I just implemented this).
- support FTP transfers (BTW, does anyone use FTP?)

There are some more significant features that I'm starting to think about.
I'm not sure any will make 4.0.0 because some involve significant changes:
- rewrite the main scheduler to allow host groups (an often requested
feature) and perhaps the ability to do absolute scheduling of backups (ie:
do a full every Friday night at 11pm).
- add plots using rrdtool based on earlier work by Ludovic Drolez and
Nicholas Hall, with recent improvements from Alexander Moisseev.
- look at SCGI support, allowing the BackupPC web server to be decoupled
from apache. That would avoid any setuid scripts or the need to run apache
as the BackupPC user.

Craig

[Craig Barratt]

Jean,

I should have been clearer in my explanation. This isn't a user feature in
4.0.0 that allows users to delete any backup.

What I was trying to say was that 4.0.0 includes the ability to delete any
backup, while maintaining the consistency of the reverse deltas. That
simplifies the deletion logic. In contrast, 3.x doesn't allow a backup to
be deleted that others depend on for the deltas. It's not a user-level
feature though.

Craig

hi,
this is great news to see v4 in the radar, we love this wonderfull piece
of software and seing a new version is exciting :)
I saw : Any backup can be deleted (deltas are merged into next older
backup if it is not filled).
in the changelogs. Can this possibility be turned off or restricted on a
user/host/general basis in the web Gui ?
One of the thing i love with backuppc is that even an hacked account
cannot be wiped out, limiting the famous "fired employee that logs and
wipe all including backup" one minute before leaving problem :)
we encountered the issue once and the fact that backuppc was read only
saved the day.
regards,
Jean

[Tomasz Chmielewski]

Any ideas how to get this compiled? This is on Debian 7.


$ tar xpf BackupPC-XS-0.10.tar.gz
$ cd BackupPC-XS-0.10
$ perl Makefile.PL
Checking if your kit is complete...
Looks good
MakeMaker (v6.57_05)
Warning (non-fatal): Target 'dynamic' depends on targets in skipped section 'dynamic_lib'
Warning (non-fatal): Target 'static' depends on targets in skipped section 'static_lib'
Writing Makefile for BackupPC::XS::md5
Writing MYMETA.yml
MakeMaker (v6.57_05)
Warning (non-fatal): Target 'dynamic' depends on targets in skipped section 'dynamic_lib'
Warning (non-fatal): Target 'static' depends on targets in skipped section 'static_lib'
Writing Makefile for BackupPC::XS::zlib
Writing MYMETA.yml
Writing Makefile for BackupPC::XS
Writing MYMETA.yml
***@srv4:~/BackupPC-XS-0.10# make
./configure.sh
configure.sh: Configuring rsync 3.0.9
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether to include debugging symbols... yes

(...)

checking whether to support ACLs... running tests:
checking for acl_get_file in -lacl... no
checking for ACL support... no
checking ACL test results... No ACL support found
checking whether to support extended attributes... Using Linux xattrs
configure.sh: creating ./config.status
config.status: creating config.h

rsync 3.0.9 configuration successful

cp lib/BackupPC/._XS.pm blib/lib/BackupPC/._XS.pm
cp lib/._BackupPC blib/lib/._BackupPC
cp lib/BackupPC/XS.pm blib/lib/BackupPC/XS.pm
make[1]: Entering directory `/root/BackupPC-XS-0.10/md5'
cc -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"\" -DXS_VERSION=\"\" -fPIC "-I/usr/lib/perl/5.14/CORE" ._md5.c
._md5.c:1:1: warning: null character(s) ignored [enabled by default]
._md5.c:1:2: error: stray ‘\5’ in program
._md5.c:1:2: error: stray ‘\26’ in program
._md5.c:1:2: error: stray ‘\7’ in program
._md5.c:1:5: warning: null character(s) ignored [enabled by default]
._md5.c:1:2: error: stray ‘\2’ in program
._md5.c:1:7: warning: null character(s) ignored [enabled by default]
._md5.c:1:9: error: unknown type name ‘Mac’
._md5.c:1:16: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘X’
._md5.c:1:17: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\2’ in program
._md5.c:1:27: warning: null character(s) ignored [enabled by default]
._md5.c:1:35: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\254’ in program
._md5.c:1:39: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\2’ in program
._md5.c:1:43: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\336’ in program
._md5.c:1:47: warning: null character(s) ignored [enabled by default]
._md5.c:1:89: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\336’ in program
._md5.c:1:97: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\230’ in program
._md5.c:1:101: warning: null character(s) ignored [enabled by default]
._md5.c:1:105: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\1’ in program
._md5.c:1:121: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\230’ in program
._md5.c:1:125: warning: null character(s) ignored [enabled by default]
._md5.c:1:129: warning: null character(s) ignored [enabled by default]
._md5.c:1:16: error: stray ‘\25’ in program
._md5.c:1:152: warning: null character(s) ignored [enabled by default]
._md5.c:1:160: error: invalid suffix "d1931d" on integer constant
._md5.c:1:160: error: expected identifier or ‘(’ before numeric constant
._md5.c:1:160: error: stray ‘\’ in program
._md5.c:1:169: error: unknown type name ‘Google’
._md5.c:1:194: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘-’ token
._md5.c:1:195: error: invalid suffix "B59" on integer constant
._md5.c:1:200: error: invalid suffix "B" on floating constant
._md5.c:1:210: error: invalid suffix "D94FFFC0866" on integer constant
._md5.c:1:222: warning: null character(s) ignored [enabled by default]
make[1]: *** [._md5.o] Error 1
make[1]: Leaving directory `/root/BackupPC-XS-0.10/md5'
make: *** [subdirs] Error 2

[Tomasz Chmielewski]

On Sat, 13 Jul 2013 14:26:40 +0800

Post by Tomasz Chmielewski
Any ideas how to get this compiled? This is on Debian 7.

(...)

Post by Tomasz Chmielewski
cc -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector
-fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"\" -DXS_VERSION=\"\"
stray ‘\5’ in program ._md5.c:1:2: error: stray ‘\26’ in
error: stray ‘\2’ in program ._md5.c:1:7: warning: null character(s)
ignored [enabled by default] ._md5.c:1:9: error: unknown type name
‘Mac’ ._md5.c:1:16: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or
‘__attribute__’ before ‘X’ ._md5.c:1:17: warning: null character(s)

Ah, there are some bogus files in the archive (some binary garbage with
"Mac OS X" string in it?) - this seems to do the trick:

find -name ._\* | xargs rm

[Craig Barratt]

Yes - you have the right solution. These files shouldn't be in the release.

I was using a new MacOSX machine, and I botched the final tarballs for
alpha1 by using the default bsdtar on MacOSX (instead of gnu tar), and it
added the MacOSX AppleDouble files, which contain metadata which is useless
in this case.

Craig

[Tomasz Chmielewski]

- An rsync "full" backup now uses --checksum (instead of
--ignore-times), which is much more efficient on the server side -
the server just needs to check the full-file checksum computed by the
client, together with the mtime, nlinks, size attributes, to see if
the file has changed. If you want a more conservative approach, you
can change it back to --ignore-times, which requires the server to
send block checksums to the client.

--checksum is quite paranoid and causes substantial IO on both sides.

Basically, it would cause md5sum calculated for each and every file, on
both sides.
If the archive is dozens of gigabytes or more - it means:

- extra CPU used,

- extra IO used,

- since we have to read all the files, anything the server had in
cache/buffers, will be purged from there (newly read files go to
cache/buffers instead, but it's not very useful there, since the
backup will be most likely made daily or less often).

All above cause visible slowdown.


I'm pretty sure that my clients don't secretly change the file content,
while preserving their size and timestamp.
In that case - will BackupPC still work correctly if I change the
default:

$Conf{RsyncFullArgsExtra} = [
'--checksum',
];


to:

$Conf{RsyncFullArgsExtra} = [
'',
];

?

[Craig Barratt]

Post by Tomasz Chmielewski
--checksum is quite paranoid and causes substantial IO on both sides.
Basically, it would cause md5sum calculated for each and every file, on
both sides.
- extra CPU used,
- extra IO used,

No, that is not true. On the server side in 4.x, so long as that file was
previously backed up with the same path for that client, the full-file
md5sum is stored in the file attributes, so it takes no more effort to
check the md5sum that comparing other attributes like the mtime and size.
(Of course, if the file isn't in the pool already, the server needs to
compress and write it to the pool, which is an expensive operation.)

On the client side you are correct - the entire file needs to be read. But
isn't that the point of a full backup? The client-side load is similar to
version 3.x, where a full backup requires the client to also read the
entire contents of every file.

There are two (typically one-time) cases where the client might do a bit
more work compared to 3.x:

- If the file isn't in the pool at all, the server will send empty block
checksums (ie: nothing to match), and the client will then send the whole
file verbatim. That requires two reads of the file on the client (first to
get the md5sum, and second to send the file). But these are close in time
and the second read will probably be cached.
- If the file is anywhere in the pool (even if it's a first backup of a
brand new client), the right pool file will be a likely match (based on
md5sum), and block digests will be sent to the client. If it is a match,
very little network traffic is needed, but the client needs to re-read the
whole file to be sure. So that also requires two reads of the file on the
client, and one on the server. Again, the two client reads are close in
time and the second read will probably be cached.

- since we have to read all the files, anything the server had in

Post by Tomasz Chmielewski
cache/buffers, will be purged from there (newly read files go to
cache/buffers instead, but it's not very useful there, since the
backup will be most likely made daily or less often).

That's a good point. Igor Sverkos recently pointed out the fadvise patch
to rsync that gives hints via the posix_fadvise call whether to cache
certain files or not. That patch significantly reduces the impact you
describe. However, in the steady state, rsync_bpc on the 4.x server isn't
reading entire files very often. I think the patch is likely to help the
client side more than the server, but I haven't tried it.

I'm pretty sure that my clients don't secretly change the file content,

Post by Tomasz Chmielewski
while preserving their size and timestamp.
In that case - will BackupPC still work correctly if I change the
$Conf{RsyncFullArgsExtra} = [
'--checksum',
];
$Conf{RsyncFullArgsExtra} = [
'',
];

If you are comfortable with mtime, size etc catching all changes to files,
then rather than disabling --checksum in $Conf{RsyncFullArgsExtra}, I
recommend you only ever do incremental backups. 4.x supports that. Set
$Conf{FullPeriod} to a very large value.

If you are doing incremental-only, you should set $Conf{FillCycle} to, say,
7. This will make sure every 7th backup is filled. The "Full" backup
delete settings (eg: "FullKeepCnt") actually mean "Filled" backups in 4.x,
so they control how many of those filled backups to keep, including the
exponential expiry option.

Note that, by default, $Conf{FillCycle} is 0, which keeps fulls filled, and
incrementals not filled (except for the most recent backup which is always
filled), so the delete settings work as you would expect. I grappled with
whether I should rename "FullKeepCnt" etc to "FillKeepCnt", but I thought
that would be more confusing for existing users (and for old 3.x backups,
FillKeepCnt would really still mean FullKeepCnt -- ugh!). Plus it's risky
to change all the per-PC client configs to rename the variables.
Suggestions are welcome.

Craig

[Craig Barratt]

I should also mentioned you can just replace --checksum in
$Conf{RsyncFullArgsExtra} with --ignore-times if you want the "full"
semantics to closely match 3.x.

In this case, 4.x will have more server load than 3.x since it has to read
all the file contents to get the block checksums, while in 3.x they are
cached at the end of the compressed file. On the other hand, 4.x is
running native C code, and 3.x is mostly in perl, so it's hard to predict
that actual difference.

I don't plan to implement block checksum caching in 4.x since I think
--checksum is a better approach. Also, rsync's 30 generations of protocols
(all backward compatible!) has 3 flavors of block checksums, so supporting
all 3 for older versions is extra work, tricky, and takes a lot more
testing. (3.x had to support two of the three.)

Craig

[Kenneth Porter]

--On Monday, July 01, 2013 12:53 PM -0700 Craig Barratt

Post by Craig Barratt
I'm attaching some short notes on the build/install steps for each package.

Has anyone created a spec file to build an RPM? I'm figuring I can adapt
the one from CentOS but don't want to re-invent the wheel if someone else
has already done it.