Craig Barratt
2017-03-04 19:12:35 UTC
BackupPC community,
I'm happy to announce that BackupPC 4.0.0 has been released on Github
<https://github.com/backuppc/backuppc/releases> and SourceForge
<https://sourceforge.net/projects/backuppc/files/backuppc/4.0.0/>.
BackupPC 4.0.0 is a significant improvement over 3.x in terms of
performance and storage efficiency. It is backward compatible with 3.x
storage, so it can be used to upgrade an existing V3 installation as well
as for brand new installs.
BackupPC 4.0.0 requires the perl module BackupPC::XS
<https://github.com/backuppc/backuppc-xs/releases> (>= 0.50) and rsync-bpc
<https://github.com/backuppc/rsync-bpc/releases> (>= 3.0.9.5).
After installing those two packages, BackupPC 4.0.0 can be installed from
the tar ball with:
tar zxf BackupPC-4.0.0.tar.gz
cd BackupPC-4.0.0
perl configure.pl
See the README.md, ChangeLog and doc/BackupPC.html
<http://backuppc.sourceforge.net/BackupPC-4.0.0.html> files for more
information. An overview of the major changes is below.
Thanks for to everyone who provided fixes and reported issues with the
alpha and github versions.
Enjoy!!
Craig
Here's a summary of the most significant changes in V4:
- No use of hardlinks (except temporarily to do atomic renames).
Reference counting is handled at the application level in a batch manner
(hardlinks will still remain for any legacy V3 backups).
- Backups are stored as "reverse deltas" - the most recent backup is
always filled and older backups are reconstituted by merging all the deltas
starting with the nearest future filled backup and working backwards.
- This is the opposite of V3 where incrementals are stored as "forward
deltas" to a prior backup (typically the last full backup or prior
lower-level incremental backup, or the last full in the case of rsync).
- Since the most recent backup is filled, viewing/restoring that backup
(which is the most common backup used) doesn't require merging any deltas
from other backups.
- The concepts of incr/full backups and unfilled/filled storage are
decoupled. The most recent backup is always filled. By default, for the
remaining backups, full backups are filled and incremental backups are
unfilled, but that is configurable.
- Uses full-file MD5 digests, which are stored in the directory attrib
files. Each backup directory only contains an empty attrib file whose name
includes its own MD5 digest, which is used to look up the attrib file's
contents in the pool. In turn, that file contains the metadata for every
file in that directory, including each files's MD5 digest.
- The Pool layout still supports chains to handle md5 collisions. While
collisions can be constructed and are now well-known, they are highly
unlikely in the wild. Pool files are never renamed or moved, unlike V3.
- Any backup can be deleted (deltas are merged into next older backup if
it is not filled).
- The reverse deltas allow "infinite incrementals" - no need for a full
backup if you are willing to trade speed for the risk that a file change
will not be detected if the metadata (eg, mtime or size) doesn't change.
- An rsync "full" backup now uses --checksum (instead of
--ignore-times), which is much more efficient on the server side - the
server just needs to check the full-file checksum computed by the client,
together with the mtime, nlinks, size attributes, to see if the file has
changed. If you want a more conservative approach, you can change it back
to --ignore-times, which requires the server to send block checksums to the
client.
- The use of rsync --checksum allows BackupPC to guess a potential match
anywhere in the pool, even on a first-time backup. In that case, the usual
rsync block checksums are still exchanged to make sure the complete file is
identical.
- Uses a modified rsync called rsync_bpc (currently based on
rsync-3.0.9) on the server side (in place of File::RsyncP), with a C code
interface to the BackupPC storage. So the whole data path for rsync is now
in compiled C code, which is much faster than perl.
- Due to the use of rsync-3.X, acls and xattrs are supported, and many
other useful options (but not all) are supported. Rsync protocol 30
supports the efficient incremental file list, which significantly improves
memory usage and startup time. It also supports MD5 full-file checksums,
which match BackupPC's new digest. That allows a full-file digest to be
checked as easily as an mtime on the server side.
- Significant portions of the BackupPC code are now compiled C code in a
new module called BackupPC::XS that is dynamically linked to perl.
- V3 migration: nothing specific is needed. V4 can browse/view/restore
V3 backups. When you install V4, no changes are made to any V3 backups. If
you are upgrading from V3, be sure to set $Conf{PoolV3Enabled} to 1 so the
old V3 pool is searched for matching files.
- When you install V4, it will notice that the V3 pool exists.
Running configure.pl should set $Conf{PoolV3Enabled} to 1 in that
case, but you should be sure to check that.
- When a V4 backup is first done, BackupPC_backupDuplicate is run to
duplicate the most recent V3 backup to create a new V4 backup. A "filled"
view of the most recent V3 backup is used to create a "filled" V4 backup
tree.
- This step could be time consuming, since every file needs to be
read (as a V3 file) and written as a V4 file. However, the V4
pooling code
knows about the V3 pool, so it will move the V3 pool file into
the V4 pool.
So this duplication process doesn't burn a lot of pool storage space, but
every file still needs to be read (to compute the MD5 digest)
and "written"
(really just matching/linking).
- Expiry: all the V3 + V4 backups are considered on a combined basis
for expiry checking.
- On a clean new V4 install, the steps of computing and checking V3
digests is eliminated.
I'm happy to announce that BackupPC 4.0.0 has been released on Github
<https://github.com/backuppc/backuppc/releases> and SourceForge
<https://sourceforge.net/projects/backuppc/files/backuppc/4.0.0/>.
BackupPC 4.0.0 is a significant improvement over 3.x in terms of
performance and storage efficiency. It is backward compatible with 3.x
storage, so it can be used to upgrade an existing V3 installation as well
as for brand new installs.
BackupPC 4.0.0 requires the perl module BackupPC::XS
<https://github.com/backuppc/backuppc-xs/releases> (>= 0.50) and rsync-bpc
<https://github.com/backuppc/rsync-bpc/releases> (>= 3.0.9.5).
After installing those two packages, BackupPC 4.0.0 can be installed from
the tar ball with:
tar zxf BackupPC-4.0.0.tar.gz
cd BackupPC-4.0.0
perl configure.pl
See the README.md, ChangeLog and doc/BackupPC.html
<http://backuppc.sourceforge.net/BackupPC-4.0.0.html> files for more
information. An overview of the major changes is below.
Thanks for to everyone who provided fixes and reported issues with the
alpha and github versions.
Enjoy!!
Craig
Here's a summary of the most significant changes in V4:
- No use of hardlinks (except temporarily to do atomic renames).
Reference counting is handled at the application level in a batch manner
(hardlinks will still remain for any legacy V3 backups).
- Backups are stored as "reverse deltas" - the most recent backup is
always filled and older backups are reconstituted by merging all the deltas
starting with the nearest future filled backup and working backwards.
- This is the opposite of V3 where incrementals are stored as "forward
deltas" to a prior backup (typically the last full backup or prior
lower-level incremental backup, or the last full in the case of rsync).
- Since the most recent backup is filled, viewing/restoring that backup
(which is the most common backup used) doesn't require merging any deltas
from other backups.
- The concepts of incr/full backups and unfilled/filled storage are
decoupled. The most recent backup is always filled. By default, for the
remaining backups, full backups are filled and incremental backups are
unfilled, but that is configurable.
- Uses full-file MD5 digests, which are stored in the directory attrib
files. Each backup directory only contains an empty attrib file whose name
includes its own MD5 digest, which is used to look up the attrib file's
contents in the pool. In turn, that file contains the metadata for every
file in that directory, including each files's MD5 digest.
- The Pool layout still supports chains to handle md5 collisions. While
collisions can be constructed and are now well-known, they are highly
unlikely in the wild. Pool files are never renamed or moved, unlike V3.
- Any backup can be deleted (deltas are merged into next older backup if
it is not filled).
- The reverse deltas allow "infinite incrementals" - no need for a full
backup if you are willing to trade speed for the risk that a file change
will not be detected if the metadata (eg, mtime or size) doesn't change.
- An rsync "full" backup now uses --checksum (instead of
--ignore-times), which is much more efficient on the server side - the
server just needs to check the full-file checksum computed by the client,
together with the mtime, nlinks, size attributes, to see if the file has
changed. If you want a more conservative approach, you can change it back
to --ignore-times, which requires the server to send block checksums to the
client.
- The use of rsync --checksum allows BackupPC to guess a potential match
anywhere in the pool, even on a first-time backup. In that case, the usual
rsync block checksums are still exchanged to make sure the complete file is
identical.
- Uses a modified rsync called rsync_bpc (currently based on
rsync-3.0.9) on the server side (in place of File::RsyncP), with a C code
interface to the BackupPC storage. So the whole data path for rsync is now
in compiled C code, which is much faster than perl.
- Due to the use of rsync-3.X, acls and xattrs are supported, and many
other useful options (but not all) are supported. Rsync protocol 30
supports the efficient incremental file list, which significantly improves
memory usage and startup time. It also supports MD5 full-file checksums,
which match BackupPC's new digest. That allows a full-file digest to be
checked as easily as an mtime on the server side.
- Significant portions of the BackupPC code are now compiled C code in a
new module called BackupPC::XS that is dynamically linked to perl.
- V3 migration: nothing specific is needed. V4 can browse/view/restore
V3 backups. When you install V4, no changes are made to any V3 backups. If
you are upgrading from V3, be sure to set $Conf{PoolV3Enabled} to 1 so the
old V3 pool is searched for matching files.
- When you install V4, it will notice that the V3 pool exists.
Running configure.pl should set $Conf{PoolV3Enabled} to 1 in that
case, but you should be sure to check that.
- When a V4 backup is first done, BackupPC_backupDuplicate is run to
duplicate the most recent V3 backup to create a new V4 backup. A "filled"
view of the most recent V3 backup is used to create a "filled" V4 backup
tree.
- This step could be time consuming, since every file needs to be
read (as a V3 file) and written as a V4 file. However, the V4
pooling code
knows about the V3 pool, so it will move the V3 pool file into
the V4 pool.
So this duplication process doesn't burn a lot of pool storage space, but
every file still needs to be read (to compute the MD5 digest)
and "written"
(really just matching/linking).
- Expiry: all the V3 + V4 backups are considered on a combined basis
for expiry checking.
- On a clean new V4 install, the steps of computing and checking V3
digests is eliminated.