[BackupPC-devel] BackupPC 4.0 features - attribute file and backup storage

[Craig Barratt]

The next topic is the attribute file format and how backups
are stored.

In 4.x the only information that appears in a backup tree
(ie: a single backup stored below $TOPDIR/pc/HOST/NNN) are
the attribute files, one per directory.

In contrast, a filled backup in 3.x includes a complete set
of files using hardlinks, in addition to the attribute file.
Also in 3.x a full directory tree had to be created even
for incremental backups.

In 4.x it is no longer necessary for a complete backup tree of
directories to be created. Only sufficient directories to store
the attribute files necessary for the reverse time deltas are
needed.

In 4.x a new attribute file format is used. A new magic number
identifies the new format. For each file in the directory,
the 4.x attributes additionally stores:

- the file's digest (ie: MD5 full file digest, with possible
extension for collisions)

- an inode number (inode)

- an inode link count (nlinks)

- extended attributes (xattr)

The inode number is generated locally during the backup process
(ie: it is not related to the inode number on the client).

The 3.x BPC_FTYPE_HARDLINK file type has been eliminated (other
than to support legacy 3.x backups). Client hardlinks are stored
in a more faithful manner that allows the inode reference count
to be accurate, and also allows attribute changes on any of the
file's hardlink instances to correctly be mirrored among all the
instances.

For client hardlinks the inode link count in the attribute file is
greater than 1. In that case, the inode number (locally generated on
the server, not the real client inode) is used to access a secondary
attribute file that stores the real attributes of hardlinked files.
These attribute files are indexed by inode number. This provides the
right semantics - multiple hardlinks really do refer to a single file
and corresponding attributes.

Inodes attributes (for client hardlinks) are stored in an additional
tree of directories below each backup directory. The attribute hash key
is the inode number. 128 directories are used to store up to 128 attrib
files. The lower 14 bits of the inode number are used to determine the
directory (bits 13 to 7) and the file name within that directory
(bits 0-6). These two 7 bit values are encoded in hex (XX and YY):

$TOPDIR/pc/HOST/NNN/inode/XX/attribYY.

where NNN is the backup number.

Craig

[Tino Schwarze]

Hi Craig,

thanks for the information. I've got some questions, though...

Post by Craig Barratt
The next topic is the attribute file format and how backups
are stored.
In 4.x the only information that appears in a backup tree
(ie: a single backup stored below $TOPDIR/pc/HOST/NNN) are
the attribute files, one per directory.

The downside of this is that I cannot easily peek into a backup on
command line any more. :-( It's been very helpful to quickly figure out
which files have bee backed up (e.g. to determine unneccessarily backed
up stuff). Now I'll lose that capability and have to resort to web page
clicking... or us that FUSE FS part of the release?

Post by Craig Barratt
In contrast, a filled backup in 3.x includes a complete set
of files using hardlinks, in addition to the attribute file.
Also in 3.x a full directory tree had to be created even
for incremental backups.
In 4.x it is no longer necessary for a complete backup tree of
directories to be created. Only sufficient directories to store
the attribute files necessary for the reverse time deltas are
needed.

Ok, so how do you implement orphan deletion - how are files detected
which are not needed in pool any more?

Post by Craig Barratt
In 4.x a new attribute file format is used. A new magic number
identifies the new format. For each file in the directory,
- the file's digest (ie: MD5 full file digest, with possible
extension for collisions)
- an inode number (inode)
- an inode link count (nlinks)
- extended attributes (xattr)
The inode number is generated locally during the backup process
(ie: it is not related to the inode number on the client).

Note that this will blow up if somebody is using ReiserFS since that
does not have real inode numbers, but invents them on access. It might
be impossible to implement a sanity check to detect that case...

Post by Craig Barratt
Inodes attributes (for client hardlinks) are stored in an additional
tree of directories below each backup directory. The attribute hash key
is the inode number. 128 directories are used to store up to 128 attrib
files. The lower 14 bits of the inode number are used to determine the
directory (bits 13 to 7) and the file name within that directory
$TOPDIR/pc/HOST/NNN/inode/XX/attribYY.
where NNN is the backup number.

Oh, or do you mean: The inode number is made up by BackupPC? Then please
don't call it inode number, but rather hardlink-ID or so.

Thanks,

Tino, happy BackupPC user since 3.0.0.

[Jeffrey J. Kosowsky]

Post by Tino Schwarze
Hi Craig,
thanks for the information. I've got some questions, though...

Post by Craig Barratt
The next topic is the attribute file format and how backups
are stored.
In 4.x the only information that appears in a backup tree
(ie: a single backup stored below $TOPDIR/pc/HOST/NNN) are
the attribute files, one per directory.

The downside of this is that I cannot easily peek into a backup on
command line any more. :-( It's been very helpful to quickly figure out
which files have bee backed up (e.g. to determine unneccessarily backed
up stuff). Now I'll lose that capability and have to resort to web page
clicking... or us that FUSE FS part of the release?

I'm a big fan of the backuppc fuse filesystem -- I mean I find it just
awesome since it converts incrementals seamlessly to fulls and
includes all the attrib file information. And since they look like
real files you have the full power of the *nix command line and
utilities available. I also find it to be not only more flexible but
significantly faster than the web interface.

It would be nice to see the fuse filesystem interface be officially
integrated into backuppc 4.x. The version I am using was by the
author's admission (I believe) a non-optimized implementation so
presumably there is room to improve on it in terms of efficiency and
maybe even capabilities. While I know that fusefs may not be available
on all systems, I imagine that most users are running versions of *nix
that would support it.

[David Birnbaum]

Craig,

I realize a lot's gone into your thinking for 4.0 - I was wondering if
it would be possible to offer some sort of "cost" parameter to the host
configuration. Specifically, we have local hosts (backing up via a 1GB
Ethernet connection) and remote hosts (often backing up at 1M or 768k
through a DSL or T1). Right now, the parallel code doesn't allow for a
different overhead cost for doing backups. I was thinking you could add
something to the code allowing for a different weighting of remote hosts.

For example, if you specify a concurrency of 5, a slow host might
contribute 0.1, a medium host 0.5, and a fast host 1.0. That would
allow a bunch of slow transfers to happen, whereas fast hosts would
count at a much higher rate (for network and I/O contention).

Something along the lines of:

$Conf{backup_cost} = "slow";

or

$Conf{backup_cost} = 0.1;

Would do the trick nicely.

Cheers,

David.

[Dan Pritts]

Post by Tino Schwarze
Craig,
I realize a lot's gone into your thinking for 4.0 - I was wondering if it would be possible to offer some sort of "cost" parameter to the host configuration. Specifically, we have local hosts (backing up via a

Along similar lines, we backup a lot of VMware VMs as well as physical hosts.
I expect my backup hardware could back up concurrent clients, but it would hammer the
vmware storage if we did more than a couple of *those* concurrently.

We can't be the only ones with this problem.

In practice we just keep the concurrency down and we make it fit in our backup window,
but that won't scale infinitely.

Dan Pritts, Sr. Systems Engineer
Internet2
office: [phone] | mobile: [phone]

[Les Mikesell]

This post might be inappropriate. Click to display it.

[Jeffrey J. Kosowsky]

Post by Les Mikesell

Post by Dan Pritts

Post by Tino Schwarze
Craig,
I realize a lot's gone into your thinking for 4.0 - I was wondering if it would be possible to offer some sort of "cost" parameter to the host configuration. Specifically, we have local hosts (backing up via a

Along similar lines, we backup a lot of VMware VMs as well as physical hosts.
I expect my backup hardware could back up concurrent clients, but it would hammer the
vmware storage if we did more than a couple of *those* concurrently.
We can't be the only ones with this problem.
In practice we just keep the concurrency down and we make it fit in our backup window,
but that won't scale infinitely.

The concurrency control really needs some kind of 'grouping' concept to
tie physical constraints together. For example, to group VM guests and
the common host, or to group remote systems sharing limited WAN
bandwidth or a router link you don't want to swamp. The idea would be
to have a setting to limit concurrency within a group while continuing
to run members outside of the group up to a different limit. Amanda had
something like that related to network segments but I've forgotten how
it was used.

It seems like there are two factors we are trying to capture
- Relative resource intensity of each client backup
- Interactions between subsets of clients

It seems like we may need a concept both of 'grouping' and 'resource
intensity'

In any case, if such changes are made, it would be worth it to think
through all the common use cases so that the structure agreed upon is
broad and flexible enough to capture them all.

[John Rouillard]

Post by Jeffrey J. Kosowsky
It seems like there are two factors we are trying to capture
- Relative resource intensity of each client backup
- Interactions between subsets of clients
It seems like we may need a concept both of 'grouping' and 'resource
intensity'
In any case, if such changes are made, it would be worth it to think
through all the common use cases so that the structure agreed upon is
broad and flexible enough to capture them all.

To handle this problem with my setup, I used a shell based semaphore
system and DumpPreUserCmd.

I keep overall concurrecy high (10), but each site has a semaphore
with some number of resources (e.g. local lan links have 6 resources
to it's site semaphore while WAN semaphores have 1 or 2 resources) and
the number of resources limits the maximum number of backups that can
run.

So if one backup is running over the WAN to site A, and another site A
host is scheduled by backuppc, the DumpPreUserCmd command tries to
acquire a semaphore for site A. If no resources are available, the
DumpPreUserCmd command exits with a non-zero exit status and the
backup isn't started.

I only have each host accessing one semaphore (based on the site) but
there is no reason this couldn't be expanded to a host having to
acquire multiple resources before the backup starts.

So you could have a set of resource intensity resources as well as a
grouping (site) set of semaphores.

[Jeffrey J. Kosowsky]

Post by Craig Barratt
The next topic is the attribute file format and how backups
are stored.

Were there previous topics that you posted to the mailing list that I
missed? :P

Post by Craig Barratt
In 4.x the only information that appears in a backup tree
(ie: a single backup stored below $TOPDIR/pc/HOST/NNN) are
the attribute files, one per directory.
In contrast, a filled backup in 3.x includes a complete set
of files using hardlinks, in addition to the attribute file.
Also in 3.x a full directory tree had to be created even
for incremental backups.
In 4.x it is no longer necessary for a complete backup tree of
directories to be created. Only sufficient directories to store
the attribute files necessary for the reverse time deltas are
needed.

Hopefully, the code will expose building-block subroutines for
reconstructing the current backup from the reverse time deltas so that
people can create their own extensions for accessing the contents of
specific backups (e.g., like the backuppc fuse file system).

Post by Craig Barratt
In 4.x a new attribute file format is used. A new magic number
identifies the new format. For each file in the directory,
- the file's digest (ie: MD5 full file digest, with possible
extension for collisions)

By extension, would it be possible to write all the routines in
checksum agnostic fashion so that the choice of MD5 or sha256sum or
whatever could be made by the user by just supplying the right perl
calling routine or wrapper. This would make it possible for users to
use whatever checksum they feel comfortable with -- potentially
trading off speed for reliability

Post by Craig Barratt
- an inode number (inode)
- an inode link count (nlinks)
- extended attributes (xattr)

1. Would it be possible to allow for simple user-extensions to add other
fields. This would make it easier and more robust to extend should the
need or desirability of adding other file attributes arise.

2. Does the attrib file also presumably contain a pool file name so
that you can determine the pool file location?

Post by Craig Barratt
The inode number is generated locally during the backup process
(ie: it is not related to the inode number on the client).
The 3.x BPC_FTYPE_HARDLINK file type has been eliminated (other
than to support legacy 3.x backups). Client hardlinks are stored
in a more faithful manner that allows the inode reference count
to be accurate, and also allows attribute changes on any of the
file's hardlink instances to correctly be mirrored among all the
instances.

Will there be a routine to convert legacy 3.x stored backups to 4.x
format?
In some ways having to keep around all the 3.x hard-links would defeat
a lot of the easy copying benefits of 4.x or would require using a
separate partition.

Post by Craig Barratt
For client hardlinks the inode link count in the attribute file is
greater than 1. In that case, the inode number (locally generated on
the server, not the real client inode) is used to access a secondary
attribute file that stores the real attributes of hardlinked files.
These attribute files are indexed by inode number. This provides the
right semantics - multiple hardlinks really do refer to a single file
and corresponding attributes.
Inodes attributes (for client hardlinks) are stored in an additional
tree of directories below each backup directory. The attribute hash key
is the inode number. 128 directories are used to store up to 128 attrib
files. The lower 14 bits of the inode number are used to determine the
directory (bits 13 to 7) and the file name within that directory
$TOPDIR/pc/HOST/NNN/inode/XX/attribYY.
where NNN is the backup number.

Couple of questions:
1. Does that mean that you are limited to 2^14 client hard links?
2. Does the hardlink attrib file contain a list of the full file paths
of all the similarly hard-linked files?

This would be helpful for programs like my BackupPC_deleteFile
since if you want to delete one hard linked file, it would be
necessary presumably to decrement the nlinks count in all the
attrib files of the other linked files. Plus in general, it is nice
to know what else is hard-linked without having to do an exhaustive
search through the entire backup.

In fact, I would imagine that the atribYY file would only need to
contains such a list and little if any other information since that
is all that is required presumably to restore the hard-links since
the actual inode properties are stored (redundantly) in each of the
individual attrib file representations.


Other questions/suggestions:
1. Are the pool files now labeled with the full file md5sums?
If so, are you able to get that right out of the rsync checksums
(for protocol >=30 or so) or do they need to be computed
separately?

How is the (unlikely) event of an md5sum collision handled?
Is it always still necessary to compare actual file contents when
adding a new file just in case the md5sums collide?

If so, would it be better to just use a "better" checksum so that
you wouldn't need to worry about collisions and wouldn't have to
always do a disk IO and CPU consuming file content comparison to
check for the unlikely event of collision?

2. If you are changing the appended rsync digest format for cpool
files using rsync, I think it might be helpful to also store the
uncompressed filesize in the digest There are several use cases
(including verifying rsync checksums where the filesize is required
to determine the blocksize) where I have needed to decompress the
entire file just to find out its size (and since I am in the pool
tree I don't have access to the attrib file to know its size).

Similarly, it might be nice to *always* have the md5sum checksum
(or other) appended to the file even when not using the rsync
transfer method. This would help with validating file
integrity. Even if the md5sum is used for the pool file name, it
may be nice to allow for an alternative more reliable checksum to
be stored in the file envelope itself.

3. In the absence of pool hard links, how do you know when a pool file
can be deleted? Is there some table where the counts are
incremented/decremented as new backups are added or deleted?


4. Do you have any idea whether 4.0 will be or less resource
intensive (e.g., disk IO, network, cpu) than 3.x when it comes to:
- Backing up
- Reconstructing deltas in the web interface (or fuse filesystem)
- Restoring data


Most importantly, 4.0 sounds great and exciting!

[Craig Barratt]

Tino,

Post by Tino Schwarze
The downside of this is that I cannot easily peek into a backup on
command line any more. :-( It's been very helpful to quickly figure out
which files have bee backed up (e.g. to determine unneccessarily backed
up stuff). Now I'll lose that capability and have to resort to web page
clicking...

That's true. Maybe I could add an "ls" facility?

Post by Tino Schwarze
or us that FUSE FS part of the release?

Not currently planned, but it's a good idea. The new set up will
make FUSE more faithful (in particular, getting nlinks right).

Post by Tino Schwarze
Ok, so how do you implement orphan deletion - how are files detected
which are not needed in pool any more?

That was described in my other posts. Reference counting is done

Post by Tino Schwarze
Note that this will blow up if somebody is using ReiserFS since that
does not have real inode numbers, but invents them on access. It might
be impossible to implement a sanity check to detect that case...

It shouldn't matter - see below.

Post by Tino Schwarze

Post by Craig Barratt
Inodes attributes (for client hardlinks) are stored in an additional
tree of directories below each backup directory. The attribute hash key
is the inode number. 128 directories are used to store up to 128 attrib
files. The lower 14 bits of the inode number are used to determine the
directory (bits 13 to 7) and the file name within that directory
$TOPDIR/pc/HOST/NNN/inode/XX/attribYY.
where NNN is the backup number.

Oh, or do you mean: The inode number is made up by BackupPC? Then please
don't call it inode number, but rather hardlink-ID or so.

Right. It's made by BackupPC on the server side. But it would be
the inode number exposed by FUSE, so I'd like to call it that.

Craig

[John Rouillard]

Post by Craig Barratt

Post by Tino Schwarze
The downside of this is that I cannot easily peek into a backup on
command line any more. :-( It's been very helpful to quickly figure out
which files have bee backed up (e.g. to determine unneccessarily backed
up stuff). Now I'll lose that capability and have to resort to web page
clicking...

That's true. Maybe I could add an "ls" facility?

That would be great especially if it can list across multiple backups
to allow me to search for a file.

Post by Craig Barratt

Post by Tino Schwarze
or us that FUSE FS part of the release?

Not currently planned, but it's a good idea. The new set up will
make FUSE more faithful (in particular, getting nlinks right).

Will the fuse filesystem be runnable on Solaris, or *bsd or is it a
linux only solution?

Post by Craig Barratt

Post by Tino Schwarze

Post by Craig Barratt
Inodes attributes (for client hardlinks) are stored in an additional
tree of directories below each backup directory. The attribute hash key
is the inode number. 128 directories are used to store up to 128 attrib
files. The lower 14 bits of the inode number are used to determine the
directory (bits 13 to 7) and the file name within that directory
$TOPDIR/pc/HOST/NNN/inode/XX/attribYY.
where NNN is the backup number.

Oh, or do you mean: The inode number is made up by BackupPC? Then please
don't call it inode number, but rather hardlink-ID or so.

Right. It's made by BackupPC on the server side. But it would be
the inode number exposed by FUSE, so I'd like to call it that.

I got the impression that the pc tree will no longer have any
indication of the the (mangled) names of the original files.

We run is an automatic build server and rely on backups to recover
prior builds of software as a last fallback to recover from a bad
software build/install. Without actual files located under the pc tree
is there a way to search (cf. gnu locate or find) for a file matching
a particular regular expressions (e.g. python-orif.*.rpm) so I can go
to that backup and restore the file?

A directory history does some of that if the directory name remains
constant, but in some cases even the directory name changes from
release to release. Plus on a busy directory the history takes a long
time to load.

[Jeffrey J. Kosowsky]

Post by John Rouillard
We run is an automatic build server and rely on backups to recover
prior builds of software as a last fallback to recover from a bad
software build/install. Without actual files located under the pc tree
is there a way to search (cf. gnu locate or find) for a file matching
a particular regular expressions (e.g. python-orif.*.rpm) so I can go
to that backup and restore the file?

That's an interesting point -- a lot of us have gotten used to doing
f-mangled searches and/or wild-card listings of backups.

Fuse filesystem search would be relatively slow since it would still
need to read all the attrib files and rebuild deltas. Similarly, for
any routine that rebuilds based on the atrrib files.

Post by John Rouillard
A directory history does some of that if the directory name remains
constant, but in some cases even the directory name changes from
release to release. Plus on a busy directory the history takes a long
time to load.
--
-- rouilj
John Rouillard System Administrator
Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111
------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev
_______________________________________________
BackupPC-devel mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[John Rouillard]

Post by Jeffrey J. Kosowsky

Post by John Rouillard
We run is an automatic build server and rely on backups to recover
prior builds of software as a last fallback to recover from a bad
software build/install. Without actual files located under the pc tree
is there a way to search (cf. gnu locate or find) for a file matching
a particular regular expressions (e.g. python-orif.*.rpm) so I can go
to that backup and restore the file?

That's an interesting point -- a lot of us have gotten used to doing
f-mangled searches and/or wild-card listings of backups.
Fuse filesystem search would be relatively slow since it would still
need to read all the attrib files and rebuild deltas. Similarly, for
any routine that rebuilds based on the atrrib files.

Well fuse filesystem isn't that great if you are running backuppc on
solaris or *bsd so you can use zfs. I think netbsd supports it and I
know there was some opensolaris support for fuse, but I still consider
fuse on solaris not ready for production use.

[Craig Barratt]

Jeffrey,

Post by Jeffrey J. Kosowsky
Were there previous topics that you posted to the mailing list that I
missed? :P

They might have arrived out of order.

Post by Jeffrey J. Kosowsky
Hopefully, the code will expose building-block subroutines for
reconstructing the current backup from the reverse time deltas so that
people can create their own extensions for accessing the contents of
specific backups (e.g., like the backuppc fuse file system).

Yes, it's pretty easy to reconstruct a backup view.

Post by Jeffrey J. Kosowsky
By extension, would it be possible to write all the routines in
checksum agnostic fashion so that the choice of MD5 or sha256sum or
whatever could be made by the user by just supplying the right perl
calling routine or wrapper. This would make it possible for users to
use whatever checksum they feel comfortable with -- potentially
trading off speed for reliability

Let me think about this. I want to match the full-file checksum
in rsync because it allows some clever things to be done. For
example, with some hacks, it would allow a new client file to
be matched even the first time if it is in the pool and you use
the --checksum option to rsync (this requires some changes to
the server-side rsync).

Post by Jeffrey J. Kosowsky
1. Would it be possible to allow for simple user-extensions to add other
fields. This would make it easier and more robust to extend should the
need or desirability of adding other file attributes arise.

Good idea. Let me think about that.

Post by Jeffrey J. Kosowsky
2. Does the attrib file also presumably contain a pool file name so
that you can determine the pool file location?

The digest directly allows the pool name to be constructed. The digest
includes an extension in case of collisions.

Post by Jeffrey J. Kosowsky
Will there be a routine to convert legacy 3.x stored backups to 4.x
format?

The pool effectively gets migrated as each pool file is needed for 4.x.
But the old backups are not.

Post by Jeffrey J. Kosowsky
In some ways having to keep around all the 3.x hard-links would defeat
a lot of the easy copying benefits of 4.x or would require using a
separate partition.

It's a good point. However, such a utility would probably have a very
long running time. It wouldn't be reversible, and I would be very
concerned about cases where it was killed or failed before finishing.

I was thinking I would add a utility that lists all the 4.x directories
(below $TOPDIR) that should be backed up. Those wouldn't have hardlinks.
That could be used with a user-supplied script to backup or replicate
all the 4.x backups.

Post by Jeffrey J. Kosowsky
1. Does that mean that you are limited to 2^14 client hard links?

No. It's 2^14 files, each of which could, in theory subject
to memory, hold a large number of entries.

Post by Jeffrey J. Kosowsky
2. Does the hardlink attrib file contain a list of the full file paths
of all the similarly hard-linked files?

No. Just like a regular file sysem the reverse lookup isn't easy.

Post by Jeffrey J. Kosowsky
This would be helpful for programs like my BackupPC_deleteFile
since if you want to delete one hard linked file, it would be
necessary presumably to decrement the nlinks count in all the
attrib files of the other linked files.

No, that's not necessary. The convention is that if nlinks >= 2
in a file entry, all that means is the real data is in the inode
entry (and the nlinks could actually be 1).

Post by Jeffrey J. Kosowsky
Plus in general, it is nice
to know what else is hard-linked without having to do an exhaustive
search through the entire backup.
In fact, I would imagine that the atribYY file would only need to
contains such a list and little if any other information since that
is all that is required presumably to restore the hard-links since
the actual inode properties are stored (redundantly) in each of the
individual attrib file representations.

True. But you have a replicate and update all the file attributes
in that case. So any operation (like chmod etc) would require all
file attributes to be changed.

Post by Jeffrey J. Kosowsky
1. Are the pool files now labeled with the full file md5sums?
If so, are you able to get that right out of the rsync checksums
(for protocol >=30 or so) or do they need to be computed
separately?

The path name of the pool file gives the md5 sum.

Post by Jeffrey J. Kosowsky
How is the (unlikely) event of an md5sum collision handled?

Chains are still used for collisions. They are obviously unlikely
in typical use, but I have been testing it with the now well-known
files that have collisions.

Post by Jeffrey J. Kosowsky
Is it always still necessary to compare actual file contents when
adding a new file just in case the md5sums collide?

Yes.

Post by Jeffrey J. Kosowsky
If so, would it be better to just use a "better" checksum so that
you wouldn't need to worry about collisions and wouldn't have to
always do a disk IO and CPU consuming file content comparison to
check for the unlikely event of collision?

Good point, but as I mentioned above I would like to take advantage
of the fact that it is the same full-file checksum as rsync.

Post by Jeffrey J. Kosowsky
2. If you are changing the appended rsync digest format for cpool
files using rsync, I think it might be helpful to also store the
uncompressed filesize in the digest There are several use cases
(including verifying rsync checksums where the filesize is required
to determine the blocksize) where I have needed to decompress the
entire file just to find out its size (and since I am in the pool
tree I don't have access to the attrib file to know its size).

Currently 4.x won't use the appended checksums. I'll explain how
I'm implemented rsync on the server side in another email. I could
add that for later, but it is more complex.

Post by Jeffrey J. Kosowsky
Similarly, it might be nice to *always* have the md5sum checksum
(or other) appended to the file even when not using the rsync
transfer method. This would help with validating file
integrity. Even if the md5sum is used for the pool file name, it
may be nice to allow for an alternative more reliable checksum to
be stored in the file envelope itself.

Interesting idea.

Post by Jeffrey J. Kosowsky
3. In the absence of pool hard links, how do you know when a pool file
can be deleted? Is there some table where the counts are
incremented/decremented as new backups are added or deleted?

Yes - see my other emails (assuming I sent them all out).

Post by Jeffrey J. Kosowsky
4. Do you have any idea whether 4.0 will be or less resource
- Backing up
- Reconstructing deltas in the web interface (or fuse filesystem)
- Restoring data

I hope it will be a lot more efficient, but I don't have any data
yet. There are several areas where efficiency will be much better.
For example, with reverse-deltas, a full or incremental backup with
no changes shouldn't need any significant disk writes. In contrast
3.x has to create a directory tree and in the case of a full make
a complete set of hardlinks.

Also, rsync on the server side will be based on a native C rsync.
I'll send another email about that.

Post by Jeffrey J. Kosowsky
Most importantly, 4.0 sounds great and exciting!

I hope to get some time to work on it again. Unfortunately I haven't
made any progress in the last 4 months. Work has been very busy.

Craig

[Jeffrey J. Kosowsky]

This post might be inappropriate. Click to display it.