[BackupPC-devel] BackupPC v3 and v4 Development (fwd)

[Stephen]

Hi Michael,

I find your report useful and interesting. However my experience is a bit
different from yours.

Hello,
I've been testing BackupPC 4.0.0alpha3 for 1 year now, for backing up 12
home machines, and to be honest, I'm quite unhappy with it.
To my opinion, it is completely unreliable, you have to regularly check
whether backups are done correctly, and most of the time you can't do a
backup without at least an error. And it's awfully slow. The big
advantage of BPC (besides being free and open-source of course) is to
manage backup of multiple machines in a single pool, hence saving space.

I've been using v4 in production for almost 10 months, and I disagree. I have
found v4 to be very stable and useful. I have two v4 servers (along with at
least 3 older v3 servers) and the largest v4 install backs up 96 hosts

- 534 full backups of total size 3495.74GiB (prior to pooling and
compression),
- 2563 incr backups of total size 18496.62GiB (prior to pooling and
compression).

I find v4's speed to be better than v3 and do not see any more errors than I
did with v3 respecting xfers, bad files, etc. In fact, I can only find 1
instance in my logs and that's due to backing up an open file. Now with either
v3 or v4, if you try to back up the wrong files you'll encounter lots of pain
(and errors). Are you excluding special files? The exact list may vary somewhat
on your clients' distro and your site policy.

For Ubuntu at my site, I currently exclude /proc, /sys, /tmp, /var/tmp,
/var/cache/openafs, /var/cache/apt/archives, /var/log/lastlog,
/var/lib/mlocate, /var/spool/torque/spool, /home, /afs, /scratch*,
/not_backed_up*, /vicep*, /srv*, /spare*, /media, /mnt*

For select hosts, /home is backed up separately, using ClientNameAlias.

My speeds vary from over 100MiB/s (when backing up a few new sparse files) to
0.51MiB/s (a tiny incremental where it took a while to determine there was
nothing to do). Average for a 3GB full seems to be about 9MiB/s.

Notice that I'm not saying BPC v4 doesn't have bugs. I've found a couple of
them and reported one - with a possible solution - to the -devel list. But any
new software is likely to have bugs and this is reflected in the fact that v4
is still alpha. You're supposed to be *testing* v4 at this point. I'm using it
in production because I think its pros outweigh its cons and I'm willing to
hack the code (or otherwise suffer consequences) when I encounter bugs.

My current backup pool is ~ 12 machine. 11 on Linux and 1 windows
machine. My backup machine is a 3TB Lacie-Cloudbox, with 256 MB memory.
Some of you might say that 256 MB is not enough. Actually I've even seen
posts on the net saying that you would need a server with several GB
RAM. This is just insane. A typical PC in my pool has ~600k files.
Representing each of them with a 256-bit hash, that's basically 20MB of
data to manage for each backup. Of course you need some metadata, etc,
but I see no reason why you need GB of memory to manage that.

You probably don't want to hear it, but the Cloudbox probably is your
bottleneck. A colleague once experimented with BackupPC on a Synology Disk
Station. It worked. But it was quite slow. We eventually turned the Synology
into an iSCSI target hanging off a commodity PC with 2GB of RAM and the speed
increased a lot (unfortunately I didn't benchmark it as I knew it was a win-win
to separate the CPU from the storage and didn't hesitate to do so).

My current server for the 96 hosts is a recycled Dell PE 1950 (8 cores, 16 GB
RAM) backing up to a Dell PE 2950 (4 cores, 8GB RAM) via 4Gbit FC. BPC runs on
teh 1950; the 2950 is just a storage server. Clients are mostly on 1Gbit
ethernet. Server and clients are Ubuntu.

If I would participate to the development of BPC, I would make more
changes to the architecture. I think that the changes from 3.0 to 4.0
are very promising, but not enough. The first thing to do is to trash
rsync/rsyncd and use a client-side sync mechanism (like unison).

I think an *optional* client-side sync mechanism (like unison), implemented as
an additional xfer option, is interesting. Especially if an end-user can
manually initiate a backup or restore via a client interface (ala Crashplan but
hopefully without the java dependency). However I'm bothered by a
recommendation to "trash rsync/rsyncd". There's *zero* reason to eliminate
those xfer methods and I think doing so would immediately make any fork much
less likely to succeed.

Then
throw away all Perl code and rewrite in C.

This is the direction that v4 is headed, but I think the use of C should be
judicious. There's little reason for some parts of the code (cgi, etc) to be in
C.

Just my two cents.

Cheers,
Stephen

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
BackupPC-***@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[Michael]

Hi Stephen,
Hi all,

Thanks for your feedback and sharing your experience.
Here some clarification on my side.


*** Regarding BPC being not reliable.

I don't deny that BPC works very well in many situations, and can sustain
heavy load etc. But in my case it was not the flawless setup I imagined at
first. Of course, the main problem is the small amount of memory. Looking
in the dmesg logs, I could spot regularly OOM messages, the kernel killing
the backuppc_dump process, etc. Now it is a bit unfair of me blaming BPC
when actually the main culprit is the lack of memory. But the thing is
that BPC is quite unhelpful in these situations. Server logs are mostly
useless, no timestamps, and there is no attempt to restart again, but
instead BPC goes into a long process of counting references, etc, meaning
most of the server time is spent in (apparently) unproductive tasks.
Again, the main culprit is the platform, and actually BPC never lost any
data (afaik), and always recovered somehow. Still, there are some traces
of corruption in the db (like warning about reference being equal to -1
instead of 0), indicating that maybe BPC is not atomic.


*** Regarding the 256MB requirement.

Admittedly this is very demanding and very far off most feedbacks I've
seen on the net. Stephen's setup of a recycled Dell PE 1950 (8 cores, 16
GB RAM) seems more typical than my poor lacie-cloudbox with 256MB. But
when I monitor memory usage, for instance when doing a full backup of
600k-file client, BPC dump/rsync processes consume consistently around
100MB memory (htop/smem). Looking at rsync page, they say that rsync 3.0+
should consume 100bytes/file, so a total of 60MB for rsync. So I don't see
any blocking point why BPC would not fit in a 256-MB memory budget. Of
course it will be slower, but it must work. This WE I again spent some
time tuning down the Lacie-Cloudbox, stripping away all useless processes,
like those hungry python stuff. Now, in idle, the Lacie has 200MB free
physical + 200MB free swap space, and in that setup BPC worked for 2 days
w/o crash doing a full 55GB, 650k files backup, in 2 hours, 7.5MB/s
(almost no changes, hence the very high speed of course). For now I
disabled all my machines but a few, and will enable the remaining ones one
by one. I have good hope that it will work again. Now, my wishes would be
to restore some other services, but this will likely require increasing
the swap space.


*** Regarding BPC being slow

I only give BPC 256MB, so I should expect too much regarding performance.
That I fully agree. However when I say it is slow, I mean it is slow even
if I take into account that fact. Transfer speed is ok-ish; it uses rsync
at its best, which requires some heavy processing sometimes. But I don't
understand why it needs so much time for the remaining tasks (ref
counting, etc). I'm actually convinced (perhaps naively) that this can be
significantly improved. See further down.


*** Regarding "trashing" rsync
+ @kosowsky.org about designing a totally new backup program

My statement was... too brutal I guess ;-) I fully agree with Stephen's
comment. And I don't want to create a new program from scratch. rsync is
one of the best open-source sync program. Unison and duplicity are
basically using rsync internally. I do think however that BPC can be
significantly improved.

- Flatten the backup hierarchy

Initially BPC was a "mere" wrapper around rsync. First duplicating a
complete hierarchy with hard-links, then rsync'ing over it. It had the
advantage of simplicity but is very slow to maintain, and impossible to
duplicate. Now the trend in 4.0alpha was to move to a custom C
implementation of rsync, where hierarchy only stores attrib files. I
think that we can improve the maintenance phase further (ref counting,
backup deletion...) by flattening this structure into a single linear
file, and by listing once for all the references in a given backup,
possibly with caching of references per directory. Directory entries
would be more like git objects, attaching a name to a reference along
with some metadata. This means integrating further with the inner
working of rsync. It would be fully compliant with rsync from the client
side. But refcounting and backup deletion should then be equivalent
to sorting and finding duplicate/unique entries, which can be very
efficient. Even on my Lacie sorting a 600k-line file with 32B random
hash entries takes only a couple seconds.

- Client-side sync

Sure, this must be an optional feature, and I agree this is not the
priority. Many clients will still simply run rsyncd or rsync/ssh. But
the client-side sync would allow to detect hard links more efficiently.
It will also decrease memory usage on the server (see rsync faq). Then
it opens up a whole new set of optimization, delta-diff on multiple
files...


*** Regarding writing in C

Ok, I'm not a perl fan. But I agree, it is useful for stuff where
performance does not matter, for website interface, etc. But I would
rewrite in C the ref counting part and similar.

Kind regards,
Michaël



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
BackupPC-***@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[Adam Goryachev]

Post by Michael
Hi Stephen,
Hi all,
Thanks for your feedback and sharing your experience.
Here some clarification on my side.
*** Regarding BPC being not reliable.
I don't deny that BPC works very well in many situations, and can sustain
heavy load etc. But in my case it was not the flawless setup I imagined at
first. Of course, the main problem is the small amount of memory. Looking
in the dmesg logs, I could spot regularly OOM messages, the kernel killing
the backuppc_dump process, etc. Now it is a bit unfair of me blaming BPC
when actually the main culprit is the lack of memory. But the thing is
that BPC is quite unhelpful in these situations. Server logs are mostly
useless, no timestamps, and there is no attempt to restart again, but
instead BPC goes into a long process of counting references, etc, meaning
most of the server time is spent in (apparently) unproductive tasks.
Again, the main culprit is the platform, and actually BPC never lost any
data (afaik), and always recovered somehow. Still, there are some traces
of corruption in the db (like warning about reference being equal to -1
instead of 0), indicating that maybe BPC is not atomic.

I think adding timestamps to logs is not a significant problem, and
shouldn't be difficult to do. However, which log entries deserve a
timestamp? Every single one? Let's assume a timestamp in this format
20160112-094440 ("YYYYMMDD-HHMMSS "), that is just 16 bytes per line,
plus some small overhead to lookup and format the data. For a logfile of
3 million files, that's 48MB of timestamps you just added to an already
massive log file. Maybe we could add a timestamp every minute or every x
minutes? However, then you make the log harder to parse because each
line is not consistent.... Maybe start a thread on this specific issue,
and lets discuss the options, and see what most people think is the most
useful.

Post by Michael
*** Regarding the 256MB requirement.
Admittedly this is very demanding and very far off most feedbacks I've
seen on the net. Stephen's setup of a recycled Dell PE 1950 (8 cores, 16
GB RAM) seems more typical than my poor lacie-cloudbox with 256MB. But
when I monitor memory usage, for instance when doing a full backup of
600k-file client, BPC dump/rsync processes consume consistently around
100MB memory (htop/smem). Looking at rsync page, they say that rsync 3.0+
should consume 100bytes/file, so a total of 60MB for rsync. So I don't see
any blocking point why BPC would not fit in a 256-MB memory budget. Of
course it will be slower, but it must work. This WE I again spent some
time tuning down the Lacie-Cloudbox, stripping away all useless processes,
like those hungry python stuff. Now, in idle, the Lacie has 200MB free
physical + 200MB free swap space, and in that setup BPC worked for 2 days
w/o crash doing a full 55GB, 650k files backup, in 2 hours, 7.5MB/s
(almost no changes, hence the very high speed of course). For now I
disabled all my machines but a few, and will enable the remaining ones one
by one. I have good hope that it will work again. Now, my wishes would be
to restore some other services, but this will likely require increasing
the swap space.

I don't think main development of BPC should be setup around what is
essentially an embedded platform. However, if there is some random piece
of BPC which is allocating memory where it isn't needed, then definitely
that can be looked at. So far, what I have seen BPC failing on is a
single directory with a large number of files (ie, 977386 currently
which has not succeeded in some time, unfortunately, the application
requires all files in a single directory). However, scaling the
requirements with the hardware is not a bad goal, even better if minimal
hardware can backup any target with the only effect being a longer time
to complete. Do we *really* need the entire list of files in RAM? Isn't
that the point of the newer rsync which doesn't need to pre-load the
entire list of files? Can't we just process "one file at a time" with a
look-ahead of 1000 files (seems to be what rsync does already)?

I expect this type of work to be a lot more complicated/involved. Don't
expect a lot of help, as few people are going to be interested in this
goal. Developers tend to scratch their own itches...

Post by Michael
*** Regarding BPC being slow
I only give BPC 256MB, so I should expect too much regarding performance.
That I fully agree. However when I say it is slow, I mean it is slow even
if I take into account that fact. Transfer speed is ok-ish; it uses rsync
at its best, which requires some heavy processing sometimes. But I don't
understand why it needs so much time for the remaining tasks (ref
counting, etc). I'm actually convinced (perhaps naively) that this can be
significantly improved. See further down.

I agree, I think v4 is doing refcounts that are not needed. I saw a
email recently (last few months) noting that v4 will refcount *all*
backups for the host, instead of only the backup that was just
completed. The current host I'm working on takes hours just for the
refcnt after a backup, this also likely involves a LOT of random I/O as
well.

Post by Michael
*** Regarding "trashing" rsync
My statement was... too brutal I guess ;-) I fully agree with Stephen's
comment. And I don't want to create a new program from scratch. rsync is
one of the best open-source sync program. Unison and duplicity are
basically using rsync internally. I do think however that BPC can be
significantly improved.

I think what I'd like to see here is the ability to add some
"intelligence" to the client side. Whether we need a BPC client or not,
I'm not sure, but currently BPC doesn't seem to "continue" a backup
well. Some cloud sync apps seem better at doing small incremental
uploads which eventually will get a consistent confirmed backup. This
includes backing up really large single files, BPC will discard and
re-start the file, instead of knowing that "this" is only half the file,
and that it should continue on the next backup.
1) Consider a file 1GB in size, on the first time BPC sees the file, it
starts a full transfer and manages to download 300MB before a network
hiccup, or timeout happens. BPC can add the file to the pool, and save
the file into the partial backup, marking the file as incomplete.
However, lets say a disaster strikes, isn't it better to recover the
first 300MB of the file rather than nothing?

2) In the scenario of BPC has a complete/valid backup of this 1GB file,
but it has changed. BPC/rsync starts to transfer the file, we complete
the changes in the first 300MB of the file before the same network
hiccup/timeout. Again, why not keep the file, and mark as incomplete?
Next time, rsync will quickly skip the first 300MB, and continue the
backup of the rest of the file. In a disaster, you have the choice to
restore the incomplete file from the partial backup, or the complete
file from the previous backup, or both and then forensically
examine/deal with the differences to potentially recover a bunch of data
you may not otherwise have had access to.

Post by Michael
- Flatten the backup hierarchy
Initially BPC was a "mere" wrapper around rsync. First duplicating a
complete hierarchy with hard-links, then rsync'ing over it. It had the
advantage of simplicity but is very slow to maintain, and impossible to
duplicate. Now the trend in 4.0alpha was to move to a custom C
implementation of rsync, where hierarchy only stores attrib files. I
think that we can improve the maintenance phase further (ref counting,
backup deletion...) by flattening this structure into a single linear
file, and by listing once for all the references in a given backup,
possibly with caching of references per directory. Directory entries
would be more like git objects, attaching a name to a reference along
with some metadata. This means integrating further with the inner
working of rsync. It would be fully compliant with rsync from the client
side. But refcounting and backup deletion should then be equivalent
to sorting and finding duplicate/unique entries, which can be very
efficient. Even on my Lacie sorting a 600k-line file with 32B random
hash entries takes only a couple seconds.

Wouldn't that require loading all 600k lines into memory? What if you
had 100 million entries or 100 billion? I think by the time you get to
looking at that, you are better off using a proper DB to store that
data, they are much better designed to handle sorting/random access of
data that some flat text file.

This might be something better looked at in BPC v5, as it's likely to be
a fairly large achitectural change. I'd need to read a lot more about
the v4 specific on-disk formats to comment further...

Post by Michael
- Client-side sync
Sure, this must be an optional feature, and I agree this is not the
priority. Many clients will still simply run rsyncd or rsync/ssh. But
the client-side sync would allow to detect hard links more efficiently.
It will also decrease memory usage on the server (see rsync faq). Then
it opens up a whole new set of optimization, delta-diff on multiple
files...

Yes, also, it mostly works well as simply "another" BPC protocol that
can sit alongside tar/rsync/smb/etc... However, finding the developers
to work on this, and then maintain it in the long term? A *nix client
may not be so difficult, but a windows client might be more useful but
harder.... A definite project all by itself!

Post by Michael
*** Regarding writing in C
Ok, I'm not a perl fan. But I agree, it is useful for stuff where
performance does not matter, for website interface, etc. But I would
rewrite in C the ref counting part and similar.

I suppose the question is how much performance improvement will this get
you? It is possible to embed C within perl, and possible to pre-compile
a perl script into a standalone executable. So certainly re-writing
sections in C is not impossible. For example, you want to re-write the
ref counting part, I suspect this is mostly disk I/O constrained rather
than CPU/code constrained, so I doubt you would see any real performance
improvement. I expect the best way to improve performance on this part
is to improve/fix the algorithm, and then translate that improvement
into the code.

eg, as was reported (by someone else that I can't recall right now) if
you have 100 backups saved for this host, and you finish a new backup
(whether completed or partial) then you redo the refcnt for all 101
backups. If this was changed to only redo the refcnt for the current
backup, then you are 100 times faster. Better than any improvement by
changing the language.

Finally, I wonder whether we will have more (or less) people able (and
actually doing it) to contribute code if it is written in perl or C? I
suspect the pragmatic process will be to keep it in perl and just patch
the things needed. Over time, some performance reliant components could
be re-written into C and embedded into the existing perl system.
Eventually, the final step could be taken to convert the remaining
portions into C.

Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
BackupPC-***@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[François]

On Tuesday, 12 January 2016, Adam Goryachev <

Post by Adam Goryachev
However, lets say a disaster strikes, isn't it better to
recover the first 300MB of the file rather than nothing?

I don't agree. 1Gb is most likely q binqry file which would be corrupted
if a part is missing. But anyway, I kind of like the idea of partial
transfer.

[Les Mikesell]

Post by François
On Tuesday, 12 January 2016, Adam Goryachev

Post by Adam Goryachev
However, lets say a disaster strikes, isn't it better to
recover the first 300MB of the file rather than nothing?

I don't agree. 1Gb is most likely q binqry file which would be corrupted if
a part is missing. But anyway, I kind of like the idea of partial transfer.

Rsync itself does have the option to save partial transfers for
restarts so it might not be that hard to add, but then you add
overhead to look for the fragments and clean them up.

[Les Mikesell]

Post by Michael
Of course, the main problem is the small amount of memory. Looking
in the dmesg logs, I could spot regularly OOM messages, the kernel killing
the backuppc_dump process, etc. Now it is a bit unfair of me blaming BPC
when actually the main culprit is the lack of memory. But the thing is
that BPC is quite unhelpful in these situations.

Not sure anything is helpful - or can be - in the OOM-killer
situation. The process in question doesn't get much of a chance to
tell you what happened.

Post by Michael
Server logs are mostly
useless, no timestamps, and there is no attempt to restart again, but
instead BPC goes into a long process of counting references, etc, meaning
most of the server time is spent in (apparently) unproductive tasks.

On a suitable platform, Linux tends to be reliable so it's not
surprising that programmers don't spend a lot of time dealing with
cases that shouldn't happen.

Post by Michael
Initially BPC was a "mere" wrapper around rsync. First duplicating a
complete hierarchy with hard-links, then rsync'ing over it. It had the
advantage of simplicity but is very slow to maintain, and impossible to
duplicate.

No, v3 has rsync completely implemented in perl so that it can
maintain the archive copies compressed while chatting block checksums
against a native remote rsync reading uncompressed files. And it
only handled the the older rsync protocol that required the entire
directory to be transferred first and held in RAM for the duration of
the run. Given the way perl stores variables, this isn't pretty, but
then again RAM is cheap.

Post by Michael
Now the trend in 4.0alpha was to move to a custom C
implementation of rsync, where hierarchy only stores attrib files. I
think that we can improve the maintenance phase further (ref counting,
backup deletion...) by flattening this structure into a single linear
file, and by listing once for all the references in a given backup,
possibly with caching of references per directory. Directory entries
would be more like git objects, attaching a name to a reference along
with some metadata.

Git does some interesting things - but I'm not all that convinced
checking in a whole machine would be a win compared to what rsync
does.

Post by Michael
This means integrating further with the inner
working of rsync. It would be fully compliant with rsync from the client
side. But refcounting and backup deletion should then be equivalent
to sorting and finding duplicate/unique entries, which can be very
efficient. Even on my Lacie sorting a 600k-line file with 32B random
hash entries takes only a couple seconds.

That kind of boils down to a question of how much work you want to do
to save a few dollars worth of RAM. Or even another box to do the
work for you over NFS or iscsi to your storage server.

Post by Michael
- Client-side sync
Sure, this must be an optional feature, and I agree this is not the
priority. Many clients will still simply run rsyncd or rsync/ssh. But
the client-side sync would allow to detect hard links more efficiently.
It will also decrease memory usage on the server (see rsync faq). Then
it opens up a whole new set of optimization, delta-diff on multiple
files...

I've always considered it one of the main attractions of BPC that it
does not require any client side setup beyond ssh keys which you
normally need anyway.

Post by Michael
*** Regarding writing in C
Ok, I'm not a perl fan. But I agree, it is useful for stuff where
performance does not matter, for website interface, etc. But I would
rewrite in C the ref counting part and similar.

It's not 'performance' that is bad for many/most things where you are
dealing with network and disk activity. It just needs (much) more
RAM. And on most platforms that is easy to accommodate. That's not
to say it can't be improved, but you are going to trade expensive
human time to save a bit of cheap hardware.
--
Les Mikesell
***@gmail.com

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
BackupPC-***@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/