[BackupPC-devel] BackupPC v3 and v4 Development

Discussion:

Adam Goryachev

2016-01-06 23:25:48 UTC

Hi,

I've been a long time user of backuppc (couple of years at least), and
in general it works really well and I'm mostly happy with the current
status. However, since I upgraded to the 4.0.0alpha3 last year, I've had
a number of minor issues (some more serious than others, like failing to
backup unchanged files, or saying the backup has failed even though it
succeeded). So far, I've not lost data due to any issue, and that is a
plus, but I'm very concerned that eventually, one of these problems will
cause actual data loss (as in, backup failed, something else caused data
loss like failed RAID array, and then can't recover from backup).

I'd like to know if there is any current person or organisation doing
development work on BackupPC, and/or interested in doing that? I'm
considering to fork the project, and try to debug/fix the remaining
issues in BPC 4, but at the same time, I'm very busy, and am not really
a "proper" coder, so working on such a large project will be difficult.

With the right group of developers, this could work (as in, a small work
load for each person, but at least better maintenance/development
efforts). My concerns are:
1) Without ongoing development/maintenance, new versions of OS or perl
or whatever will cause breakages, while manual/minor patches or config
changes might solve these, over time it will become more of a nightmare.
2) The point of using a "standard" open source product is that we all
get the advantage of experience (ie, more users finding problems), and
improvements/patches. I could have built (probably never as good as the
current BPC) my own solution.

So, are you interested in developing/contributing?
What is the current status/plans around BPC?
Do you have any patches that are not applied to either v3 or v4 releases?
Thoughts/discussions?

PS, BPC is an excellent product, and I greatly appreciate all the time
and effort that has been invested into it, I would ideally like to see
it continue under the leadership of Craig, he has done an amazing
development job so far. I really really do not want to see it basically
waste away, with people moving to other products simply because it is
unmaintained, and has a few small problems (which is where I currently
stand, either I move to another product, or I start working harder on
the current one).

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

Alexander Moisseev

2016-01-07 08:52:15 UTC

Permalink

Post by Adam Goryachev
So, are you interested in developing/contributing?

People on the list now and then are asking about repository where they could contribute. I believe a few people will be involved if such a repository will be created and maintained.

I am maintaining sysutils/backuppc and sysutils/backuppc-devel FreeBSD ports. At some point I realized I need a repository just to keep tracking patches between Craig's timeouts (your concern #1). Of course I am rather interested in contributing to a community repository (your concern #2).

I am not a "proper" coder and busy too. I'd like to contributing on occasional basis.

Actually, I am using v3 and not considering move to v4 while it is unmaintained. I think with v3 it will be easier to fix on my own potential breakages related to external changes (OS, Perl, ...).
Also, as far as I know policies of some OS prohibit including nonstable (alpha, beta) software in distribution. So a lot of users will be stick to v3 for a long time.

Post by Adam Goryachev
What is the current status/plans around BPC?

Unmaintained/abandoned - I am really sorry, but no any feedback from developer more than a year.

Post by Adam Goryachev
Do you have any patches that are not applied to either v3 or v4 releases?

So, some patches are there:

https://github.com/moisseev/BackupPC

This test script illustrates potential unexpected loss of a few backups after modifying BackupPC settings:

https://gist.github.com/moisseev/d5a8a499a7b69b1f0428

--
Alexander

Joe Bordes

2016-01-07 09:42:49 UTC

Permalink

Hi,

Happy New Year to all :-)

I am willing to help and think that this is a good idea. I am totally
convinced that this project deserves it: @Craig, you have done an
incredible job.

I am a "proper" programmer and very busy too.

I am using V4 for my own personal use and we have all our official
company and production backups on V3. I would prefer to push V4 forward.

I will help and support as best I can.

Joe
TSolucio

Post by Alexander Moisseev

Post by Adam Goryachev
So, are you interested in developing/contributing?

People on the list now and then are asking about repository where they could contribute. I believe a few people will be involved if such a repository will be created and maintained.
I am maintaining sysutils/backuppc and sysutils/backuppc-devel FreeBSD ports. At some point I realized I need a repository just to keep tracking patches between Craig's timeouts (your concern #1). Of course I am rather interested in contributing to a community repository (your concern #2).
I am not a "proper" coder and busy too. I'd like to contributing on occasional basis.
Actually, I am using v3 and not considering move to v4 while it is unmaintained. I think with v3 it will be easier to fix on my own potential breakages related to external changes (OS, Perl, ...).
Also, as far as I know policies of some OS prohibit including nonstable (alpha, beta) software in distribution. So a lot of users will be stick to v3 for a long time.

Post by Adam Goryachev
What is the current status/plans around BPC?

Unmaintained/abandoned - I am really sorry, but no any feedback from developer more than a year.

Post by Adam Goryachev
Do you have any patches that are not applied to either v3 or v4 releases?

https://github.com/moisseev/BackupPC
https://gist.github.com/moisseev/d5a8a499a7b69b1f0428
--
Alexander
------------------------------------------------------------------------------
_______________________________________________
BackupPC-devel mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

--
Un saludo
Joe
TSolucio

François

2016-01-07 10:55:39 UTC

Permalink

On 7 January 2016 at 00:25, Adam Goryachev

Post by Adam Goryachev
Hi,

Hi,

Post by Adam Goryachev
I've been a long time user of backuppc (couple of years at least), and
in general it works really well and I'm mostly happy with the current
status. However, since I upgraded to the 4.0.0alpha3 last year, I've had
a number of minor issues (some more serious than others, like failing to
backup unchanged files, or saying the backup has failed even though it
succeeded).

I've been using version 3 for years, and obviously it just works.
Altough it has performances issues, so I've been considering using v4.
I'm not the one who setup that, so I'm not fluent with backuppc.
However, I was told that version 4 was interesting.
That's why I've created repositories on Github to host the code,
because I couldn't find official dvcs.

- https://github.com/fser/BackupPC
- https://github.com/fser/rsync-bpc
- https://github.com/fser/BackupPC-XS
- and a last one : https://github.com/fser/backuppc-debian-package

My first issue was to get .deb packages, and IIRC, one of those 3 was
an issue for automating. But someone on github did the work to achieve
deb packages. I had to have a look, but didn't find time yet, and now,
I've lost the project issue.

Post by Adam Goryachev
So, are you interested in developing/contributing?
What is the current status/plans around BPC?
Do you have any patches that are not applied to either v3 or v4 releases?
Thoughts/discussions?

So what I mean is yes, I'm interrested but like all of you I don't
have much time. As you may have understood, I'm more into v4 than v3,
which is already available in the wild so I think it would be more
interesting to have packages and automation to let people easily use
backuppc4.

btw I don't remember if I did the same call here that you did, but I'm
glad it makes things move along!

--
François

------------------------------------------------------------------------------

Michael

2016-01-10 00:25:27 UTC

Permalink

Hello,

I've been testing BackupPC 4.0.0alpha3 for 1 year now, for backing up 12
home machines, and to be honest, I'm quite unhappy with it.
To my opinion, it is completely unreliable, you have to regularly check
whether backups are done correctly, and most of the time you can't do a
backup without at least an error. And it's awfully slow. The big
advantage of BPC (besides being free and open-source of course) is to
manage backup of multiple machines in a single pool, hence saving space.

My current backup pool is ~ 12 machine. 11 on Linux and 1 windows
machine. My backup machine is a 3TB Lacie-Cloudbox, with 256 MB memory.
Some of you might say that 256 MB is not enough. Actually I've even seen
posts on the net saying that you would need a server with several GB
RAM. This is just insane. A typical PC in my pool has ~600k files.
Representing each of them with a 256-bit hash, that's basically 20MB of
data to manage for each backup. Of course you need some metadata, etc,
but I see no reason why you need GB of memory to manage that.

If I would participate to the development of BPC, I would make more
changes to the architecture. I think that the changes from 3.0 to 4.0
are very promising, but not enough. The first thing to do is to trash
rsync/rsyncd and use a client-side sync mechanism (like unison). Then
throw away all Perl code and rewrite in C. Also add a timestamp to log
files because debugging BPC failures without timestamps is just a f***
nightmare. And finally make it much more reliable and resistant to
connection issues or interrupt.

What I like in BPC:
- Mutualization of backups in a single pool
- Clean interface
- Free and open-source!

What I hate in BPC:
- BPC seemingly spending more time in backupref_count, fsck or whatever
than in doing actual file transfer.
- Seeing "rsync: read error: Connection reset by peer" in my client log,
followed by even more fsck whatever on the server for ages.
- Not resiliant to interruption, making it very inefficient and unreliable.
- no timestamps in server logs!
- Mostly unhelpful logs.

What I would love to see in BPC:
- Possibility to move the processing (delta) to the client.
- More efficient maintenance, less overhead processing.
- Flawless execution on a 256MB memory server.

Some ideas:
- Use client-side sync and delta detection mechanism (like unison or
duplicity)
- Use ZFS

My gripes and wishes for 2016
Michaël

Post by Adam Goryachev
Hi,
I've been a long time user of backuppc (couple of years at least), and
in general it works really well and I'm mostly happy with the current
status. However, since I upgraded to the 4.0.0alpha3 last year, I've had
a number of minor issues (some more serious than others, like failing to
backup unchanged files, or saying the backup has failed even though it
succeeded). So far, I've not lost data due to any issue, and that is a
plus, but I'm very concerned that eventually, one of these problems will
cause actual data loss (as in, backup failed, something else caused data
loss like failed RAID array, and then can't recover from backup).
I'd like to know if there is any current person or organisation doing
development work on BackupPC, and/or interested in doing that? I'm
considering to fork the project, and try to debug/fix the remaining
issues in BPC 4, but at the same time, I'm very busy, and am not really
a "proper" coder, so working on such a large project will be difficult.
With the right group of developers, this could work (as in, a small work
load for each person, but at least better maintenance/development
1) Without ongoing development/maintenance, new versions of OS or perl
or whatever will cause breakages, while manual/minor patches or config
changes might solve these, over time it will become more of a nightmare.
2) The point of using a "standard" open source product is that we all
get the advantage of experience (ie, more users finding problems), and
improvements/patches. I could have built (probably never as good as the
current BPC) my own solution.
So, are you interested in developing/contributing?
What is the current status/plans around BPC?
Do you have any patches that are not applied to either v3 or v4 releases?
Thoughts/discussions?
PS, BPC is an excellent product, and I greatly appreciate all the time
and effort that has been invested into it, I would ideally like to see
it continue under the leadership of Craig, he has done an amazing
development job so far. I really really do not want to see it basically
waste away, with people moving to other products simply because it is
unmaintained, and has a few small problems (which is where I currently
stand, either I move to another product, or I start working harder on
the current one).
Regards,
Adam

b***@kosowsky.org

2016-01-10 03:37:08 UTC

Permalink

I have been using it for almost 10 years without problems... it's
reliable and stable as anything. Just because other solutions don't
tell you they have errors, doesn't mean there aren't errors... they
just ignore them...

Post by Michael
If I would participate to the development of BPC, I would make more
changes to the architecture. I think that the changes from 3.0 to 4.0
are very promising, but not enough. The first thing to do is to trash
rsync/rsyncd and use a client-side sync mechanism (like unison). Then
throw away all Perl code and rewrite in C. Also add a timestamp to log
files because debugging BPC failures without timestamps is just a f***
nightmare. And finally make it much more reliable and resistant to
connection issues or interrupt.

i.e., I want to create a totally new backup program using a totally
different language and methodology...

Great... nothing is stopping you from creating your own backup
program, but I doubt you will find much willingness here to create a
completely different backup program from scratch...

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
BackupPC-***@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Les Mikesell

2016-01-10 05:38:15 UTC

Permalink

Post by Michael
My current backup pool is ~ 12 machine. 11 on Linux and 1 windows
machine. My backup machine is a 3TB Lacie-Cloudbox, with 256 MB memory.
Some of you might say that 256 MB is not enough. Actually I've even seen
posts on the net saying that you would need a server with several GB
RAM. This is just insane. A typical PC in my pool has ~600k files.
Representing each of them with a 256-bit hash, that's basically 20MB of
data to manage for each backup. Of course you need some metadata, etc,
but I see no reason why you need GB of memory to manage that.

And yet your complaint is that your server is slow... With a
reasonable amount of RAM, much of the directory structure, inodes, and
the next parts of files currently being read will already be in cache
when you need them and writes will be substantially buffered.
Otherwise you'll wait for the disk head to bounce around and always be
in the wrong place.
--
Les Mikesell
***@gmail.com

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
BackupPC-***@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Joe Bordes

2016-01-10 11:32:07 UTC

Permalink

Hi all,

I think that we shouldn't get into exact development details so soon.
Our first decision is to see if it is viable to continue dedicating time
to this project or not, if there are enough people or interest. Once we
decide that we can discuss how to go about it: version to start from,
problems, enhancements,...

I see a clear need for it but I would like to know Craig's position.

@Craig, you are the creator of the product so I would like to know your
position before doing anything. I understand that you have moved on to
other things and have stopped dedicating time to BackupPC, but,

Is that a definite situation?
Do you plan on continuing the development in the future?
Can we count on you if we decide to continue ourselves?
What do you think about the future of BackupPC and how would you
continue if you had the time/resources to do so?

I would also like to set a time out on those questions. Craig hasn't
been responsive in the past months, so If, let's say, at the end of this
week (15th Jan) Craig hasn't answered I would propose to start and email
vote on the version to fork.

I suppose we all agree on managing the project on github, so our next
step would be to decide who starts that project and after that I would
go for a, more or less, agreed list of things to do.

From there it should be easy to just follow the issues and pull
requests and we can discuss how we should make some noise on social
networks or similar to create awareness about the project.

I am open for discussion
Have a nice day :-)
Joe
TSolucio

Post by Michael
Hello,
I've been testing BackupPC 4.0.0alpha3 for 1 year now, for backing up 12
home machines, and to be honest, I'm quite unhappy with it.
To my opinion, it is completely unreliable, you have to regularly check
whether backups are done correctly, and most of the time you can't do a
backup without at least an error. And it's awfully slow. The big
advantage of BPC (besides being free and open-source of course) is to
manage backup of multiple machines in a single pool, hence saving space.
My current backup pool is ~ 12 machine. 11 on Linux and 1 windows
machine. My backup machine is a 3TB Lacie-Cloudbox, with 256 MB memory.
Some of you might say that 256 MB is not enough. Actually I've even seen
posts on the net saying that you would need a server with several GB
RAM. This is just insane. A typical PC in my pool has ~600k files.
Representing each of them with a 256-bit hash, that's basically 20MB of
data to manage for each backup. Of course you need some metadata, etc,
but I see no reason why you need GB of memory to manage that.
If I would participate to the development of BPC, I would make more
changes to the architecture. I think that the changes from 3.0 to 4.0
are very promising, but not enough. The first thing to do is to trash
rsync/rsyncd and use a client-side sync mechanism (like unison). Then
throw away all Perl code and rewrite in C. Also add a timestamp to log
files because debugging BPC failures without timestamps is just a f***
nightmare. And finally make it much more reliable and resistant to
connection issues or interrupt.
- Mutualization of backups in a single pool
- Clean interface
- Free and open-source!
- BPC seemingly spending more time in backupref_count, fsck or whatever
than in doing actual file transfer.
- Seeing "rsync: read error: Connection reset by peer" in my client log,
followed by even more fsck whatever on the server for ages.
- Not resiliant to interruption, making it very inefficient and unreliable.
- no timestamps in server logs!
- Mostly unhelpful logs.
- Possibility to move the processing (delta) to the client.
- More efficient maintenance, less overhead processing.
- Flawless execution on a 256MB memory server.
- Use client-side sync and delta detection mechanism (like unison or
duplicity)
- Use ZFS
My gripes and wishes for 2016
Michaël

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
BackupPC-devel mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/