[BackupPC-devel] [patch][4.0.0alpha3] switch to cmdSystemOrEvalLong in BackupPC_rrdUpdate

[Alexander Moisseev]

Switch to cmdSystemOrEvalLong from cmdSystemOrEval when getting stdout because sub cmdSystemOrEval merges stdout and stderr together.

BackupPC_rrdUpdate creates broken images if HOME environment variable set to path non-readable by backuppc user (e.g. HOME=/root).
BackupPC_rrdUpdate writes stderr warnings in image files:
"(process:2430): Pango-WARNING **: error opening config file '/root/.config/pango/pangorc': Permission denied".


diff -ruN ./BackupPC_rrdUpdate.orig ./BackupPC_rrdUpdate
--- ./BackupPC_rrdUpdate.orig 2013-12-09 17:06:45.000000000 +0400
+++ ./BackupPC_rrdUpdate 2013-12-10 11:22:05.000000000 +0400
@@ -140,7 +140,7 @@
#
# Get each pool max value from RRD
#
- $bpc->cmdSystemOrEval(
+ $bpc->cmdSystemOrEvalLong(
[
$Conf{RrdToolPath},
"graphv", "-",
@@ -158,7 +158,7 @@
if ( $_[0] =~ /^print\[([0-3])\] = "([.0-9]+)"$/ ) {
$poolMax[$1] = $2 unless ( $2 == 0 );
}
- });
+ }, 0, undef);

my $poolSizeGraph = [
"$Conf{RrdToolPath}",
@@ -228,6 +228,6 @@
print("Can't open/create $LogDir/poolUsage$weeks.png\n");
return;
}
- $bpc->cmdSystemOrEval($poolSizeGraph, sub { print $fdOut $_[0] });
+ $bpc->cmdSystemOrEvalLong($poolSizeGraph, sub { print $fdOut $_[0] }, 1, undef);
close($fdOut);
}

--
Alexander

[Craig Barratt]

Alexander,

Thanks for the patches.

Also, I didn't merge in your code (rrd_2_v4.pl) to upgrade any V3 rrd file.
I wanted to do that without requiring the user to install RRD::Simple.
However, I couldn't see how to do the conversion using the command-line
rrdtool.

Craig

Post by Alexander Moisseev
Switch to cmdSystemOrEvalLong from cmdSystemOrEval when getting stdout
because sub cmdSystemOrEval merges stdout and stderr together.
BackupPC_rrdUpdate creates broken images if HOME environment variable set
to path non-readable by backuppc user (e.g. HOME=/root).
"(process:2430): Pango-WARNING **: error opening config file
'/root/.config/pango/pangorc': Permission denied".
diff -ruN ./BackupPC_rrdUpdate.orig ./BackupPC_rrdUpdate
--- ./BackupPC_rrdUpdate.orig 2013-12-09 17:06:45.000000000 +0400
+++ ./BackupPC_rrdUpdate 2013-12-10 11:22:05.000000000 +0400
@@ -140,7 +140,7 @@
#
# Get each pool max value from RRD
#
- $bpc->cmdSystemOrEval(
+ $bpc->cmdSystemOrEvalLong(
[
$Conf{RrdToolPath},
"graphv", "-",
@@ -158,7 +158,7 @@
if ( $_[0] =~ /^print\[([0-3])\] = "([.0-9]+)"$/ ) {
$poolMax[$1] = $2 unless ( $2 == 0 );
}
- });
+ }, 0, undef);
my $poolSizeGraph = [
"$Conf{RrdToolPath}",
@@ -228,6 +228,6 @@
print("Can't open/create $LogDir/poolUsage$weeks.png\n");
return;
}
- $bpc->cmdSystemOrEval($poolSizeGraph, sub { print $fdOut $_[0] });
+ $bpc->cmdSystemOrEvalLong($poolSizeGraph, sub { print $fdOut $_[0] }, 1, undef);
close($fdOut);
}
--
Alexander
------------------------------------------------------------------------------
Sponsored by Intel(R) XDK
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
_______________________________________________
BackupPC-devel mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-devel
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[Alexander Moisseev]

Post by Craig Barratt
Alexander,
Thanks for the patches.
Also, I didn't merge in your code (rrd_2_v4.pl <http://rrd_2_v4.pl>) to upgrade any V3 rrd file. I wanted to do that without requiring the user to install RRD::Simple. However, I couldn't see how to do the conversion using the command-line rrdtool.
Craig

The only way I know: convert database to xml by rrddump, make changes in xml file and convert it back to the RRD by rrdrestore.

--
Alexander

[Alexander Moisseev]

Post by Alexander Moisseev

Post by Craig Barratt
Alexander,
Thanks for the patches.
Also, I didn't merge in your code (rrd_2_v4.pl <http://rrd_2_v4.pl>) to upgrade any V3 rrd file. I wanted to do that without requiring the user to install RRD::Simple. However, I couldn't see how to do the conversion using the command-line rrdtool.
Craig

The only way I know: convert database to xml by rrddump, make changes in xml file and convert it back to the RRD by rrdrestore.

Craig,

By the way, you change the CF type of an RRA from AVERAGE in V3 to LAST in V4alpha3.
I think that conversion of the CF type of RRA is very hard (or may be impossible).

--
Alexander

[Tyler J. Wagner]

Post by Alexander Moisseev
By the way, you change the CF type of an RRA from AVERAGE in V3 to LAST in V4alpha3.
I think that conversion of the CF type of RRA is very hard (or may be impossible).

It is not possible. If you consolidate the data as AVERAGE, the LAST data
is lost. You can begin consolidating new data as LAST, but the older data
will show the AVERAGE data (and you'll mistakenly think it is LAST). This
isn't a big issue, as AVERAGE and LAST are usually similar. It's a big
problem when you want to keep MIN/MAX data and it's not available.

Generally speaking, once you start using rrdtool, converting your data is a
pain. Get it right when you design it, or expect that you'll have to toss
it out some day.

However, RRDs are small!* There is no need to throw away the AVERAGE data.
We can store both AVERAGE and LAST (and MIN and MAX if we want) in the same
RRD file. Most Cacti installs keep 2-4 sets of consolidated data in this way.

* OK, not always small, but at least static in size.

Regards,
Tyler

[Alexander Moisseev]

Post by Tyler J. Wagner

Post by Alexander Moisseev
By the way, you change the CF type of an RRA from AVERAGE in V3 to LAST in V4alpha3.
I think that conversion of the CF type of RRA is very hard (or may be impossible).

It is not possible. If you consolidate the data as AVERAGE, the LAST data
is lost. You can begin consolidating new data as LAST, but the older data
will show the AVERAGE data (and you'll mistakenly think it is LAST). This
isn't a big issue, as AVERAGE and LAST are usually similar. It's a big
problem when you want to keep MIN/MAX data and it's not available.

I meant "very hard (or may be impossible)" with available tools. In our case consolidation interval equal to 1, therefore consolidated data numerically equal regardless of CF. We can't convert database on the fly with rrdtool or some known Perl module, but we can dump database, edit as needed and restore back (or fetch data, create new database and fill it with old data) without data loss.

Post by Tyler J. Wagner
However, RRDs are small!* There is no need to throw away the AVERAGE data.
We can store both AVERAGE and LAST (and MIN and MAX if we want) in the same
RRD file. Most Cacti installs keep 2-4 sets of consolidated data in this way.

Adding new RRA or DS to existing RRD relatively easy, but what to do next? Hypothetically (I don't really know), we can draw graphs of both old and new RRA on same picture, and may be some how calculate current/average/maximum value for a legend.

When I wrote the code, I decided to add new DSes and leave RRA format as is to avoid database conversion. Since we about to change RRD format, I propose discuss it.

I suppose we would add separate RRA for 52 week graph to properly consolidate data. I am not sure about CF. AVERAGE or MAX? Another way is to use DEF with :reduce=<CF> option (consolidation on the fly).

Craig, you create RRA with 732 data points. Do you have some intentions about usage of 2 year data in the future?


About V3 to V4 rrd upgrade script. Craig, I guess you want to include it in configure.pl? Please, don't do it. Nowadays many users install software from packages rather then source or ports. In case of package, configure.pl runs on machine where package builds (e.g. OS vendor build cluster) at package build time. So package always contain config files with default options. It's OK for new installations, but if user doing an upgrade he/she must edit configuration files by hand or install BackupPC from sources/port (or build himself package on production server then install).
Ideally instead of single configure.pl we need separate install and postinstall scripts. Postinstall script should do all upgrade procedures: make changes in configuration files, database conversion and so on.

--
Alexander

[Alexander Moisseev]

Post by Alexander Moisseev
@@ -158,7 +158,7 @@
if ( $_[0] =~ /^print\[([0-3])\] = "([.0-9]+)"$/ ) {
$poolMax[$1] = $2 unless ( $2 == 0 );
}
- });
+ }, 0, undef);
my $poolSizeGraph = [
"$Conf{RrdToolPath}",

I found typo in the patch. $ignoreStderr needs to be 1.

@@ -158,7 +158,7 @@
if ( $_[0] =~ /^print\[([0-3])\] = "([.0-9]+)"$/ ) {
$poolMax[$1] = $2 unless ( $2 == 0 );
}
- });
+ }, 1, undef);

my $poolSizeGraph = [
"$Conf{RrdToolPath}",