[BackupPC-devel] v4.0alpha2 backup error

[Steve Palm]

Continuing testing...

One windows client was backed up OK, but on another I received this error.. Looks like something on the BackupPC server side, a crash in tarExtract ? Or is it a system problem ?

Just posting in case it needs looking into, if this is better put in a bug tracker (is there one?) let me know.

.
.
.
new d 755 0/0 *** glibc detected *** /usr/bin/perl: realloc(): invalid next size: 0x093bd438 ***
tarExtract: ======= Backtrace: =========
tarExtract: /lib/libc.so.6[0x190f11]
tarExtract: /lib/libc.so.6(realloc+0xe6)[0x192ea6]
tarExtract: /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so(Perl_safesysrealloc+0x47)[0x36dd87]
tarExtract: /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so(Perl_sv_grow+0x73)[0x3890c3]
tarExtract: /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so(Perl_sv_setsv_flags+0x60c)[0x392b7c]
tarExtract: /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so(Perl_pp_aassign+0x1a8)[0x3843d8]
tarExtract: /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so(Perl_runops_standard+0x1f)[0x37c9df]
tarExtract: /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so(perl_run+0x2ee)[0x32224e]
tarExtract: /usr/bin/perl(main+0x13e)[0x804921e]
tarExtract: /lib/libc.so.6(__libc_start_main+0xdc)[0x13aebc]
tarExtract: /usr/bin/perl[0x8049021]
tarExtract: ======= Memory map: ========
tarExtract: 00106000-00121000 r-xp 00000000 fd:00 4554776 /lib/ld-2.5.so
tarExtract: 00121000-00122000 r-xp 0001a000 fd:00 4554776 /lib/ld-2.5.so
tarExtract: 00122000-00123000 rwxp 0001b000 fd:00 4554776 /lib/ld-2.5.so
tarExtract: 00125000-0027c000 r-xp 00000000 fd:00 4554788 /lib/libc-2.5.so
tarExtract: 0027c000-0027e000 r-xp 00156000 fd:00 4554788 /lib/libc-2.5.so
tarExtract: 0027e000-0027f000 rwxp 00158000 fd:00 4554788 /lib/libc-2.5.so
tarExtract: 0027f000-00282000 rwxp 0027f000 00:00 0
tarExtract: 00284000-002ab000 r-xp 00000000 fd:00 4554993 /lib/libm-2.5.so
tarExtract: 002ab000-002ac000 r-xp 00026000 fd:00 4554993 /lib/libm-2.5.so
tarExtract: 002ac000-002ad000 rwxp 00027000 fd:00 4554993 /lib/libm-2.5.so
tarExtract: 002af000-002b2000 r-xp 00000000 fd:00 4554978 /lib/libdl-2.5.so
tarExtract: 002b2000-002b3000 r-xp 00002000 fd:00 4554978 /lib/libdl-2.5.so
tarExtract: 002b3000-002b4000 rwxp 00003000 fd:00 4554978 /lib/libdl-2.5.so
tarExtract: 002b4000-002bc000 r-xp 00000000 fd:00 12747484 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Encode/Encode.so
tarExtract: 002bc000-002bd000 rwxp 00007000 fd:00 12747484 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Encode/Encode.so
tarExtract: 002bd000-002c2000 r-xp 00000000 fd:00 12750525 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Socket/Socket.so
tarExtract: 002c2000-002c3000 rwxp 00004000 fd:00 12750525 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Socket/Socket.so
tarExtract: 002c3000-002c5000 r-xp 00000000 fd:00 13344994 /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi/auto/IO/Dirent/Dirent.so
tarExtract: 002c5000-002c6000 rwxp 00001000 fd:00 13344994 /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi/auto/IO/Dirent/Dirent.so
tarExtract: 002cb000-002e1000 r-xp 00000000 fd:00 4554842 /lib/libpthread-2.5.so
tarExtract: 002e1000-002e2000 r-xp 00015000 fd:00 4554842 /lib/libpthread-2.5.so
tarExtract: 002e2000-002e3000 rwxp 00016000 fd:00 4554842 /lib/libpthread-2.5.so
tarExtract: 002e3000-002e5000 rwxp 002e3000 00:00 0
tarExtract: 002e5000-002e6000 r-xp 002e5000 00:00 0 [vdso]
tarExtract: 002e6000-002ee000 r-xp 00000000 fd:00 12749472 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Data/Dumper/Dumper.so
tarExtract: 002ee000-002ef000 rwxp 00007000 fd:00 12749472 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Data/Dumper/Dumper.so
tarExtract: 002ef000-002f9000 r-xp 00000000 fd:00 4554830 /lib/libnss_files-2.5.so
tarExtract: 002f9000-002fa000 r-xp 00009000 fd:00 4554830 /lib/libnss_files-2.5.so
tarExtract: 002fa000-002fb000 rwxp 0000a000 fd:00 4554830 /lib/libnss_files-2.5.so
tarExtract: 002fb000-00426000 r-xp 00000000 fd:00 12747455 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
tarExtract: 00426000-0042b000 rwxp 0012a000 fd:00 12747455 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
tarExtract: 0042b000-0042d000 rwxp 0042b000 00:00 0
tarExtract: 0042d000-00438000 r-xp 00000000 fd:00 4554910 /lib/libgcc_s-4.1.2-20080825.so.1
tarExtract: 00438000-00439000 rwxp 0000a000 fd:00 4554910 /lib/libgcc_s-4.1.2-20080825.so.1
tarExtract: 00518000-0051a000 r-xp 00000000 fd:00 12747452 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Cwd/Cwd.so
tarExtract: 0051a000-0051b000 rwxp 00001000 fd:00 12747452 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Cwd/Cwd.so
tarExtract: 0051b000-0054f000 r-xp 00000000 fd:00 13271271 /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi/auto/BackupPC/XS/XS.so
tarExtract: 0054f000-00551000 rwxp 00033000 fd:00 13271271 /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi/auto/BackupPC/XS/XS.so
tarExtract: 00551000-006bd000 rwxp 00551000 00:00 0
tarExtract: 008a8000-008ab000 r-xp 00000000 fd:00 12747467 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Digest/MD5/MD5.so
tarExtract: 008ab000-008ac000 rwxp 00003000 fd:00 12747467 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Digest/MD5/MD5.so
tarExtract: 00a55000-00a58000 r-xp 00000000 fd:00 12747496 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Fcntl/Fcntl.so
tarExtract: 00a58000-00a59000 rwxp 00002000 fd:00 12747496 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Fcntl/Fcntl.so
tarExtract: 00bdb000-00bf0000 r-xp 00000000 fd:00 4554976 /lib/libnsl-2.5.so
tarExtract: 00bf0000-00bf1000 r-xp 00014000 fd:00 4554976 /lib/libnsl-2.5.so
tarExtract: 00bf1000-00bf2000 rwxp 00015000 fd:00 4554976 /lib/libnsl-2.5.so
tarExtract: 00bf2000-00bf4000 rwxp 00bf2000 00:00 0
tarExtract: 00bfc000-00c0d000 r-xp 00000000 fd:00 4554996 /lib/libresolv-2.5.so
tarExtract: 00c0d000-00c0e000 r-xp 00010000 fd:00 4554996 /lib/libresolv-2.5.so
tarExtract: 00c0e000-00c0f000 rwxp 00011000 fd:00 4554996 /lib/libresolv-2.5.so
tarExtract: 00c0f000-00c11000 rwxp 00c0f000 00:00 0
tarExtract: 00d8d000-00da0000 r-xp 00000000 fd:00 12749485 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Storable/Storable.so
tarExtract: 00da0000-00da1000 rwxp 00012000 fd:00 12749485 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Storable/Storable.so
tarExtract: 00dc2000-00dda000 r-xp 00000000 fd:00 12747481 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Encode/Byte/Byte.so
tarExtract: 00dda000-00dec000 rwxp 00018000 fd:00 12747481 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Encode/Byte/Byte.so
tarExtract: 0211e000-02127000 r-xp 00000000 fd:00 4554981 /lib/libcrypt-2.5.so
tarExtract: 02127000-02128000 r-xp 00008000 fd:00 4554981 /lib/libcrypt-2.5.so
tarExtract: 02128000-02129000 rwxp 00009000 fd:00 4554981 /lib/libcrypt-2.5.so
tarExtract: 02129000-02150000 rwxp 02129000 00:00 0
tarExtract: 07fd4000-07fd6000 r-xp 00000000 fd:00 4557132 /lib/libutil-2.5.so
tarExtract: 07fd6000-07fd7000 r-xp 00001000 fd:00 4557132 /lib/libutil-2.5.so
tarExtract: 07fd7000-07fd8000 rwxp 00002000 fd:00 4557132 /lib/libutil-2.5.so
tarExtract: 08048000-0804b000 r-xp 00000000 fd:00 12677530 /usr/bin/perl
tarExtract: 0804b000-0804c000 rw-p 00002000 fd:00 12677530 /usr/bin/perl
tarExtract: 090bf000-09cf0000 rw-p 090bf000 00:00 0 [heap]
tarExtract: b4ab3000-b4bb4000 rw-p b4ab3000 00:00 0
tarExtract: b4cb5000-b4db6000 rw-p b4cb5000 00:00 0
tarExtract: b5400000-b5421000 rw-p b5400000 00:00 0
tarExtract: b5421000-b5500000 ---p b5421000 00:00 0
tarExtract: b55b7000-b7dd9000 rw-p b55b7000 00:00 0
tarExtract: b7dd9000-b7fd9000 r--p 00000000 fd:00 12662314 /usr/lib/locale/locale-archive
tarExtract: b7fd9000-b7fdc000 rw-p b7fd9000 00:00 0
tarExtract: b7fec000-b7fed000 rw-p b7fec000 00:00 0
tarExtract: bfe65000-bfef5000 rw-p bff6e000 00:00 0 [stack]
Error writing to tar file - Connection reset by peer
Didn't get entire file. size=963584, nread=65520
BackupPC_tarExtract exited with fail status 6
Xfer PIDs are now
Got fatal error during xfer (BackupPC_tarExtract exited with fail status 6)
Backup aborted (BackupPC_tarExtract exited with fail status 6)
BackupFailCleanup: nFilesTotal = 0, type = full, BackupCase = 1, inPlace = 1, lastBkupNum =
Removing empty backup #0
BackupPC_backupDelete: removing #0
BackupPC_backupDelete: No prior backup for merge
BackupPC_backupDelete: got 0 errors
Running BackupPC_refCountUpdate -h win2k8vm -f on win2k8vm
xferPids 11676
BackupPC_refCountUpdate: host win2k8vm got 0 errors
BackupPC_refCountPrint: total errors: 0
xferPids
Finished BackupPC_refCountUpdate (running time: 0 sec)

[Craig Barratt]

Steve,

This looks like some sort of memory or freelist corruption in
BackupPC_tarExtract:

*** glibc detected *** /usr/bin/perl: realloc(): invalid next size:
0x093bd438 ***


This error is deep inside perl, and normally shouldn't happen when you are
running any sort of perl code. But... a dynamically linked module like
BackupPC::XS could be a culprit. I did run tests on the underlying library
using the address sanitizer checks in LLVM, and it was clean. But maybe I
didn't have enough test cases.

The challenge is to replicate this on my machine (or get me access to a
machine that shows the problem). I'm guessing it is subtly dependent on
the exact tar file being piped into BackupPC_tarExtract from the client.

The good new is the error appears to happen very early in the backup, which
makes debugging a bit easier. Can you set the XferLOG level to say 9 and
send me the whole XferLOG file? (Don't copy it to the list.)

Craig