bpb4crash

(BreakPoint Before Crash)

This little script allows a buggy program to be stopped just before it crashes, at the start of the function that crashes (or optionally, in any other function in a call stack).

This is useful in the following circumstances:

• When you have a crashing program, and
• it crashes on a large input, that makes reverse-execution too slow, and
• the crashing function is called multiple times (so you can just simply set a plain breakpoint on it).

NOTE: It only works for programs that are both deterministic and non-interactive!

Dependencies

• gdb
• mkfifo
• POSIX shell (e.g. Dash, Bash, ...)

How it works

It first runs the program under gdb, using the GDB/MI interface, to analyze how many times the failing function runs in total (before the crash happens).

Then starts gdb normally, but with a pre-set breakpoint with an ignore count such that it stops just before the crash.

From there you can inspect the function arguments, step through the function, etc.

Usage

By default, it stops at start of the function where the crash or assertion failure happens:

$ ./bpb4crash.sh ./test_crash

You can also override which function in should stop in:

$ TRACKED_FUNC=crash_loop ./bpb4crash.sh ./test_crash

If you want to stop not a the last call before the crash, but at an earlier call, then you can overrides this too:

$ STEPS_BEFORE=2 ./bpb4crash.sh ./test_crash

Example use case

The test_crash.c file has an assertion failure, but it can happen in several cases:

void check_number()
{
    if (num == -90) goto fail;
    if (num == 0)   goto fail;
    if (num == 200) goto fail;
    return;
  fail:
    num = -1;
    assert(0);
}

When running it, we can't know which number triggered the assertion failure:

$ ./test_crash
test_crash: test_crash.c:35: check_number: Assertion `0' failed.
Aborted

And gdb isn't very helpful either:

$ gdb ./test_crash
Reading symbols from ./test_crash...
(gdb) run
Starting program: /home/privmisc/code/beforecrash/test_crash 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
test_crash: test_crash.c:35: check_number: Assertion `0' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, 
    no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44  ./nptl/pthread_kill.c: Filen eller katalogen finns inte.
(gdb) f 6
#6  0x0000555555555197 in check_number () at test_crash.c:35
35      assert(0);
(gdb) p num
$1 = -1

That doesn't tell anything at all :(

But we can use the bpb4crash to get a breakpoint at the last call to this function, just before the assertion failure happened.

$ ./bpb4crash.sh ./test_crash
First run (analysis)...
test_crash: test_crash.c:35: check_number: Assertion `0' failed.
#6  0x0000555555555197 in check_number () at test_crash.c:35
#7  0x00005555555551ba in crash_loop (i=123) at test_crash.c:42
#8  0x00005555555551ed in intermediate_func (i=123) at test_crash.c:49
#9  0x0000555555555209 in main (argc=1, argv=0x7fffffffe298) at test_crash.c:55
Second run (to breakpoint)...
test_crash: test_crash.c:35: check_number: Assertion `0' failed.
Reading symbols from ./test_crash...
Breakpoint 1 at 0x113d: file test_crash.c, line 29.
Will ignore next 122 crossings of breakpoint 1.
Starting program: /home/privmisc/code/beforecrash/test_crash 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, check_number () at test_crash.c:29
29      if (num == -90) goto fail;

From here we can see the state at the beginning of the function call:

(gdb) p num
$1 = 0