Debugging Segmentation Faults in Mono

I have been doing this way too much this afternoon so I figured I would write up a quick blog post on how to do it both so I remember how to do it 🙂 and for anyone that comes across this needing to do it.

Sometimes you might get a nasty failure out of mono that looks something like this:

mono() [0x4b8008]
mono() [0x50ff9b]
mono() [0x424322]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfbb0) [0x7f4ccf1edbb0]
mono() [0x5fc8e7]
mono() [0x5fdec3]
mono() [0x5d9aff]
mono() [0x5df36e]
mono() [0x5df839]
mono() [0x5f5dd9]
mono() [0x5f5fe3]
[0x4116b7f9]

Thanks makes me a

sad_panda

 

 

 

 

 

 

Not much information is given but you can get much more information if you use gdb.

To start with let’s bring up gdb

greg@goblin:~/src/EventStore/bin/$ gdb mono

Now mono does some weird things internally with some signals so we will need to ignore some of them

(gdb) handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint

Signal Stop Print Pass to program Description
SIGXCPU No No Yes CPU time limit exceeded
SIGPWR No No Yes Power fail/restart
SIG33 No No Yes Real-time event 33
SIG35 No No Yes Real-time event 35

now run the program that you want to run.

(gdb) run –debug YourProgram.exe –some –parameters=3

Your program is now running. At some point in the future it will die and drop you back to a gdb prompt this is where things get interesting. Your prompt might looks something like this:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed5f2700 (LWP 16643)]
0x00000000005e1270 in alloc_obj (vtable=0x7ffff65312a0, size=-324507488,
pinned=0, has_references=1) at sgen-marksweep.c:740
740 int size_index = MS_BLOCK_OBJ_SIZE_INDEX (size);
(gdb)

OK now we can get some more information. Want to know what this thread is doing? Try backtrace

(gdb) backtrace
#0 0x00000000005e1270 in alloc_obj (vtable=0x7ffff65312a0, size=-324507488,
pinned=0, has_references=1) at sgen-marksweep.c:740
#1 0x00000000005fb5f4 in alloc_for_promotion (has_references=1,
objsize=3970459808, obj=0x7ffff6531148 “\24022S\366\377\177”,
vtable=0x7ffff65312a0) at sgen-simple-nursery.c:35
#2 copy_object_no_checks (obj=obj@entry=0x7ffff6531148,
queue=queue@entry=0x983120 <gray_queue>) at sgen-copy-object.h:112
#3 0x00000000005fc382 in simple_nursery_serial_copy_object_from_obj (
queue=0x983120 <gray_queue>, obj_slot=0x7fffc8277e10)
at sgen-minor-copy-object.h:206
#4 simple_nursery_serial_scan_object (start=<optimized out>,
queue=0x983120 <gray_queue>) at sgen-scan-object.h:64
#5 0x00000000005d8a6f in sgen_drain_gray_stack (max_objs=max_objs@entry=-1,
ctx=…) at sgen-gc.c:1194
#6 0x00000000005de27e in collect_nursery (unpin_queue=unpin_queue@entry=0x0,
finish_up_concurrent_mark=finish_up_concurrent_mark@entry=0)
at sgen-gc.c:2631
#7 0x00000000005de749 in collect_nursery (finish_up_concurrent_mark=0,
unpin_queue=0x0) at sgen-gc.c:3547
#8 sgen_perform_collection (requested_size=4096, generation_to_collect=0,
reason=0x70b51a “Nursery full”, wait_to_finish=0) at sgen-gc.c:3483
#9 0x00000000005f4b49 in mono_gc_alloc_obj_nolock (
vtable=vtable@entry=0xa1ee88, size=size@entry=568) at sgen-alloc.c:288
#10 0x00000000005f4e04 in mono_gc_alloc_string (vtable=0xa1ee88, size=568,
len=270) at sgen-alloc.c:563
#11 0x0000000040021059 in ?? ()
#12 0x00007fffa8002540 in ?? ()
#13 0x00007ffff67aeef0 in ?? ()
#14 0x00007ffff67f5950 in ?? ()
#15 0x0000000000000238 in ?? ()
#16 0x00007fffed5f1070 in ?? ()
#17 0x00007ffff67f5718 in ?? ()
#18 0x00007fffed5f26f0 in ?? ()
#19 0x0000000000a1ee88 in ?? ()
#20 0x000000000000010e in ?? ()
#21 0x0000000040016514 in ?? ()
#22 0x000000000000002b in ?? ()
#23 0x00007fffed5f1390 in ?? ()
#24 0x00007ffff67aeef0 in ?? ()
#25 0x00007ffff67aeef0 in ?? ()
#26 0x00007ffff67aecc8 in ?? ()
#27 0x00007ffff67f54d8 in ?? ()
#28 0x00007fffc5390068 in ?? ()
#29 0x00007ffff43b6a56 in string:CreateString (this=<optimized out>,
val=0x10e) at <unknown>:2907
#30 0x000000004002aa73 in ?? ()
#31 0x0000000000000000 in ?? ()
(gdb)

ouchies its the garbage collector crashing. Yeah guess we will be looking at that for a while. You may notice that in your backtrace only unmanaged calls show up not the mono calls. You can resolve the mono calls with mono_pmip (also a useful mono_backtrace function here http://www.mono-project.com/Debugging#Debugging_with_GDB)

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: