Did you compile with debugging enabled? See my post about gdb for details. Also ensure you remove the stuff inside SMAUG that catches segmentation faults (see the same post).
Before waiting 3 days, simulate a crash (make a command dereference nil or something) and check that you are catching it.
Could the segfault be in a thread or attached process, perhaps? It's possible that if it's not your process that's dying, gdb won't catch it properly. That being said, if it crashes enough to kill the program, it should crash enough to make gdb catch it...
Are you using threads/attached processes in the first place? You're using MySQL, right? Maybe it's an issue with that...
If you are running from inside gdb, then it won't generate a backtrace upon a segfault, you need to tell it to do that afterward. You would still be within a live run of the game at that point.
Of course if the crash was bad enough to kill the debugger too, that's going to be difficult to track down unless you know how to make it happen.
Try adding some simple log statements (or print statements and redirect the output) to the latest code additions. Print out the values of things. Chances are you have a corrupt pointer somewhere.
If this started happening since the Arthmoor upgrades, I'd make sure that you recompiled [i]every single object file[/i] that you have to make sure that everything is linking against the proper libraries etc. It's possible that if you have old libraries, something ever so slightly incompatible exists in some obscure location.
I haven't upgraded anything. Could you explain (or link to) details about threads? I'm pretty sure there was a mention of threads when I originally ran the exec with gdb which I've never seen.
My understanding is that gdb doesn't behave very well with threads. I had several class projects that we had to debug the old-school, old-fashioned way by hand because gdb couldn't work with the threads. Apparently, if one thread crashes it doesn't always bring you to the thread that crashed, but it'll interrupt the "main thread", or something odd like that.
It's possible that it's just the version of gdb we were using, but I've heard from several people in several places that gdb doesn't behave too well. Unfortunately I don't have any articles or manuals lying around that would explain it.
It's just very strange that it doesn't seem to do anything at all. You say that when you run it without gdb, it doesn't crash?
Maybe what you could do is add log entries to all kinds of places all over the code, like when the update function is run etc. That might at least help track down what part of the code is dying -- that is getting closer to the old-fashioned way of debugging things. :-)
Nah, it'll crash after ~3 days with or without gdb attached.
This is what I meant by seeing threads:
(gdb) run
Starting program: /home/zeno/chub/src/chub
[Thread debugging using libthread_db enabled]
[New Thread 16384 (LWP 26181)]
Now if I do Ctrl+C:
Program received signal SIGINT, Interrupt.
[Switching to Thread 16384 (LWP 26181)]
0x401723c2 in select () from /lib/libc.so.6
(gdb) bt
#0 0x401723c2 in select () from /lib/libc.so.6
#1 0x00000006 in ?? ()
#2 0x00000000 in ?? ()
#3 0xbffff710 in ?? ()
#4 0x0804953c in game_loop (control=0) at socket.c:227
#5 0x08049294 in main (argc=1, argv=0xbffff794) at socket.c:79
It looks like when you continue, a new thread is launched that wasn't there before, and it didn't stop the old one. Can't figure out why that would be, though.
What libraries are you linking with? Any idea what shared libraries it's loading on startup?
A crash that does not produce helpful information can often be caused by a badly corrupted stack. If the stack itself is wiped out, then doing a backtrace can be a problem.
Wiping the stack isn't that hard, overwriting a local variable can easily do it (that is, memset or strcpy into more space than was allocated).
Nick's right about the local variables being smashed. Valgrind would help you with that.
As for pthread, it should be easy to find out where you're using it... just grep for 'pthread'. (And I actually meant what you're linking with at link time i.e. what -l flags you send to g++ at link phase.)
OK, it looks like you create a thread called "thread_lookup"; I imagine you are using that to do the DNS lookups so that the whole MUD doesn't hang if the DNS is slow? That's something I've been meaning to do for a while.
Anyhow, have you observed any correlation at all between someone connecting and the MUD crashing a short time later?
It's weird that it's happening so regularly after 3 days. Any chance of an overflow somewhere on a timer?
As for hostnames: it's possible that the feature doesn't quite work. Or that you have to type 'users' to see the hostnames, because it's in the process of resolving them. But the point of the feature is to prevent the entire MUD from hanging if the DNS lookup hangs. Generally that happens if the DNS resolver is having issues.
The memory quota? I have this installed on my local machine. Is there really a memory quota included, or is it normally set by the admin? Because I haven't set it.
Definitely looks like time for "old school" debugging. Lots of log messages sprinkled about. Start general. Log message at the beginning of game_loop, in between the descriptor input and the descriptor output, beginning and end of the update_handler, and the end of game_loop. Then when it just up and dies, if you haven't killed the thing with 3 days worth of log spam, you can begin to narrow down the search with log messages in more specific parts. Pain in the ass, but some bugs just demand this kind of attention.