Nanny Problem?

Posted by Nick Cash on Thu 08 Jan 2004 02:41 AM — 26 posts, 65,199 views.

Nick Cash USA Thu 08 Jan 2004 02:41 AM #0

This is confusing. Let me describe the situation.

I try to connect to the mud, it prompts for my name. I enter my name (Odis), and it prompts for my password. Now, after I enter my password I get the normal Welcome crap and the the standard load messages for my area:

Loading...
Linking exits...
Done.

Now, so far so good. However, right after that I press enter, and BOOM! It crashes. Now...I've tried for a while to figure this out. No, it doesn't generate a core file. No, it doesn't give me any errors. This seems to happen to everyone that trys to get on. If they don't have an area, naturally, it doesn't show the loading area messaged. I've tried placing a few bug statments in certain locations within the nanny function and I get two of three in the logs.
So, after that, I figured, well, there must be a bug in the code I just installed. However, the only thing it really modified was the loading of asteroids (no snippet, custom code here) and the updating of these asteroids (it compiles with no errors btw). So I uninstalled everything to do with asteroids and it still does the same. So I guess its not that.

My one hint to this problem:
SEGMENTATION VIOLATION

Its in all the logs. However, its very unspecific. So, I come to you to tell me where to search. Aparently it is having trouble loading people. Btw, read the next post for the log file information and the places on where I put bug statements in nanny.

Nick Cash USA Thu 08 Jan 2004 02:46 AM #1

From the log file:

Wed Jan 7 19:41:21 2004 :: [*****] BUG: Area Loaded
Wed Jan 7 19:41:36 2004 :: [*****] BUG: Enter Has Been Pressed
Wed Jan 7 19:41:36 2004 :: SEGMENTATION VIOLATION

Corrosponding code in the nanny function:

At the very end of case CON_GET_OLD_PASSWORD:


      if ( ch->pcdata->area )
                do_loadarea (ch , "" );
        bug("Area Loaded");
        break;

At the very top of case CON_PRESS_ENTER:


   case CON_PRESS_ENTER:
          send_to_pager( "\014", ch );

          bug("Enter Has Been Pressed");

Note: those are the two that show up in the logs

Last one: at the very top of CON_READ_MOTD:


case CON_READ_MOTD:
        send_to_desc_color( "&z\n\r\n\r", d );
        add_char( ch );

        bug("Displaying MOTD");

        d->connected    = CON_PLAYING;

Nick Gammon Australia Forum Administrator Thu 08 Jan 2004 03:30 AM #2

Core dump may be suppressed. There was a suggestion recently from Ksilyan, I think, about changing your environment variables to allow that.

Anyway, just run under gdb to find the problem. ie.

cd area
gdb ../src/smaug
run

Then when it crashes type bt to find what call did it.

Nick Gammon Australia Forum Administrator Thu 08 Jan 2004 03:33 AM #3

Has send_to_desc_color been called before this? That doesn't seem to be standard SMAUG. Maybe it has a bug.

Nick Cash USA Thu 08 Jan 2004 03:37 AM #4

#0 0x265e5e62 in ?? ()
Cannot access memory at address 0x265f7026

Err...what exactly does that mean? What line is it that it can't access the memory right?

Thanks for the help...I'm a gdb noob.

Nick Cash USA Thu 08 Jan 2004 03:39 AM #5

Yes, it has, in almost every case statment in the switch it is called to send color to the descriptor. Switching back to write_to_buffer did not seem to help the cause any.

Nick Cash USA Thu 08 Jan 2004 03:48 AM #6

Hmm..I just noticed something. The second bug call (Enter Has Been Pressed) comes right before a bunch of do_help calls. However, the last bug call (Displaying MOTD) is not displayed, so I think its safe to guess its not getting there. Perhaps there is a problem with do_help? Maybe help.dat got corrupted.

Samson USA Thu 08 Jan 2004 03:51 AM #7

send_to_desc_color isn't a standard Smaug color function. It's usually associated with some form of color add-on. One such color add-on is my custom color snippet, which also doubles as a fix for the broken Smaug color system and is in the 3 FUSS packages I distribute.

However, the particular function is not setup to be used anywhere in the FUSS packages and could very well have an unforseen bug in it. This could be the source of the problem. I'm in the process of converting a rewrite of this code someone sent me for AFKMud, which has been tested with the colored descriptor stuff.

What bothers me is that "SEGMENTATION VIOLATION" is showing up in the logs. This suggests to me the codebase being used has a signal handler which is interfering with things. I've been opposed to the interception of SIGSEGV ever sicne I realized how evil doing so can be. It would always produce unusable results. More often than not, people pair it with an auto-copyover thing. The best bet is to just let the code crash, dump its core, and then let the startup script reboot it. Crashes happen. Initiating an auto-copyover is not wise since the crash could very well have come from a corrupt player, or a corrupt descriptor.

So my first bit of advice is to remove the signal trap for SIGSEGV and then let it crash. Chances are you'll get better results and the cause can be narrowed down faster.

Nick Cash USA Thu 08 Jan 2004 04:02 AM #8

Indeed the SIGSEV signals are being intercepted. However, all it does is log the last player command and shutdownt he mud using exit( 1 );, which I think I heard is supposed to generate a core. Now, I've heard lots of bad things about SigSev and why its originally commented out in smaug/swr distributions, however, I found a site that another SWR coder had put up discussing the speculation and giving a fix for a couple things and how to handle it and all of that, so I figured I'd do the same. So far, it has worked just fine, and it is not stopping results except in the fact that it won't generate a core file. Why it does that is something I'm unsure of, and something you guru's can probably tell me. I am running off of SWR 1.0 FUSS, so it does have the color snippet you wrote Samson. I do not think the problem is with send_to_desc_color. When I did get a core file it just told me the same error:

#0 0x265e5e62 in ?? ()
Cannot access memory at address 0x265f7026

Anyways, if you really think I should, I will comment out SIGSEV again.

Thanks for the help, time for me to go sleep.

Samson USA Thu 08 Jan 2004 04:12 AM #9

Doing an exit(1) won't generate a core. exit() is a valid system call which literally tells the mud to stop running. When it does so, it shuts down normally.

You may be able to make it work by calling abort() instead, but as my past experience tells me, intercepting SIGSEGV just doesn't work as well as one might like. I spent the better part of several weeks chasing my tail with an error relating to descritpors and players and the lastcommand variable before I realized the signal handler was trying to act on data that itself had become corrupted. It never dropped core files.

Given that you are using the FUSS package, and that there was a bug in our color code that caused crashes with NULL characters, it's high probablility this is what's happening.

Now I don't know what all the guy's site had to say on the issue, but I think he got lucky it hasn't caused him problems thus far. In any case, "Cannot access memory at address 0x265f7026" isn't enough to go on. Would need to see the backtrace info to see what calls led to that.

David Haley USA Thu 08 Jan 2004 04:13 AM #10

Turn off the signal handler (comment out the line that registers it.) Run it using GDB as Nick suggested, then it'll show you precisely what went wrong. Backtrace if necessary to see the call stack (list of functions that have been called to reach this point.)

To backtrace: "bt"
To go "up" the stack frame: "up"
To go "down" ... "down"

Those basic commands are enough to let you navigate through the stack frame.

At any time in GDB you can just type "help" and it'll give you a list of topics you may need help in; but they are a bit obscure unless you know how to use a debugger.

Nick Cash USA Thu 08 Jan 2004 04:35 AM #11

Thanks for the gdb help. Anythoughts on the problem at hand? It doesn't seem to be a help.are problem...

David Haley USA Thu 08 Jan 2004 05:20 AM #12

With the backtrace, it'll be trivial to see where the problem is... without it, personally I have no clue given what you posted. :)

It could be below CON_PRESS_ENTER, below the bit that does the log... but again, I cannot stress how much easier things will be with a backtrace. :)

Nick Gammon Australia Forum Administrator Thu 08 Jan 2004 05:20 AM #13

Virtually any bug could cause a segmentation fault.

Having the signal handler simply confuses the debug, because the core dump, if it was generated, which it isn't, would be from the signal handler, not the *real* problem place.

Turn off the SIGSEV handler, and then report what the backtrace says. Anything else right now is just guessing.

Nick Cash USA Thu 08 Jan 2004 10:09 PM #14

Sorry for not being clear. After I commented it out it gave the same back trace:

#0 0x265e5e62 in ?? ()
Cannot access memory at address 0x265f7026

David Haley USA Thu 08 Jan 2004 11:49 PM #15

Did you run it through GDB? Not analyze the core, but actually run the program with GDB? (as Nick Gammon said to do)

What exact line did you comment out?

Nick Cash USA Fri 09 Jan 2004 02:09 AM #16

Yes, both the core and running through the program gave that EXACT error. This is the line I ended up commenting out:

Line and surround lines included:


    signal( SIGPIPE, SIG_IGN );
    signal( SIGALRM, caught_alarm );
/*  signal( SIGSEGV, SegVio ); */
    signal( SIGTERM, SigTerm ); /* Catch kill signals */

David Haley USA Fri 09 Jan 2004 02:35 AM #17

Did you recompile? (dumb question but I have to ask)

What is the code below the last successful log?

Did you type in "bt" in GDB to get the full backtrace, not just the last function call?

Are you compiling with debug information enabled?

Nick Cash USA Fri 09 Jan 2004 04:58 AM #18

Yes, I recompiled. (I'm not that new :P)

Yes, I typed bt to get the full back trace. It was the only thing in the stack.

And no, I am not compiling with debug enabled, I've never used it.

Code containing the last successful bug statement:


  case CON_PRESS_ENTER:
          send_to_pager( "\014", ch );

        bug("Enter Has Been Pressed");

        if ( IS_IMMORTAL(ch) )
        {
          send_to_pager( "&WImmortal Message of the Day&w\n\r", ch );
          do_help( ch, "imotd" );
        }
        if ( ch->top_level > 0 )
        {
          send_to_pager( "\n\r&WMessage of the Day&w\n\r", ch );
          do_help( ch, "motd" );
        }
        if ( ch->top_level >= 600)
        {
          send_to_pager( "\n\r&WAvatar Message of the Day&w\n\r\n\r", ch );
          do_help( ch, "amotd" );
        }
        if ( ch->top_level == 0 )
          do_help( ch, "nmotd" );
        send_to_pager( "\n\r&WPress [ENTER] &Y", ch );

        d->connected = CON_READ_MOTD;
        break;

Like I said before, maybe its a problem with do_help?

Thanks for the help.

Nick Gammon Australia Forum Administrator Fri 09 Jan 2004 06:13 AM #19

It seems odd you don't get a backtrace, but maybe the bug is wiping out the stack.

Compile with debug, I don't recall the option, but it is something like -G 3 (do a man gcc to find it).

Then run under gdb, setting a breakpoint at the last point where you think it works (eg. the last successful message). Then step a line at a time until it crashes.

David Haley USA Fri 09 Jan 2004 06:40 AM #20

I think it's -g3, but as Nick said go check man gcc.

If you're not compiling with debug info, that may be why you don't have any. :P

If you still can't get the stack trace or gdb to work, you can just do this the brute force method. After every function call (every single one, I'm not kidding) in CON_GET_ENTER, add a new log... "Got here #1", "Got here #2" for instance. Look for the last "Got here" and then see what is called immediately after that.

Also put those in CON_GET_MOTD if you make it that far.

This is very similar to the "printf" method. Primitive, but effective. :)

Here's something else you might need to do. Did you make clean and then remake from scratch? If you modified a header file, but did not do a full recompile, it can get broken somewhere. This is not likely to be the case, but it's something to check.
(What happens is that the compiler stores memory address of the struct members. If you change the order of the members but don't recompile all files that access these members, you'll be accessing the wrong ones. This is a pretty complicated concept, though, so the safest rule is to always recompile everything if you change a structure in mud.h)

Nick Gammon Australia Forum Administrator Fri 09 Jan 2004 10:54 PM #21

Does the log get flushed to disk after every write?

I think the gdb method should work.

Nick Cash USA Sat 10 Jan 2004 03:57 AM #22

Hmm. I'll get right to your way Ksilyan here in a bit, trying some other stuff. I just wanted to note something. I just removed all relevent code and support code to the new thing I installed and there was absolutely no change. Just thought I'd note that fact so you don't send me scouring through my asteroid code that I have not yet had the chance to test looking for a bug I wouldn't know of yet. Anyways, on to logging each case. Post back in a few.

Nick Cash USA Sat 10 Jan 2004 05:36 AM #23

I actually think I found the problem, having to do with a notify snippet I installed. If this is really the problem I'm going to kick myself in the face...seriously...

Nick Cash USA Sat 10 Jan 2004 05:51 AM #24

Alright, well, I've at least cleared the notify code of blame. I replaced the entire nanny function with a copy of one from a backup I made just a little under a month ago, with no changes, and it worked fine for the backup. However...for some reason I think I may have corrupted my code or something.

The backup I unzipped and tested works perfectly. Going to transfer some of the source files to the backup and see what happens.

Nick Cash USA Sat 10 Jan 2004 06:08 AM #25

Ok, sorry for so much trouble, and to my standards this is by no means a fix. As to a failure to perform correctly (due to corrupt files or something similar) I transfered the import files that had the changes in them to a backup that was just taken before it started messing up. Now, at least for the moment, the login seems to be working correctly and the mud is responding well to the new installation. Sorry to waste your time. If it does it again I'll spend more time finding the real issue, however, I do think that some of my files were corrupted some how.

Thanks for your effort, lets hope I don't need to renew this post.