Multi-line triggers

Posted by Shadis on Sun 22 Jun 2008 04:35 PM — 17 posts, 53,573 views.

Shadis Sun 22 Jun 2008 04:35 PM #0

I'm fairly new to MUSHclient (Great program! I love it!). I've been trying to do something farily ambitious with parsing all of the text that comes from a MUD (which is wrapped in XML tags) and returning it, parsed and modified in some cases, to the output window. I've gotten a lot of it working, but I'm having some problems with multi-line triggers. Here's an example of the data that I'm trying to match:

<s-new-loc loc-num='2082' name='Shipyard' w='2081'>   This is a desolate location.  There are no ships here ...
   Siryn&apos;s laws will allow spaceship repairs here only, to reduce the impact on the environment ...
   The only way out is to the landing pad to the west.</s-new-loc>

The indented lines above are where the newlines occur. (Edit: Whoops! Sorry for the wide screen. Though I guess that shows that we're really lookin at multiple lines. The whole thing shows up as about 12 lines on my output widow. I've trimmed the first two lines to be shorter).

More generically, it will be in the form of: "<s-sometag optional='arguments'>Some multiline data</s-sometag>"

What I'm trying to do is pass this data to an XML parser that will strip off the tags, save any options, and return the data to the output window.

Unfortunately, one of the problems that I have is that I don't know how many lines I'll get. If I do a WHO command, I could get 150 lines.

I think I need one of two things: Either a good way to recombine the multiple lines back into one data structure that I can pass to my XML parser (I tried just concatenating the strings together, but the parser choked), or a way to trigger on a start tag, send the following data to the parser, and stop sending when it reaches an end tag. Is there any way to say "Once I match <s-something>, grab everything until </s-something> and then process it all"? Or even to just continually grab all of the data from the MUD and process it every time we see an end tag?

I realize this is probably something that MUSHclient wasn't originally intended to do, and if it's beyond the scope of the client I understand. But if it's at all possible, I'd love to be able to get it to work.

Oh, and the parser that I'm using is in Python, but I'm also familiar with VBScript and could possibly use that. And the parser works fine if I hand it a single line complete with both tags. In fact, my workaround right now is to first detect a few single-line cases, and then after that I will detect a start tag by itself, and then wrap that with an end tag and process is, or an end tag by itself, and wrap that with a start tag and process it. Unfortunately, in the case of 3 or more lines, the middle lines don't get processed.

Any ideas or help would be greatly appreciated.

Thanks!

Shadis Sun 22 Jun 2008 04:43 PM #1

Whoops, sorry, almost forgot the trigger that I'm using right now:

<triggers>
  <trigger
   enabled="y"
   lines_to_match="25"
   match="^\<(.*?)\>(.*?)\<\/(.*?)\>$"
   multi_line="y"
   name="multiline"
   regexp="y"
   script="xml_multiline"
   sequence="99"
  >
  </trigger>
</triggers>

It's the one autogenerated by MUSHclient when I put in "<*>*</*>". I'm not very good with regex yet, sorry.

Shadis Sun 22 Jun 2008 06:04 PM #2

Ok, I've pretty much determined that it's a problem in my Python code, I guess. I have changed my triggers so that they pass everything to the script, and I've scripted it so that what I end up with is one big block of text wrapped in XML tags, and the XML parser (I'm using xml.sax.make_parser()) doesn't want to handle it. I think what's happening is that my code to convert the string into an object isn't liking it for some reason. (I'm just doing cStringIO.StringIO(line)).

Anyway, so, it's definitely not a MUSHclient problem. If anyone has any ideas, I'd still love to hear them, but at this point I'll just keep looking for better ways to handle my data in my script.

Shadis Sun 22 Jun 2008 06:23 PM #3

Well THAT was easy! haha

In the parser that I'd written, I was taking the string data, and passing it to a separate function that converted it to an object. The function just did cStringIO.StringIO(line). It worked for single lines but not for multiple lines. But for some reason, when I took out the call to the function and just put the cStringIO.StringIO(line) directly in my XML parser, it worked.

So, just for in formational purposes, here's roughly what I did:

I set up 4 separate triggers. One that matches single tags (in the form <tag options="option"/>). The next one matches end tags (like </s-sometag>). The third one that fires looks for start tags (like <s-sometag>) and the last one catches everything else, as it will fall in the middle of a set of tags (all of the MUD data is enclosed in some sort of tag).

My script then checks to see what kind of tag it's gotten. If it's a single, it processes it. If it's an end, it processes a global string that is holding all of the multiline stuff. If it's a start, it starts a new global string. Everything else gets added to the existing global sting.

When the end string (or a single tag) is processed, it is converted to an object with the CStringIO line, then passed to an XML parser based on xml.sax. This function has some logic to process the data based on the tag, and any options. Most of it is just passed back to the window. Some of it is processed first and then sent back.

The nice thing about this is that I'm now just doing single line triggers, so I can omit all of the data from the output until after I process it.

heh, I really did work on this for several hours yesterday and most of this morning before I came asking for help. I guess just organizing my thoughts on the problem helped me get there. I hope this helps someone in the future. :D

Nick Gammon Australia Forum Administrator Sun 22 Jun 2008 09:35 PM #4

Yes, as you discovered, it is a common general technique (see the FAQ) to make a "start" trigger, then a "catch-all" trigger, then a "stop" trigger. Between them they can catch lots of lines.

Just FYI MUSHclient has an XML parser built into it, accessible from Lua, which is the same parser it uses to parse world files and plugins.

Shadis Sun 22 Jun 2008 10:27 PM #5

Sheesh, is there anything it can't do? heh. Maybe I could get MUSHclient to do my laundry for me. hehe

Well, I've gotten things working pretty satisfactorily, and since I'm trying to learn Python I'll stick with what I've got for now. I may get curious enough to try lua one day. Butright now I know vbscript and some Python so I'll stick with them.

Oh, one thing I noticed. I don't know if there are some timing issues in the Note or ColourNote function, but any time I write over 50 or 60 lines with one, they won't redraw the screen properly. I've worked around it by just putting in a redraw command after the Note. Just didn't know if it was something that should be hapening or not.

Nick Gammon Australia Forum Administrator Mon 23 Jun 2008 04:56 AM #6

It shouldn't be happening however I can't reproduce it. Using Lua:


for i = 1, 100 do
  Note (string.rep (string.char (i + 32), 79))
end

That draws 100 lines without any artifacts that I can see. I am using Windows XP, version 4.28 of MUSHclient.

Worstje Netherlands Mon 23 Jun 2008 05:34 AM #7

I reproduced it easily, Nick.

Python example code:

blah = ""
for i in range(100):
  blah = blah + str(i) + "\n"
world.Note(blah)

Essentially, stuff all the newlines in the same Note() call.

Nick Gammon Australia Forum Administrator Mon 23 Jun 2008 06:16 AM #8

Like this?


s = ""

for i = 1, 100 do
  s = s .. string.rep (string.char (i + 32), 79) .. "\n"
end 

Note (s)

I still don't see it.

Quote:

... they won't redraw the screen properly ...

Can you be more specific? Or show a screendump somewhere?

Is there some output option that affects it?
What font and size are you using?
Which operating system?
Do you have smooth scrolling turned on?

Worstje Netherlands Mon 23 Jun 2008 06:31 AM #9

You can see the effect @ http://qs.merseine.nu/images/redraw_problem.png. That's screencapped right after closing the immediate window, executed with the Lua script you posted just now. I tried it in two worlds, both which use different fonts. One shows the problem, the other does not. So it's font-related. :)

Font WITHOUT issues: Monaco, 9pt.
Font WITH issues: Dina, 10pt. Link: http://www.donationcoder.com/Software/Jibz/Dina/index.html

Besides that, I'm using XP with a visual theme, no smooth scrolling, and I'm not in a position to play with all of my output settings at this moment. But since the font seems to make the crucial difference, I'll hold that off until it's really needed to locate the issue. =)

Nick Gammon Australia Forum Administrator Mon 23 Jun 2008 07:23 AM #10

Well I downloaded Dina, which looks nice I must say, but I don't get the rather bizarre effects you got. I tried 9 pt and 10 pt.

From your screen dump it appears it somehow overwrites what was there previously. I don't quite understand how that happened. I am having to assume that the screen invalidation is not happening properly.

I think we need a barberpole effect, please try this:


s = ""

for i = 1, 100 do
  s = s .. string.format ("%3i: ", i)
  for j = 1, 79 do
    s = s .. string.char (math.fmod (i + j, 90) + 32)
  end -- for j
  s = s .. "\n"
end -- for i

Note (s)

You should see something like this:


  1: "#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnop
  2: #$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopq
  3: $%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
  4: %&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
  5: &'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
  6: '()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
  7: ()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
  8: )*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
  9: *+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
 10: +,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
 11: ,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy 
 12: -./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !
 13: ./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"
 14: /0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#
 15: 0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#$
 16: 123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#$%
 17: 23456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#$%&
 18: 3456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#$%&'
 19: 456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#$%&'(
 20: 56789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy !"#$%&'()

Worstje Netherlands Mon 23 Jun 2008 03:55 PM #11

What is happening is that it does not register properly that the entire window changed, and it only ends up updating the things it actually needs to redraw - the last line and the parts the Immediate window was over. (The later suggests your InvalidateRect() code is doing it's job properly in my eyes.) I am short on time again right now, but I'll play with the code later and see if I can find a barrier of sorts where it triggers/does-not-trigger.

Images of your latest code snippet:
hxxp://qs.merseine.nu/images/redraw_problem2a.png - after executing and closing the Immediate dialog.
hxxp://qs.merseine.nu/images/redraw_problem2b.png - After scrolling a bit so that it has redrawn everything.

It shows the (predictable) effect of forgetting to update some characters as opposed to printing the wrong stuff at the wrong places, but I figured I'd include it none-the-less.

Nick Gammon Australia Forum Administrator Mon 23 Jun 2008 09:29 PM #12

Hmm, your screen isn't set as wide as mine, you might want to change:


for j = 1, 79 do

to something smaller, like:


for j = 1, 70 do

Anyway, I see I am sorting of looking for the wrong thing. What you are basically saying is, that if you draw quickly it simply "forgets" to invalidate everything?

I had been looking for more subtle things, like artifacts under the lines or something.

By the way, what's with the hxxp://xxx? In your original post I assumed that was a typo and changed it to http://xxx, and that worked (once changed), but you are still doing it.

I am checking out the calls to Invalidate, but it would help to narrow things down a bit.

Can you check these things for me please?

If you do Edit menu -> Select All, is everything inverted?
If you scroll back and forwards vigorously with the scroll thumb, are things redrawn correctly?
If not, does it make a difference if you scroll a small amount (ie. less than a screenful) or a large amount (back to previous "pages")?
If you hit Ctrl+End to go to the end of the buffer (from earlier up), does it redraw correctly?
With the script problem, can you compare a script that does less than a screenful (so that some of the existing lines are still there), to one that does more than a screenful (so everything should be redrawn).
If you use the scroll-bar "arrows" (the ones that scroll one line at a time), does that work correctly?
If you have the problem you showed with the screen dump, cover the MUSHclient window with another app, and then uncover it, does that redraw correctly? (That should force an invalidate).

Can you also confirm the version if possible? Which version are you using? Which version, if any, do you know did NOT have this problem? If it always happened I am surprised not to have heard about it earlier.

Nick Gammon Australia Forum Administrator Mon 23 Jun 2008 10:02 PM #13

I think the relevant code in MUSHclient is this, from mushview.cpp:


void CMUSHView::ScrollToPosition ( POINT pt, const bool bSmooth)
  {
CMUSHclientDoc* pDoc = GetDocument();
ASSERT_VALID(pDoc);

SCROLLINFO ScrollInfo;
RECT r;
int iDeltaY = m_scroll_position.y - pt.y;

  RemoveToolTip ();
  GetClientRect (&r);

  // if we can do a smooth scroll, well let's do it!
  if (abs (iDeltaY) < r.bottom)
    {
    
    // very smooth scrolling
    int iSmoothDelta = iDeltaY < 0 ? -1 : 1;

    if (App.m_bSmoothScrolling)
      {
      for (int i = 0; i < abs (iDeltaY); i++)
        {
        m_scroll_position.y -= iSmoothDelta;
        // update scroll bar
        GetScrollInfo (SB_VERT, &ScrollInfo, SIF_POS);
        ScrollInfo.nPos = m_scroll_position.y;
        SetScrollInfo (SB_VERT, &ScrollInfo, SIF_POS);
        ScrollWindow (0, iSmoothDelta);
        UpdateWindow ();
        }
      } // end of smooth scrolling
    else
      {
      ScrollWindow (0, iDeltaY);
      m_scroll_position = pt;
      if (App.m_bSmootherScrolling)
        UpdateWindow ();    // redraw immediately if wanted
      } // end of not smooth scrolling
    }
  else
    // more than a screenfull away? just redraw the whole lot
    {
    Invalidate ();
    }

  m_scroll_position = pt;

  // update scroll bar
  GetScrollInfo (SB_VERT, &ScrollInfo, SIF_POS);
  ScrollInfo.nPos = pt.y;
  SetScrollInfo (SB_VERT, &ScrollInfo, SIF_POS);
 
  if (pDoc->m_bAutoFreeze)
    m_freeze = GetScrollPos (SB_VERT) < GetScrollLimit(SB_VERT);

  } // end of CMUSHView::ScrollToPosition

The important lines, I think, are in bold. First it checks if we are scrolling less than a screenful, in which case it does different code. That calls ScrollWindow which *should* simply scroll existing lines upwards (by simply copying bits), and then invalidate the rest of the window, so it is redrawn.

If it detects that they would be scrolled off the top of the window anyway, it simply invalidates the whole window.

The "very smooth" scrolling does a line at a time, which won't apply if you don't have it turned on, and the "smoother" scrolling forces a redraw each time through, which makes the blank bits get filled immediately, rather than waiting for the next time around the event loop.

Nick Gammon Australia Forum Administrator Mon 23 Jun 2008 10:28 PM #14

Worstje, can you also try, if practical:

Make a new MUSHclient world with default settings. See if it happens there. Then gradually add back in your normal settings (like the new font).
Preferably also uninstall and reinstall MUSHclient, which will clear out the Registry settings, which will be all the global preferences, and then gradually add them back in.
Switch to default Windows XP theme, and then try adding back in the theme you were using.

Worstje Netherlands Tue 24 Jun 2008 04:25 AM #15

.. I'd love to reproduce it, except... it stopped happening for me all of a sudden. I had to restart my computer for some kind of silly program, and right now I can't seem to trigger the bug anymore. I never ran into it prior to this, so it wasn't a problem for me in the first place.. but I'll keep trying over the course of the next few days to see whether I can reproduce it. My buffers tend to be set to gigantic sizes, which might make a difference in triggering it.

First though, I'll go install 4.29. =)

Shadis Sat 28 Jun 2008 05:18 AM #16

I can't really get it to happen again either. It did it once, and when I tried to reproduce it, it wouldn't. I think the reason it's not hapening for me anymore is that I took the code that was displaying the problem out into its own subroutine, and also added a bit of code, so it's not just spamming 80 notes in a row.

I do know that when it was happening, I had just installed 4.28, so it was a fresh install. And I haven't changed anything with my fonts. I use the default fixedsys 9pt font. I'm on Vista, using the default theme. And it didn't happen when I wasn't doing any scripting. So it seems to be exclusively the scripting that's doing it, and only when spamming a ton of notes at once. And only occasionally. heh

Sorry I'm not able to give any better info. Hope that helps.