ansi 16 after ansi 256 ignores bold?

Posted by Fiendish on Wed 04 May 2011 08:05 AM — 22 posts, 88,045 views.

USA Global Moderator #0
If I execute:

Simulate(ANSI(38,5,226).."aaa"..ANSI(31,1).."bbb")
Simulate(ANSI(33,1).."ccc"..ANSI(31,1).."ddd")

aaa and ccc are the same colors (as expected), but bbb and ddd are not. bbb is not showing up as bold even though I have ,1 in there. Why?
USA #1
Oh, how interesting. MUSHclient has a bad case of order-sensitivity. Try ANSI(1,31) on for size.

Also Nick, I just noticed that AnsiNote() doesn't interpret xterm-256 color codes properly. It's reading it as a normal sequence, interpreting the 5 as the italic flag.
Australia Forum Administrator #2
Twisol said:

Also Nick, I just noticed that AnsiNote() doesn't interpret xterm-256 color codes properly. It's reading it as a normal sequence, interpreting the 5 as the italic flag.


Template:function=AnsiNote
AnsiNote

The documentation for the AnsiNote script function is available online. It is also in the MUSHclient help file.



From there:

Quote:

The ANSI sequences defined in the ANSI function will be recognised.


It is behaving as documented.
Australia Forum Administrator #3
Twisol said:

MUSHclient has a bad case of order-sensitivity. Try ANSI(1,31) on for size.


Very helpful clue, thank you.

I can explain it, the tricky bit is to fix it.

If you look at the line info for that line:


Line 390 (390), Thursday, May 05, 6:59:08 AM
 Flags = End para: no, Note: no, User input: no, Log: no, Bookmark: no
 Length = 12, last space = -1
 Text = "aaabbbcccddd"

4 style runs

1: Offset = 0, Length = 3, Text = "aaa"
 No action.
 Flags = Hilite: no, Underline: no, Blink: no, Inverse: no, Changed: no
 Foreground colour RGB   : R=255, G=255, B=0
 Background colour RGB   : R=0, G=0, B=0

2: Offset = 3, Length = 3, Text = "bbb"
 No action.
 Flags = Hilite: YES, Underline: no, Blink: no, Inverse: no, Changed: no
 Foreground colour RGB   : R=128, G=0, B=0
 Background colour RGB   : R=0, G=0, B=0

3: Offset = 6, Length = 3, Text = "ccc"
 No action.
 Flags = Hilite: YES, Underline: no, Blink: no, Inverse: no, Changed: no
 Foreground colour RGB   : R=255, G=255, B=0
 Background colour RGB   : R=0, G=0, B=0

4: Offset = 9, Length = 3, Text = "ddd"
 No action.
 Flags = Hilite: YES, Underline: no, Blink: no, Inverse: no, Changed: no
 Foreground colour RGB   : R=255, G=0, B=0
 Background colour RGB   : R=0, G=0, B=0

12 columns in 4 style runs

------ (end line information) ------


Notice how the bbb and the ddd lines are both marked "Flags = Hilite"? So it has correctly processed the highlight flag.

But why doesn't it display correctly? Well ... MUSHclient has three storage modes for lines, ANSI-lookup, custom-lookup, and RGB.

The first two only store an index (palette number) and at display time that index is looked-up. So, if you are using ANSI or custom colours you can change the colour configuration and the screen redraws in the new colours (even text already received).

So if you were in those modes, the order of receiving colour/highlight or highlight/colour would not matter. The colour lookup is deferred anyway.

But by using the 256-ANSI code, it switches the style-run to RGB-colour. These colours are looked-up now, not at display time. Now the problem arises.

It gets 31 (red) and looks that up in the ANSI table (from world ANSI configuration page). It finds non-bold red (ie. 128) and stores that.

Then it gets 1 (bold), but this does not trigger a colour lookup. It just turns on the bold flag. Now this would work for ANSI/Custom colours because the colours are looked up later. But for the RGB colours, it really needs to relookup the colour. But this is the rub, it hasn't remembered 31. It remembers R=128,G=0,B=0. So it can't relookup the previously processed colour.
Amended on Wed 04 May 2011 11:29 PM by Nick Gammon
USA #4
Nick Gammon said:
Quote:

The ANSI sequences defined in the ANSI function will be recognised.


It is behaving as documented.

You're technically correct, Nick... but that's one of the worst kinds of correct. I want to know why it doesn't support xterm sequences (which are ANSI sequences themselves), and why it acts any different from incoming-data processing. Does it not use the same ANSI parser? Is there any reason AnsiNote shouldn't support xterm-256? Pointing to the documentation is hand-wavy.

Also, \e[5m works but isn't documented.


Nick Gammon said:
Very helpful clue, thank you.

Sure!

Nick Gammon said:
But by using the 256-ANSI code, it switches the style-run to RGB-colour. These colours are looked-up now, not at display time. Now the problem arises.

It gets 31 (red) and looks that up in the ANSI table (from world ANSI configuration page). It finds non-bold red (ie. 128) and stores that.

Ahh, I see. When I wrote the renderer for Aspect, I delayed style lookup until the moment the text was rendered. I simply take the xterm palette as the base, and map the ANSI color sequences onto the first eight. When the stylerun is created, I check the bold flag and add 8 to the value (boosting it into the second eight of the xterm palette), and then resolve the color.

If the last color sent used the xterm sequence instead of the standard sequence, I set a flag to prevent 8 from being added, so the bold flag has no effect on xterm colors.

[...other methods...]
newStyleRun: function() {
  var fg = this.fg;
  if (this.bright && !this.xterm) {
    fg += 8;
  }
  
  var bg = this.bg;
  if (this.negative) {
    bg = fg;
    fg = this.bg;
  }
  
  if (this.concealed) {
    fg = bg;
  }
  
  var span = ["<span class=\"fg" + fg + " bg" + bg];
  if (this.italic) { span.push(" italic"); }
  if (this.strike) { span.push(" strike"); }
  if (this.underline) { span.push(" underline"); }
  span.push("\">");
  
  return span.join("");
}
[...]
Amended on Wed 04 May 2011 11:43 PM by Twisol
Australia Forum Administrator #5
Twisol said:

Is there any reason AnsiNote shouldn't support xterm-256? Pointing to the documentation is hand-wavy.


Sigh. The full ANSI implementation is done with a state machine that has (rather complex) calculations for the bold/underline/RGB/non-RGB/xterm-256 and so on handling in it. And in a very similar vein to the Simulate, it isn't particularly easy to "inject" the AnsiNote stuff into it. Doing that is likely to change the state for the current incoming line. So AnsiNote just implements a subset of the more commonly-used codes. And as noted, it supports the codes you can generate with the ANSI function. So it is symmetrical.

Twisol said:

Ahh, I see. When I wrote the renderer for Aspect, I delayed style lookup until the moment the text was rendered.


Actually I can't think offhand why that particular style run can't just degrade back to ANSI lookups. After all, once we hit the 31 code, we are no longer processing RGB colours. So that particular style should switch back to ANSI (not RGB) style, and then the bold problem goes away.
USA #6
Nick Gammon said:
Sigh. The full ANSI implementation is done with a state machine that has (rather complex) calculations for the bold/underline/RGB/non-RGB/xterm-256 and so on handling in it. And in a very similar vein to the Simulate, it isn't particularly easy to "inject" the AnsiNote stuff into it. Doing that is likely to change the state for the current incoming line. So AnsiNote just implements a subset of the more commonly-used codes.

Thanks. That's a lot more helpful to me. :)

IIRC, MUSHclient's ANSI implementation is built into the CMUSHClientDoc object, correct? I can definitely see how that would be a problem. If I were writing AnsiNote() for Aspect, I'd create a new ANSI parser object and run the text through it, so as to leave the original untouched.

Nick Gammon said:
And as noted, it supports the codes you can generate with the ANSI function.

ANSI(38,5,9) generates a perfectly valid sequence \e[38;5;9m, and in fact you can generate any arbitrary ANSI-formatted sequence using ANSI().
Amended on Wed 04 May 2011 11:59 PM by Twisol
Australia Forum Administrator #7
Twisol said:

ANSI(38,5,9) generates a perfectly valid sequence \e[38;5;9m, and in fact you can generate any arbitrary ANSI-formatted sequence using ANSI().


Perhaps I should clarify that. It supports the *documented* codes on the ANSI help page.




Now as for switching back from RGB mode to ANSI mode - I see I had the code to do that right here (ansi.cpp line 38 onwards):


  // switch back to ANSI colour if required
/*   obsolete
  if ((iCode >= 30 && iCode <= 37) ||
      (iCode >= 40 && iCode <= 47))
    {
    // we can't mix them - if we go back to ANSI colours we have to discard
    // the RGB codes
    if ((iFlags & COLOURTYPE) == COLOUR_RGB)
      {
      iForeColour = WHITE;
      iBackColour = BLACK;     
      }
    iFlags &= ~COLOURTYPE;  // clear RGB or custom bits
    iFlags |= COLOUR_ANSI;
    }

*/


But it is commented out with the note "obsolete". Now I don't take functionality away without a good reason. I can't remember what that reason was however.
Amended on Thu 05 May 2011 02:48 AM by Nick Gammon
USA #8
Nick Gammon said:
But it is commented out with the note "obsolete". Now I don't take functionality away without a good reason. I can't remember what that reason was however.

The blame output for the file on GitHub doesn't show any changes to the file since you originally pushed it, so it must have been an old change. It doesn't look wrong, anyways, and it's apparently not obsolete any more, so...
USA #9
I just ported my ANSI parser to C++, in case you want to use it. The "MyParser" demo I wrote does pretty much everything except actually pass data/styles to the output area.

If you don't use it that's fine, I just felt like doing something in C++.

https://gist.github.com/956475
Australia Forum Administrator #10
It would be done to solve a subtle bug. One I don't particularly want to re-introduce.

Well first, how big an issue is this? Whilst you have identified a bug (Fiendish) the easy work-around is to make it bold, and then change the colour. And it only shows up in the 256-ANSI mode.
USA Global Moderator #11
Quote:
Well first, how big an issue is this?

*shrug* :)
USA #12
This isn't really the place to rant, but honestly, the xterm sequences should've used an intermediate byte instead of an arcane, unusual sequence of parameters. "\e[156/m" would have made much more sense.

And just to spite the MUD gods, I'm going to implement exactly that. So nyah. >_<


(Intermediate bytes are specified in section 5.4 of the ANSI control sequence specification, found here: http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-048.pdf)

[EDIT]: The unallocated bit combinations are reserved for future standardization and shall not be used.
So it doesn't want me to use "/m". Mrrrrgh. Maybe I can use one of the sequences from the private-use table... Okay, standards issues aside, it makes a lot more sense than "38;5;<x>".
Amended on Thu 05 May 2011 11:12 AM by Twisol
USA Global Moderator #13
Fiendish said:

Quote:
Well first, how big an issue is this?

*shrug* :)

New answer: I just ran into this problem again because my StylesToANSI function favored 256ish equivalents for the standard 1-16 color values and I was trying to use AnsiNote instead of Simulate. I think it's weird for AnsiNote to not work the same way that Simulate does (or even in the same way that a series of consecutive ColourTells would), since re-displaying output from 256 enabled MUDs is such a useful concept.
Amended on Fri 30 Dec 2011 08:29 PM by Fiendish
Australia Forum Administrator #14
Do you actually want a bold font, or just the bolded colours?
USA Global Moderator #15
Oh sorry. The bold thing isn't my issue right now. I guess I'm looking just at the 38;5 thing.
Amended on Sat 31 Dec 2011 02:38 PM by Fiendish
Australia Forum Administrator #16
I am reluctant to fiddle with the colour-handling, particularly when there is code there that would do it, if it hadn't been commented out. That's a pretty sure sign that there is another subtle bug awaiting someone if it is changed.

If, when this bug was reported, there was a simple fix, I am sure I would have done the fix. As it is, I can't even remember what the bug is exactly.
USA Global Moderator #17
Note a huge deal, I guess. I just tried using AnsiNote on 256ified data, and had to figure out the problem and then change the display method after mining the forum and re-finding this thread.
USA Global Moderator #18
LOL. I just found this thread again. Same xterm 256 color reason. Why again does AnsiNote interpret xterm colors differently than Simulate?
Amended on Thu 20 Dec 2012 12:59 AM by Fiendish
Australia Forum Administrator #19
I'm not sure. That code was written in April 2003.

As I recall the idea of AnsiNote was to make it easy to inject ANSI sequences into your notes. However it doesn't go through the lengthy state machine that normal input (including Simulate) does. Basically the "usual" ANSI sequences were turned into the equivalent foreground/background colours. It may have bugs compared to the "main" state machine.

Judging by the release notes it was specifically written to support the Chat interface, where incoming chats had ANSI sequences in them, but were handled by scripts, and not the main "incoming MUD data" state machine.

Changing it now could be problematic. It might fix your problems but cause ones for others who are used to how it works now.

Why do you need to use it anyway?
USA Global Moderator #20
Nick Gammon said:

Changing it now could be problematic.
It might fix your problems but cause ones for others who are used to how it works now.
Currently xterm color just doesn't work. Wouldn't unifying with the other way keep the other bits working still and just make this work too?

Quote:
Why do you need to use it anyway?

I don't remember why I wanted to use it last time, but apparently I keep wanting to piece together ansi sequences without looping over colourtell. In this instance I was going to make a trigger to replace all server output with random ansi colors.
Amended on Thu 20 Dec 2012 04:48 AM by Fiendish
Australia Forum Administrator #21
Fiendish said:

If I execute:

Simulate(ANSI(38,5,226).."aaa"..ANSI(31,1).."bbb")
Simulate(ANSI(33,1).."ccc"..ANSI(31,1).."ddd")

aaa and ccc are the same colors (as expected), but bbb and ddd are not. bbb is not showing up as bold even though I have ,1 in there. Why?


Fixed in version 4.95.