memory leak?

Posted by Jcl on Thu 03 Sep 2015 11:15 AM — 20 posts, 76,691 views.

Jcl Thu 03 Sep 2015 11:15 AM #0

I updated the MUSH from 4.91 to 4.99, and very sorry to tell you that the memory usage are increasing slowly.
I run 4.99 for two days, the memory usage increased from 10M to 100M.
The 4.91 has no similar problems.

Nick Gammon Australia Forum Administrator Thu 03 Sep 2015 08:28 PM #1

Can you narrow it down for me? Eg. try 4.95 and see if you have the leak. If not, try 4.97. If so, try 4.93. Keep going until you find which version caused the leak (eg. 4.93 to 4.94).

I have a copy open not connected to a MUD, and memory usage is still low (19 KB).

Jcl Fri 04 Sep 2015 07:20 AM #2

Nick Gammon said:

Can you narrow it down for me? Eg. try 4.95 and see if you have the leak. If not, try 4.97. If so, try 4.93. Keep going until you find which version caused the leak (eg. 4.93 to 4.94).

I have a copy open not connected to a MUD, and memory usage is still low (19 KB).

OK, I will try.

Jcl Mon 07 Sep 2015 12:25 AM #3

Got it. What I did is running multiple MUSH of different versions at the same time, connecting to the same MUD with the same plugins and do the same things for a long time, and then compare the memory usage. And I found out that from version 4.93 the memory usage began to increase.

Fiendish USA Global Moderator Tue 08 Sep 2015 05:12 PM #4

Is 4.93 the last good one or the first bad one?

Jcl Wed 09 Sep 2015 01:29 AM #5

Fiendish said:

Is 4.93 the last good one or the first bad one?

4.92 is the last good one.
4.93 ~ 4.96 are bad versions.
4.97 ~ 4.99 are worse versions.

Nick Gammon Australia Forum Administrator Wed 16 Sep 2015 02:54 AM #6

I tried various things with 4.99 to make its memory go up (once the output buffer filled up). Nothing really seemed to do it. In particular I tried the things mentioned in the release notes for version 4.93.

http://www.gammon.com.au/scripts/showrelnote.php?version=4.93&productid=0

I can't really fix this unless I have a clue where to look.

Nick Gammon Australia Forum Administrator Wed 16 Sep 2015 04:26 AM #7

Somewhat luckily, the changes in version 4.93 do not include the changes to any of the included libraries, so the bug (if there is one) has to be in the commits in the link above.

If you can find it, I would be obliged.

Meerclar USA Wed 16 Sep 2015 12:58 PM #8

I'm also not noticing memory increases once the buffer is full in 4.94 but I don't run any complex plugins. Now, the process of getting the buffer full is interesting in terms of memory use but that's because I run at 100k lines in buffer but even full I'm rarely over 40k memory per client and I've never been over 50k per client even after leaving clients up for a week at a time.

Nick Gammon Australia Forum Administrator Wed 16 Sep 2015 07:56 PM #9

In my test I sent 103535 lines to the client (8 MB of text) and with the default 5000 line output buffer, it is only using 14 KB of RAM, and that is leaving it running overnight.

Jcl Fri 18 Sep 2015 07:06 AM #10

I did the following things.
Step 1:
I let the plugin keep idle, the memory usage did not increased.

Step 2:
I let the plugin start, the memory usage began to increase.

Step 3:
I reinstalled the plugin, the memory usage had not been reduced.

Step 4:
I clear the output buffer, the memory usage had not been reduced.

Step 5:
I closed the world, the memory usage reduced back to the initial state.

Look, the memory can be released by the world, but can not be reused. Maybe the memory is not leaking.

Can you provide some tools to view the details of the memory usage for each world?

Jcl Fri 18 Sep 2015 07:21 AM #11

Nick Gammon said:

Somewhat luckily, the changes in version 4.93 do not include the changes to any of the included libraries, so the bug (if there is one) has to be in the commits in the link above.

If you can find it, I would be obliged.

The output from server contained both Chinese and ASCII characters. See the release notes for 4.93, is this problem related to UTF-8 ?

Donecce Fri 18 Sep 2015 03:30 PM #12

I am curious if you are using the Mush script function "ImportXML" anywhere.

Nick Gammon Australia Forum Administrator Fri 18 Sep 2015 08:53 PM #13

If we look at the release notes (in summary):

1. Fixed bug in the StripANSI function where it stopped at a semicolon.

Does the plugin call StripANSI?

2. Extra code to handle inadvertent injection of non-UTF-8 data into a UTF-8 stream (when the output window is set to UTF-8).

Is the output window set to UTF-8?

3. The AddFont function now allows you to add multiple fonts (one at a time).

Does the plugin call AddFont?

4. Fixed bug with selecting UTF-8 (Unicode) characters with the mouse, where if you clicked and dragged some characters would disappear and reappear.

Does the problem occur if you click and drag with the mouse? Or does it happen anyway?

5. Fixed problem with triggers where, if you switched to or from UTF-8 mode in the Output window configuration, triggers would expect (or not expect) UTF-8, incorrectly, possibly giving errors when new lines arrive from the MUD.

Do you switch to and from UTF-8 mode?

6. Fixed bug where, under obscure circumstances, you might get the error message "Not a valid base64 string - incorrect length (not mod 4)." when opening the triggers/aliases lists or similar windows in world configuration.

Do you open the triggers/aliases lists when using this plugin?

7. Added Lua script function: utils.utf8convert

Does the plugin use utils.utf8convert ?

8. Added support for MTTS (MUD Terminal Type Standard)

Does the world query for MTTS?

9. Added Lua script function: utils.glyph_available

Does the plugin use utils.glyph_available ?

Jcl Sat 19 Sep 2015 11:27 AM #14

Quote:
1. Fixed bug in the StripANSI function where it stopped at a semicolon.
Does the plugin call StripANSI?

No.

Quote:
2. Extra code to handle inadvertent injection of non-UTF-8 data into a UTF-8 stream (when the output window is set to UTF-8).
Is the output window set to UTF-8?

No, I didn't check the UTF-8(Unicode) option of the output configuration.

Quote:
3. The AddFont function now allows you to add multiple fonts (one at a time).
Does the plugin call AddFont?

No.

Quote:
4. Fixed bug with selecting UTF-8 (Unicode) characters with the mouse, where if you clicked and dragged some characters would disappear and reappear.
Does the problem occur if you click and drag with the mouse? Or does it happen anyway?

Because of not UTF-8 mode，this problem does not exist.

Quote:
5. Fixed problem with triggers where, if you switched to or from UTF-8 mode in the Output window configuration, triggers would expect (or not expect) UTF-8, incorrectly, possibly giving errors when new lines arrive from the MUD.
Do you switch to and from UTF-8 mode?

No.

Quote:
6. Fixed bug where, under obscure circumstances, you might get the error message "Not a valid base64 string - incorrect length (not mod 4)." when opening the triggers/aliases lists or similar windows in world configuration.
Do you open the triggers/aliases lists when using this plugin?

No.

Quote:
7. Added Lua script function: utils.utf8convert
Does the plugin use utils.utf8convert ?

No.

Quote:
8. Added support for MTTS (MUD Terminal Type Standard)
Does the world query for MTTS?

I am not sure about this.

Quote:
9. Added Lua script function: utils.glyph_available
Does the plugin use utils.glyph_available ?

No.

Jcl Sat 19 Sep 2015 12:11 PM #15

Donecce said:

I am curious if you are using the Mush script function "ImportXML" anywhere.

No, I never called this function.

Jcl Sat 19 Sep 2015 12:17 PM #16

I call these functions a lot.

1. wait.regexp(regexp)

2. wait.regexp(regexp, timeout)

3. wait.time(seconds)

4. string.find and string.match

5. Execute

The plugin's structure is roughly the way.


wait.make(function()
    repeat
        do_before_maze()

        walk(fight)

        do_after_maze()
    until stop
end)

function walk(f)
    while f() == 0 do
        go_next_room()
    end
end

function fight()
    repeat
        Perform skills and wait for kill over.
    until kill_over
    if all_over then
        return 1
    else
        return 0
    end
end

function go_next_room()
    Do look and wait for the exits message.
    Decide the next door to enter.
end

Nick Gammon Australia Forum Administrator Sat 19 Sep 2015 08:39 PM #17

Jcl said:

The output from server contained both Chinese and ASCII characters. See the release notes for 4.93, is this problem related to UTF-8 ?

...

No, I didn't check the UTF-8(Unicode) option of the output configuration.

So you are using a Chinese character set, but not UTF-8? How does that even work? Aren't there more than 256 Chinese characters (glyphs)?

Jcl Sun 20 Sep 2015 04:29 AM #18

Nick Gammon said:

So you are using a Chinese character set, but not UTF-8? How does that even work? Aren't there more than 256 Chinese characters (glyphs)?

The charset is GBK. There is 21,886 Chinese characters. It's mixed with single byte characters (ASCII) and double byte characters (Chinese).

Meerclar USA Sun 20 Sep 2015 06:02 AM #19

Nonstandard character set, expect nonstandard results. I can see where, once upon a time, the minor bandwidth savings GBK offers over a proper UTF-8 character set would have been beneficial but on today's internet I just don't see an appreciable advantage. Most likely, the 'leak' you're experiencing is an accumulation of the 2nd bits used for characters that aren't represented in ascii that aren't presented to the ascii text handler in a proper format to be recognized as releasable bits come time to recycle memory allocations. Doesn't happen with an actual utf-8 or ascii charset because they follow proper ascii or utf-8 encoding rules rather than trying to be both and neither at the same time.