Lua script gobbling memory

Posted by Robin Lionheart on Sun 15 Oct 2006 06:57 AM — 10 posts, 30,155 views.

Robin Lionheart Sun 15 Oct 2006 06:57 AM #0

In MUSHclient 3.80 and 3.81 (at least), my Lua script rapidly [strike]leaks[/strike]consumes mass quantities of memory at about 300K/second. On one occasion, after a couple hours, MUSHclient crashed with a shower of tiny "Out of memory" dialog boxes, and Task Manager reported it was eating 1.2GB of RAM!

[Ed: Rewrote to make clear that my script is to blame]

Nick Gammon Australia Forum Administrator Sun 15 Oct 2006 07:10 AM #1

It depends a bit on what the scripts do.

Lua uses memory allocation for things like strings, tables, functions and so on.

I haven't had a problem myself, but if you made a script that made millions of something it would probably eventually run out of memory.

Are you suggesting this is a new problem in version 3.80? That is, it didn't happen in version 3.79?

Can you post the problem script, or confirm that the script does not create many strings/tables etc?

Robin Lionheart Sun 15 Oct 2006 08:09 AM #2

It does string concatenation operations [Ed: pretty much whenever I call Send(), including] assembling large strings [Ed: (about 4KB)] from the contents of tables.

I've noticed that when I minimize the program with Alt-Space Minimize or the minimize button in the upper right corner of the window, the memory usage in Task Manager shot down from 500,000K to about 10,000K. It seems this gets MUSHclient to garbage collect its memory space.

So I tried using 4NT to do

activate "MUSHclient*" min

and

activate "MUSHclient*" restore

, but it apparently doesn't garbage collect when it minimizes in response to IDE messages.

Nick Gammon Australia Forum Administrator Sun 15 Oct 2006 08:21 AM #3

The gradual decrease in memory is more likely to be that the operating system is swapping memory of inactive applications out to the page file on disk.

If you want to force a garbage collection simply do:



collectgarbage ("collect")

I suggest you read this technical note by one of the authors of Lua:

http://www.lua.org/notes/ltn009.html

He describes how doing string concatenation can be quite expensive, and a solution that is much more efficient.

Nick Gammon Australia Forum Administrator Sun 15 Oct 2006 08:26 AM #4

Quote:

It does string concatenation operations assembling large strings ...

I would like to clarify that this isn't really an established "bug" at this stage.

If you concatenate large strings, then it is likely to "consume memory". This is a different thing to "leak memory". The word "leak" implies there is something wrong.

The recently-implemented spell-checker stores around 70,000 words in Lua tables, and I haven't had a problem with "out of memory" errors.

Robin Lionheart Sun 15 Oct 2006 08:34 AM #5

Sorry to misattribute the problem to a memory leak.

I believe the primary culprit was a status line parser (which I added to my script after I upgraded from 3.79).

Changing

local status_line = hp .. " " .. mana .. " " .. move .. " " .. time .. " " .. append_to_status
SetStatus(status_line)

SetStatus(hp, " ", mana, " ", move, " ", time, " ", append_to_status)

apparently caused MUSHclient to stop hemorrhaging memory.

Now it consumes memory more slowly, so I'm going to have to tighten up my script. I constantly used

Send(string1 .. string2 .. string3)

where I could have used

Send(string1, string2, string3)

to avoid filling the heap with waste strings.

However, I do wonder why garbage collector allowed the memory usage to get up as high as 1.2GB over a couple hours; I recently let it get up to 200KB and minimized the program to flush it down to about 10KB again.

[Edited to combine 4 successive posts into 1 and correct my misstatements above]

Nick Gammon Australia Forum Administrator Sun 15 Oct 2006 10:39 AM #6

It sounds strange. You haven't changed the garbage collection thresholds have you?

Your new method would certainly reduce the number of strings Lua would have to create, so that is more efficient, but I would have expected its garbage collector to cope.

I might see if I can reproduce the problem tomorrow. It sounds like a simple script like your SetStatus line, run in a loop, should show whether or not there is a major problem.

What operating system are you running on? (XP or something else?).

Can you give an estimate of the number of times a minute you would update the status line like that? Once a second? Or much more often?

Also, how many lines are you allocating to the output window? Over time, a lengthy output buffer will necessarily use quite a bit of memory.

Robin Lionheart Sun 15 Oct 2006 10:50 AM #7

I didn't know I could change the garbage collection thresholds. The number of lines in my output buffer is 2500.

I'm using speedwalks and other macros which cause dozens of status/prompts to come in all at once, to be caught by a regex trigger breaking each into pieces and passing them to a script routine (without sending it to output) that then sends them to the status line. I figure the most it could be being called is about 100 times a minute, multiplied by the number of tabs I have open using this script.

I'm using Windows XP x64 Edition with an AMD Athlon 64 X2 multiprocessor CPU.

I've moved most of the code to the trigger to let trigger substitutions do the work of string concatenations:


<triggers>
  <trigger
   custom_colour="5"
   enabled="y"
   group="prompt"
   keep_evaluating="y"
   match="^&lt;(?P&lt;H&gt;\d+/\d+) (?P&lt;V&gt;\d+/\d+) (?P&lt;M&gt;\d+/\d+)(?: (?P&lt;X&gt;\d+/?\d*))?(?: (?P&lt;T&gt;\d+[ap]m))?&gt;$"
   name="StatusPrompt"
   omit_from_log="y"
   omit_from_output="y"
   regexp="y"
   send_to="12"
   sequence="1"
   variable="game_status"
  >
  <send>game_status = "H:%&lt;H&gt; V:%&lt;V&gt; M:%&lt;M&gt; %&lt;X&gt; %&lt;T&gt;"
update_status_line()</send>
  </trigger>
</triggers>

with the following in the script file:


game_status = ""
append_to_status = ""

function update_status_line()
	SetStatus(game_status, " ", append_to_status)
end

Doing this and repeatedly sending a flood of 500 say commands now causes my script to eat from 12K-24K/sec (and the scrollback should be full long since).

Typing

/collectgarbage("collect")

after doing so appears to have no effect on the memory usage displayed in Windows Task Manager.

Nick Gammon Australia Forum Administrator Sun 15 Oct 2006 08:43 PM #8

Quote:

I didn't know I could change the garbage collection thresholds.

I would advise you not to, the Lua manual suggests you can make things worse, I was just checking.

Thanks for your detailed response. You clearly have a problem, I am trying to narrow down what it might be.

I don't think it is the SetStatus part itself, nor the Lua script engine, per se. I ran this test in the Immediate (scripting) window on version 3.81, using Windows XP:


print ("memory usage before = ", collectgarbage ("count"))
hp = 4200
mana = 2200
move = 53
time = 1234
append_to_status = "the quick brown fox"

t1 = os.time ()
for i = 1, 500000 do
  local status_line = hp .. " " .. mana .. " " .. move .. " " .. time .. " " .. append_to_status
  SetStatus(status_line)
end -- loop
t2 = os.time ()
print ("time taken = ", t2-t1)

print ("memory usage after = ", collectgarbage ("count"))

This does 500,000 lots of the string concatenation you originally had (with the .. syntax) and then a SetStatus. I made up some figures, I presume they are similar to yours.

Looking at the XP task manager, these were my memory usage figures:

Before opening the world = 7,100 Kb
Before doing the SetStatus test = 8,556 Kb
After doing the SetStatus = 9,184 Kb

First, I am confused about your reported figures, where you said you got the usage down to 10 Kb. I think you must mean 10 Mb. (1 Mb = 1000 Kb).

Even with no world open my copy uses 7,100 Kb (that is, over 7 Mb).

Opening a world seems to take up about 1.5 Mb, which seems OK to me. However after doing 500,000 SetStatus and Lua string concatenations, the memory only rises aby 0.5 Mb.

Plus, I checked the memory usage inside Lua by doing this:

/collectgarbage ("count")

My results were:



memory usage before =  83.6337890625

time taken =  8

memory usage after =  83.849609375

Thus the memory that Lua thinks it has used is only 0.2 Kb - hardly anything.

I think your memory consumption must be elsewhere, although I must admit I am puzzled as to why the operating system reports 500 Kb of usage, where Lua reports less than 1 Kb.

While I investigate this further, it might help if you can make a test script that reproduces the problem. Along the lines of what I have done above with SetStatus in a loop, you might try pushing a lot of data through one of your other script routines to see if you can find what is taking all the memory.

You mentioned the use of tables, it would be possible to consume a lot of memory if you did something like this:



t = {}  --> make a table once



table.insert (t, "something")  -- for each line from the MUD

If you were doing something like this then the table t would gradually get larger, and not be garbage-collected.

Nick Gammon Australia Forum Administrator Sun 15 Oct 2006 10:50 PM #9

I have rerun the above test, and this time the Windows memory does not seem to increase at all (or by more than 1K or so) when running the above script. It is hard to know what is going on, when tests run at different times give totally different results.