This forum is a read-only archive of the Gammon Software forum (2000–2026). No new posts can be made. Search the archive.

Alert! New version of Lua is being developed (Lua 5.2)

Posted by Nick Gammon on Wed 10 Feb 2010 07:31 PM — 28 posts, 79,473 views.

Nick Gammon Australia Forum Administrator #0
I note that the Lua developers are working on version 5.2 and releasing work versions.

http://www.lua.org/work/doc/

Below is a copy of the tentative changes announced so far:


Changes since Lua 5.1

Everything may change in the final version.

Here are the main changes in Lua since its last release. The reference manual lists the incompatibilities that had to be introduced.

Language

  • new lexical environments.
  • no more fenv for threads.
  • ephemeron tables.
  • yieldable pcall/metamethods.
  • tables honor the __len metamethod.
  • max constants per function raised to 2^26.
  • hex escapes in strings.
  • no more verification of opcode consistency.


Libraries

  • new library for bitwise operations.
  • new metamethods __pairs and __ipairs.
  • arguments for function called through xpcall.
  • optional argument to load (to control binary x text).
  • loadlib may load libraries with global names (RTLD_GLOBAL).
  • new function package.searchpath.
  • optional base in math.log.
  • file:write returns file.
  • closing a pipe returns exit status.
  • os.exit may close state.
  • new option 'isrunning' for collectgarbage and lua_gc.
  • frontier patterns.
  • ipairs now goes until #t.


Implementation

  • emergency garbage collector (core forces a full collection when allocation fails).
  • ephemeron tables (tables with weak keys only visit value when key is accessible)
  • handling of non-string error messages in the standalone interpreter.
  • udata with finalizers are kept in a separated list for the GC.
  • CallInfo stack now is a linked list.
  • internal (immutable) version of ctypes.
  • parser uses much less C-stack space (no more auto arrays).
  • new hash for floats.
Nick Gammon Australia Forum Administrator #1
Frontier patterns have been around for a while:

Template:post=6034
Please see the forum thread: http://gammon.com.au/forum/?id=6034.
Nick Gammon Australia Forum Administrator #2
It is interesting and amusing to note that one of the functions I was using heavily to load up imported "game" data was setfenv, and that has been removed from Lua 5.2

However the functionality has not.

For example, if I read it correctly, instead of:


local t = {} 
local f = assert (loadstring (arg))
setfenv (f, t) 
f ()


you would use:


local t = {} 
local f = assert (loadin (t, arg))
f ()


... where "loadin" loads a string "arg" in the environment of "t".

Or, maybe this:


local t = {} 
in t do  
  local f = assert (loadstring (arg))
  f ()
end -- in


It was also interesting to note on the Lua mailing list someone saying:

Quote:

I too was sad to see those function go as I frequently use them to isolate data (formated as lua code) loading in my games.


So it seems it is not just me that is using Lua as a method of transporting data from client to server in a game.
Twisol USA #3
Looking over the 5.2 reference manual, it looks like this will be a very useful release. I especially like the idea of lexical environments (in <x> do <y> end), and I expect the loss of setfenv/getfenv won't be such a loss after all.

Once it's finalized, will you be including Lua 5.2 with MUSHclient? I imagine it would break plenty of existing plugins/libraries, so perhaps a choice between 5.1 and 5.2 could be offered. The current 'language' XML attribute could keep 'Lua' as meaning 5.1, but also add explicit identifiers 'Lua5.1' and 'Lua5.2', so newer plugins can choose which one to use.

EDIT: Any idea how a construct like this might work? Specifically, is the table used for 'in' reused or replaced over every iteration? I would expect it to be replaced, but I can't really test it.

for _,v in ipairs(funcs) do
  in {} do
    v()
  end
end
Amended on Wed 10 Feb 2010 09:18 PM by Twisol
Nick Gammon Australia Forum Administrator #4
It is easy enough to download and build it:

http://www.lua.org/work/

Quote:

in exp do block end

...

Expression exp is evaluated only once in the beginning of the statement and it is stored in a hidden local variable named (environment).


Twisol USA #5
Ahh, ok.

I did just build it, actually. I tested this code in the interpreter:

foo = function()
  print(test)
end

test = 1

do
  local foo = foo
  in {test = 2} do
    foo()
  end
end


It printed 1. It makes sense in retrospect, functions keep the environment they had when they were defined.
Nick Gammon Australia Forum Administrator #6
I did something similar:


function f1 ()
  print "f1"
end -- f1

function f2 ()
  print "f2"
end -- f2

funcs = { f1, f2 }

for _,v in ipairs(funcs) do
  in {} do
    v()
    print "hello"
  end
end


I was a bit surprised it printed "f1" before throwing the error that "print" was not defined.

I suppose this is what they mean by a "lexical environment". The environment applies within the "in" block, but anything previously defined retains its original environment.
Amended on Wed 10 Feb 2010 09:50 PM by Nick Gammon
Nick Gammon Australia Forum Administrator #7
And what do you suppose this will do?


in {} do
  function f1 ()
    print "f1"
  end -- f1

  function f2 ()
    print "f2"
  end -- f2
end -- in

funcs = { f1, f2 }

for _,v in ipairs(funcs) do
  local print = print
  in {} do
    print "hello"
    v()
  end
end


  • Print: hello, f1, hello, f2?
  • Print: hello, then raise an error that print does not exist?
  • Print nothing at all?
  • Give a compile error?

Twisol USA #8
It does make a lot of sense. I suppose it's not a huge loss to do something like this instead:

function make_env()
  local _G = {}
  
  do
    local _G_meta = {
      print = print,
    }
    _G_meta.__index = _G_meta
    setmetatable(_G, _G_meta)
  end
  
  in _G do
    f1 = function()
      print("f1")
    end
    
    f2 = function()
      print("f2")
    end
  end
  
  return _G
end


It's no different from a typical closure, after all. Although I had to do some odd contortions to hide everything but the local _G from the 'in' scope.
Twisol USA #9
At first glance, I would say that it would print "hello", then complain about 'print' not being defined. When I run it though... nothing happens visibly. How interesting.

EDIT: Actually, this snippet is effectively a no-op, since you don't store the table the functions are in.

in {} do
  function f1 ()
    print "f1"
  end -- f1

  function f2 ()
    print "f2"
  end -- f2
end -- in


EDIT 2: Ahh, now it makes sense. f1 and f2 are nil in the main scope, so funcs is basically an empty table, since you basically did {nil, nil}. So the for loop does absolutely nothing.
Amended on Wed 10 Feb 2010 10:34 PM by Twisol
Nick Gammon Australia Forum Administrator #10
Twisol said:

Once it's finalized, will you be including Lua 5.2 with MUSHclient? I imagine it would break plenty of existing plugins/libraries, so perhaps a choice between 5.1 and 5.2 could be offered.


Well I am reluctant to offer a choice, because then plugin developers for x years won't be able to rely on any of the new features.

However obviously we can't go cold turkey on the new version because the setfenv in particular is a technique I have been using and recommending for loading Lua variables into plugins. (Actually, now I look at it, the default suggested method does *not* use setfenv).

I note in luaconf.h, as expected, they have a compatibility option:


/*
@@ LUA_COMPAT_FENV controls the presence of functions 'setfenv/getfenv'.
** You can replace them with lexical environments, 'loadin', or the
** debug library.
*/
#define LUA_COMPAT_FENV


I think that will have to be set initially, so that plugins will just work, but you can still write using the new features from the start.

Then we can allow a couple of months for plugin-writers to adapt their plugins, maybe something like this:


local t = {}
if loadin then
  assert (loadin (t, arg)) ()
else
  setfenv (assert (loadstring (arg)), t) ()
end -- if


A plugin with that in it would work on both Lua 5.1 and Lua 5.2.

And, while the compatability option was turned on, old plugins would work with Lua 5.2 as well.

Then later on, a version is released that turns off the compatibility option, and plugin writers can then adapt their plugins to require the new version only (by the MUSHclient version number).

There is probably nothing too compelling in Lua 5.2 to rush into turning off compatibility - although having hex constants in strings will be nice.
Nick Gammon Australia Forum Administrator #11
Twisol said:

EDIT 2: Ahh, now it makes since. f1 and f2 are nil in the main scope, so funcs is basically an empty table, since you basically did {nil, nil}. So the for loop does absolutely nothing.


Yes, I guessed wrongly, same as you. :P

But it makes sense when you look at it.
Twisol USA #12
Just to note, using loadstring() in an 'in' block doesn't quite work the way you'd expect. Since loadstring comes from the original environment, no matter where you call it, the loaded chunk will refer to the original environment.

I'm experimenting with loadin() now...
Twisol USA #13
Nick Gammon said:
I note in luaconf.h, as expected, they have a compatibility option:


/*
@@ LUA_COMPAT_FENV controls the presence of functions 'setfenv/getfenv'.
** You can replace them with lexical environments, 'loadin', or the
** debug library.
*/
#define LUA_COMPAT_FENV


I think that will have to be set initially, so that plugins will just work, but you can still write using the new features from the start.


Excellent, I didn't think of that.
Twisol USA #14
How would you suggest loading a file into a given environment? There's no loadfilein() function, and in order to use loadin() you'd have to have access to file I/O methods, which are disabled in untrusted plugins/worlds.
Nick Gammon Australia Forum Administrator #15
In order to emulate what existing behaviour?
Twisol USA #16
This:

foo = loadfile("filename.lua")
setfenv(foo, env)
foo()
Nick Gammon Australia Forum Administrator #17
I can't get loadstring to work at all as I would expect, but since we have loadin that isn't a big deal.

One example of many I tried:


local arg = " a = 1; b = 2 "

-- need these inside the local environment
local print = print
local loadstring = loadstring
local assert = assert

local t = {}
in t do
  local f = assert (loadstring (arg))
  f ()
  print (a, b)
end -- in

print (t.a, t.b)


Prints:


nil     nil
nil     nil

Nick Gammon Australia Forum Administrator #18
Twisol said:

This:

foo = loadfile("filename.lua")
setfenv(foo, env)
foo()



Well you have got me on that one. I would have assumed:


in env do
  foo = loadfile("filename.lua")
  foo()
end


But since my loadstring doesn't work, I guess that won't either.
Twisol USA #19
Nick Gammon said:

I can't get loadstring to work at all as I would expect, but since we have loadin that isn't a big deal.

One example of many I tried:

...


Notice that if you change the final "print (t.a, t.b)" to "print (a, b)", it prints "1 2", confirming what I said before about loadstring (and loadfile by extension) loading in to their own environments, not the caller's environment.
Twisol USA #20
Nick Gammon said:

Twisol said:

This:

foo = loadfile("filename.lua")
setfenv(foo, env)
foo()



Well you have got me on that one. I would have assumed:


in env do
  foo = loadfile("filename.lua")
  foo()
end


But since my loadstring doesn't work, I guess that won't either.


loadstring itselfs returns a compiled chunk, and loadstring's environment is the one it was defined in (i.e. the original) rather than the one in the 'in' statement. Hence, the chunk's environment is also the original one. The only way to get around this is to get the contents of the file itself and pass that in to loadin(). This applies equally to loadfile - the only difference is where the chunk's code is obtained.
Amended on Wed 10 Feb 2010 11:02 PM by Twisol
Nick Gammon Australia Forum Administrator #21
You seem to be quite correct on both counts. Browsing the Lua list, they appear to suggest at one point as a replacement for your loadfile problem:


file = assert (io.open (path))
source = file:read ("*all")
file:close ()
func = loadin (env, source, "@"..path)


However this means making io.open available, and unless you play around a bit, it makes io.open for *output* available, which is a bit of a security hole, letting plugins write arbitrary files to your hard disk. Even reading them is bad enough. For example, a plugin might open all your world files, looking for player names and passwords, and then secretly pass them on using various mechanisms to the author.

At least loadfile would only load Lua code so its use for general reading of passwords was quite limited, and you couldn't use it to write to disk.

I don't know the answer to this ... do we need loadfile to be used with an environment? I admit I suggest it here:

Template:post=6294
Please see the forum thread: http://gammon.com.au/forum/?id=6294.


But would it be generally used?
Twisol USA #22
Nick Gammon said:
But would it be generally used?


Generally? Well, considering the usual complexity of a typical plugin - from what I've seen, not very - I doubt it. But I'm working a bit on a small library to complement my 'structured plugin' idiom, which does use this technique a bit. I call it Plugger, and it's something of a lightweight 'plugin framework'. I'm using addxml as the standard means of defining aliases, triggers, etc - which I collectively call 'reflexes' since they cause something to happen when an certain event occurs - and it's much simpler and pleasing to be able to write a file that looks like this:

alias {
  match = [[^\s*test\s+alias\s*$]]
  send = [[
    "stand"
  ]]
}

trigger {
  -- etc
}
Nick Gammon Australia Forum Administrator #23
I've posted a summary of this problem on the Lua list, maybe someone will suggest something. Probably they will suggest I make io.open available.
Twisol USA #24
Or you could write your own loadfile() wrapper. Loadfile only takes one argument currently, it shouldn't be so bad if you write a wrapper.

The below code is Lua, but you'd probably actually implement it in C. (lua_setfenv is still valid in the C API, as far as I know)


function loadfile(env, filename)
  -- support loadfile(filename) form
  if filename == nil then
    filename = env
    env = nil
  end
  
  local chunk = loadfile(filename)
  if env then
    setfenv(chunk, env)
  end
  
  return chunk
end


Then, users simply use "t = {}; loadfile(t, filename)", using the same argument order as loadin.

EDIT: Belatedly, I forgot the last 'end'. Added.
Amended on Thu 11 Feb 2010 12:20 AM by Twisol
Twisol USA #25
*laughs* The guy who responded on the mailing list basically suggested the same thing I just did. ^_^ He wrote his version in backwards-compatible Lua, whereas I'd suggest implementing it in C, but it's the same idea.
Amended on Thu 11 Feb 2010 01:41 AM by Twisol
Nick Gammon Australia Forum Administrator #26
Seems to me this is throwing functionality out the window. After all, the original loadfile (and setfenv) were core Lua functions. Using io.read means you have to incorporate the io library, something you may conceivably not want to do.

Your suggestion of a workaround function is OK (I mean, it can be solved one way or the other), but I think I'll plug at the Lua developers a bit more. After all, a new release shouldn't remove core functionality without giving a reasonable workaround (short of everyone furiously putting things back through custom functions).
Twisol USA #27
I fully agree. :)