Addition to serialize.lua - Functions

This is a suggested addition to the serialize.lua file, described and implmented here: http://www.gammon.com.au/forum/bbshowpost.php?id=4960

Fadedparadox and I were chatting a bit, and he had shown me his serialization functions to pack a Lua table into a MUSHclient variable. I noticed that it didn't handle functions, and he explained why. It stuck in my head though, I guess.

Today we were talking, and it came up again. I experimented a bit, dumping a basic function and iterating over each of its bytes (printing out the char number for each). The functions all have a lot of null bytes, so I figured that, since MUSHclient is written in C++, and cstrings are null-terminated, that's probably why storing/getting a dumped function doesn't work: it only stores "LuaQ".

A half-hour's pondering came up with a handy solution: encode the functions in base64 before storing them. I initially tried Base64Encode(), which didn't work because it was a C function exposed to Lua, so I wasted a few hours trying to write my own encoder/decoder. Then Fadedparadox pointed out utils.base64encode, heh.

In short: Given a function, get its string.dump representation, and use utils.base64encode on it to make it MUSH-variable friendly. To retrieve, use utils.base64decode and loadstring(). Example:



If this technique is used to store a function that's part of a table, you can build in the loadstring yourself, though obviously the user still has to loadstring the overall table:

As an example, this is my table save function I modified using Twisol's idea:

http://trevize.pastebin.com/m781a063

It seems to work perfectly.

*thumbsup twisol*

Why would you want to do this? Variables change (eg. hp goes up), but functions don't, as they are written by a human.

I know some Lua functions "generate" other functions, which is effectively done by adding local variables to them (ie. upvalues). However the documentation for string.dump says "The function must be a Lua function without upvalues.".

This idea may work in the short term, but if you serialize stuff which is saved to disk for use later, and Lua has a version upgrade, the serialized function may not work in the new version.

This would be nice for completeness' sake, for one. For another, I have a relatively contrived example:

If you have a table containing two numeric values, and a variable to store a function, you could assign functions to that variable which would have different effects on the values. For example, add, subtract, exponentiation, etc. When you serialize the table (plus function), it remembers what "mode" it was meant to be in. In this case, upvalues wouldn't be a problem.

Here's some example code that I tested (also rather contrived), though it doesn't actually make use of the scenario above. It just shows how upvalues aren't a problem here.




EDIT: Possible example of the usefulness of serializing a function, completely apart from the table serialization bit, is that it can be sent to another plugin to modify its behavior, if it's built for such tinkering.


EDIT 2: For further clarification, you don't need to use a function that generates a function for this to be of use. It's basically just changing an algorithm that the object uses, and persisting that over saves.

Well, I can see good reasons for it, and a bad thing about it...

Good: If you use Tables as objects, you can persist the whole object, instead of designing the data segment to be separate and saving just that.

Bad: Distributing base64 encoded bytecode means you better be damn sure you trust whoever is giving you plugins. You lose the transparency of having functions in plain text so people can reassure themselves you're not trying to do Bad Things(tm) to them.

Bad2: Encoding multiple copies of the same function in different variables bloats the world/state file. These values aren't as small as just a simple string or integer after all.

Good2: I can see passing functions encoded to another plugin, that can be added to an array to simulate your own plugin callbacks... Just in case you like over engineering solutions and are afraid there'll be some random plugin that you didn't code (thus, obviously coded wrong :) ) that doesn't check source or message number or something and responds to EVERY broadcast...

I suppose, but the whole idea of a scripting language is it is high-level. Serializing functions is really serializing the internal byte-codes used by the Lua interpreter. It just seems to me this is going against the grain of a high-level scripting language.

Of course, you are welcome to do this in your own scripts, but I think to add it into the stock distribution might imply functionality that isn't really there (eg. the upvalues, problems with future versions, etc.).

@ WillFa: For Bad #1, that's true, there's a touchy side of this I hadn't considered. But you can do a lot of bad even without this "sneaky" bas64 thing, and it certainly wouldn't need to be added to serialize.lua to happen. Plus, I'm certain that a compiled .lua file could be supplied with a plugin, and all that's needed to use it is to dofile() it. There's absolutely no base64 there, and the point of the compiled Lua is hidden from sight. That's not something my technique creates: it's there already.

For Bad #2, that's also true, but consider that most functions wouldn't need serialization. This is a niche technique that is definitely useful where needed, but can usually be left alone. In my opinion, it's good for completeness' sake.

For Good #2, that's true, except I'd much rather use OnPluginBroadcast or CallPlugin anyways; using serialized functions seems hacky in this case.


@ Nick: When you're interacting with C++ via a Lua interface, I think it becomes slightly less high-level. Consider: if all we ever used here was pure Lua, then there would be no need for the obfuscation of the dumped function. It's only against the grain because it's a necessary evil.

Also, being able to serialize a function and save it somewhere is already supported by Lua by virtue of string.dump(), and this sort of thing could be emulated (albeit roughly) by saving the serialized functions to a file, and calling another plugin with the name of the file to load. It's, again, a bit hacky, so adding in-built support for serialized functions - which isn't terribly hard as I've displayed - simply makes it easier to do.

Upvalues are only a problem with closure functions so far as I've seen, and you can serialize the closure function itself, just not the function it creates (so far as I know). With future versions... you always have those kinds of problems, with loss of backwards compatability. I don't think high-value functions will be stored this way, though -and if they are, it's either fleetingly for passing it elsewhere within the client, or the coder better know the inherent risk.


EDIT: Anyways, thanks for the comments. At the very least it helped me understand some of the implications of this technique, and if I find a bona fide use for it, you can bet I'll be using it. Something to add to my "Oh nifty" list!