LuaSocket downloading Https?

Posted by Rivius on Tue 18 Dec 2018 01:35 AM — 9 posts, 33,187 views.

Rivius Tue 18 Dec 2018 01:35 AM #0

So I have a function that currently downloads .json files from a MUD's associated API. It's worked for years for me and looks like this


function download_json(host, file)
	local c = assert(socket.connect(host, 80))
	c:send("GET "..file.." HTTP/1.1\r\nUser-agent: "..socket._VERSION.."\r\nHost: "..host.."\r\nConnection: close\r\n\r\n")
	
	local js = {}
	while true do
		local s, stat, p = receive(c)
		if not s and p then
			if (#p > 0) then
				s = p
			end
		end
		js[#js + 1] = s
		if (stat == "closed") then
			break
		end
	end
	c:close() -- close the connection
	return string.match(table.concat(js, "\n"), "(%{.+%})")
end

It seems like now, whenever I try to download the file using this function, I get the error:

1="HTTP/1.1 301 Moved Permanently"
2="Date: Tue, 18 Dec 2018 01:17:35 GMT"
3="Server: Apache/2.4.10 (Debian)"
4="Location: https://api.lusternia.com/characters/rivius.json"
5="Cache-Control: max-age=0"
6="Expires: Tue, 18 Dec 2018 01:17:35 GMT"
7="Content-Length: 339"
8="Connection: close"
9="Content-Type: text/html; charset=iso-8859-1"
10=""
11="<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">"
12="<html><head>"
13="<title>301 Moved Permanently</title>"
14="</head><body>"
15="<h1>Moved Permanently</h1>"
16="<p>The document has moved <a href="https://api.lusternia.com/characters/rivius.json">here</a>.</p>"
17="<hr>"
18="<address>Apache/2.4.10 (Debian) Server at api.lusternia.com Port 80</address>"
19="</body></html>"

I have poor knowledge of these things, but I suspect there was a silent change forcing https, which I'm guessing Luasocket doesn't support. Is there a relatively painless way of overcoming this to get the function working again?

Nick Gammon Australia Forum Administrator Tue 18 Dec 2018 06:46 AM #1

Does this help?

http://www.gammon.com.au/forum/?id=14093

Rivius Tue 18 Dec 2018 10:18 AM #2

Thank you. That does help a bit, though a lot more complicated than I thought. The issue is portability there though. It sounds like I'd need to install a number of files in the mushclient directory itself which makes it tricky to just distribute a 'system' that you simply have to unzip. I don't suppose there's an easier workaround to simply getting a .json file off the web?

Nick Gammon Australia Forum Administrator Tue 18 Dec 2018 07:48 PM #3

You can get a JSON file from the web, that isn't the problem. The problem is that they appear to be forcing you into HTTPS mode with a redirect.

Rivius Tue 18 Dec 2018 09:39 PM #4

Hrm. What are your thoughts of including luasec or otherwise into the Mush installer for future versions? I imagine more sites will move to https in time.

Fiendish USA Global Moderator Tue 18 Dec 2018 11:11 PM #5

Honestly, I'm not sure that MUSHclient should even include luasocket. That alone is already out of scope.

Rivius Wed 19 Dec 2018 02:23 AM #6

Fair enough.

I decided to go full frankenstein and did this instead.


	local web_address = "https://api.lusternia.com/"..webobj
	local time_start = GetInfo(304)
	local job_name = string.match(webobj, "([a-z]+)%.json")
	local file_name = "_"..job_name..".json"
	local bitsadmin = "/C bitsadmin /transfer "..job_name.." /download /priority normal "..web_address.." "..GetInfo(67)..file_name
	
	utils.shellexecute("cmd", bitsadmin, nil, "open", 0)
	while (GetInfo(304) - time_start < 10) do
		local local_file = io.open(file_name, "r")
		if local_file then
			local json_string = local_file:read("*all")
			local_file:close()
			os.remove(file_name)
			return string.match(json_string, "(%{.+%})")
		end
		coroutine.yield()
	end

A little slower, only works on windows, but it works.

Fiendish USA Global Moderator Wed 19 Dec 2018 11:12 PM #7

Another alternative would be to include a copy of wget.exe and use that. It still means distributing a dependency, but at least then you wouldn't have a bunch of fragments scattered around like it takes to add https to luasocket.

Nick Gammon Australia Forum Administrator Sat 22 Dec 2018 04:48 AM #8

Fiendish said:

Honestly, I'm not sure that MUSHclient should even include luasocket. That alone is already out of scope.

Yes, I try to not make the client too bloated. A while ago someone wanted me to add in FTP client functionality to "make it easier to download files from the MUD", something which I declined to do. It's just another thing to support, keep updating, troubleshoot, etc. when it is not really part of the core functionality.