Generic directory scanning algorithm

Posted by Nick Gammon on Mon 07 Dec 2009 11:54 PM — 3 posts, 14,190 views.

Australia Forum Administrator #0
The function below can be used to scan your disk from any given starting point, recursing into subdirectories. For each file found it calls a supplied function to do something with the file.


function scan_dir (path, f)

  -- find all files in that directory
  local t = assert (utils.readdir (path .. "\\*"))

  for k, v in pairs (t) do
   if not v.hidden and 
      not v.system and
      k:sub (1, 1) ~= "." then
      
      -- recurse to process file or subdirectory
      if v.directory then
        scan_dir (path .. "\\" .. k, f)
      else
        f (path .. "\\" .. k, v)
      end -- if 
     
   end -- if
  
  end -- for

end -- scan_dir 


It excludes files starting with a "." (which stops it going into the current directory (.) or the owner (..). However if you wanted to process files starting with a dot, you could omit that test, and change it to not process a directory with those exact names.

It also excludes system and hidden files, you could remove those tests too. (See next post for how to do that).

An example of it in use is:


function scan_dir (path, f)

  -- find all files in that directory
  local t = assert (utils.readdir (path .. "\\*"))

  for k, v in pairs (t) do
   if not v.hidden and 
      not v.system and
      k:sub (1, 1) ~= "." then
      
      -- recurse to process file or subdirectory
      if v.directory then
        scan_dir (path .. "\\" .. k, f)
      else
        f (path .. "\\" .. k, v)
      end -- if 
     
   end -- if
  
  end -- for

end -- scan_dir 

plugins = {}

-- this function is called for every found file
function load_file (name, stats)
 
  if stats.size > 0 and
     string.match (name:lower (), "%.xml$") then
       table.insert (plugins,  name)
  end -- if
  
end -- load_file

scan_dir (GetInfo (60), load_file)

require "tprint"
tprint (plugins)  -- display results


In this case I pass the function "load_file" to the "scan_dir" function. For every file found in the directory tree, load_file is called with the file name, and the file information (see utils.readdir for the exact things you get). You might test the file size (as I do above), and use a string.match to narrow the file names down to specific ones.

You could use this to quickly locate plugins, sounds, pictures, MUSHclient world files, or indeed anything you want. The starting location should *not* end with a slash (although GetInfo (60) puts one there, that seems to work anyway). Thus you could scan your entire C drive like this:


scan_dir ("C:", load_file)


I wrote this because I found I was doing lots of variants of directory scanners, with the test for the file names etc. inside the scanner, whereas this version splits the scanning, and processing, into more logical phases.
Amended on Tue 08 Dec 2009 12:03 AM by Nick Gammon
Australia Forum Administrator #1
This version omits the test for files starting with ".", and hidden and system files, so you get absolutely everything:


function scan_dir (path, f)

  -- find all files in that directory
  local t = assert (utils.readdir (path .. "\\*"))

  for k, v in pairs (t) do
      
    -- recurse to process subdirectory
    if v.directory then

      if k ~= "." and k ~= ".." then
        scan_dir (path .. "\\" .. k, f)
      end -- not current or owner directory

    else -- call supplied function
      f (path .. "\\" .. k, v)  
    end -- if 
 
  end -- for

end -- scan_dir 

Australia Forum Administrator #2
For each file the following is returned as a table value of the second argument to your function:
  • size - file size in bytes
  • create_time - creation time (except for FAT filesystems, where it is omitted)
  • access_time - last access time (except for FAT filesystems, where it is omitted)
  • write_time - time written
  • archive - true if archive. Set whenever the file is changed, and cleared by the BACKUP command.
  • hidden - hidden file. Not normally seen with the DIR command
  • normal - normal file. File can be read or written to without restriction.
  • readonly - read-only. File cannot be opened for writing, and a file with the same name cannot be created.
  • directory - Subdirectory.
  • system - system file. Not normally seen with the DIR command.