Large text files

I am using the FileSystemObject to read from a text file and write to another text file. I am using the read(n) method of the Textstream object for optimal performance. The program works for files up to 4 megabytes but does not work for files larger than this. How can I overcome this? thanks in advance. the code:


Sub Prgm_Parse8

const ForReading = 1
const TristateFalse = 0
dim sName
dim outputFile
dim fso
dim objFile
dim objTS
set fso = CreateObject("Scripting.FileSystemObject")
sName = "W:\Copy of 2L14_9C016_B.wrl"
set objFile = fso.GetFile(sName)
set objTS = objFile.OpenAsTextStream(ForReading, TristateFalse)

'***********-----
contents = objTS.Read(objFile.Size)

dim fFile, contents, aFile, iLine, iLines, sLine, sWord

'loading file into a string
set fFile = fso.OpenTextFile( sName, 1 ) '1 = ForReading
'fFile.Close

'splitting string into an array
aFile = Split( contents, vbCrLf )

'scanning lines
iLines = UBound( aFile ) + 1
iLine = 0
do while iLine < iLines
sLine = aFile( iLine )

dim c
c=Left(sLine,1)
While c=Chr(9) or c=" "
sLine=Mid(sLine,2)
c=Left(sLine,1)
Wend
sWord=Left(sLine,3)

if ( sWord = "DEF" ) then

str1 = left(sLine, 5)
str2 = mid(sLine, 7)
str3 = UCase(Mid(sLine, 6, 1))

dim check_for_num
check_for_num = IsNumeric(str3)

if (check_for_num = true) then
sLine = str1 + "X" + str2
else
sLine = str1 + str3 + str2
end if

dim position

position = instr(sLine, ("Separator"))
position = (position - 5)
sLine = mid(sLine, 5, position)
sLine = CleanString( sLine, Chr(34) & "/,.*-&:" & " ")
str1 = left(str1, 4)
sLine = str1 + sLine + " Separator {"
end if

if ( sWord = "USE" ) then

str1 = left(sLine, 5)
str2 = mid(sLine, 7)
str3 = UCase(Mid(sLine, 6, 1))

'dim check_for_num
check_for_num = IsNumeric(str3)

if (check_for_num = true) then
sLine = str1 + "X" str2

else
sLine = str1 + str3 + str2

end if

sLine = mid(sLine, 5)
sLine = CleanString( sLine, Chr(34) & "/,.*-&:" & " ")
str1 = left(str1, 4)
sLine = str1 + sLine
end if

'- put it back in array with
aFile( iLine ) = sLine

iLine = iLine + 1
loop

'rejoining array items into a string, writing to output file
contents = Join( aFile, vbCrLf )

set fFile = fso.CreateTextFile( sName, True )
fFile.Write contents

fFile.Close

end sub

function CleanString( sInput, sSymbols )

dim sClean
sClean = ""

dim l, i, c
l = len( sInput )
i = 1

'- while more input string
do while i <= l

'- snip out a character
c = mid( sInput, i, 1 )

'- if not a symbol, add to cleaned string
if InStr( sSymbols, c ) = 0 then
sClean = sClean & c
end if

i = i + 1
loop

CleanString = sClean
end function

You are doing pretty well even reading 4 Mb in a file system object. I would break it down (eg. 4 Mb at a time) - the efficiency will not be much less, however you might have a problem at the boundary, as it may read in the middle of a line.

Can't you just do it in smaller chunks - eg a line at a time? If time is really critical I wouldn't be using a script engine anyway, they are not really intended for global find-and-replace. For instance, Perl might be better and faster.

Your efficiency gains are minimal after the natural cluster size on the disk (say, 4 Kb).

If you break the files up, it *should* be possible to do so by line count rather than total size to avoid breaking midline. Nick does have a good point about perl being more suited to something like this most likely though, esp if time is a factor.

time is not a factor. however, the program simply freezes when it tries to process larger files. what is the limit on the number of characters an array can hold?

I've already said "Can't you just do it in smaller chunks - eg a line at a time?" - I'll put it a bit more strongly:

Don't do it that way. Read a line at a time.

You are reading in a text file of over 4 Mb into a single variable, which VBscript is probably not designed to do, and then splitting that file up into lines anyway using Split, so now you have two copies of the 4 Mb (one in the variable, one in the array). The array may well have over 200,000 entries in it if each line is 20 bytes long.

You are then attempting to join that huge array back into a single variable and write it back. It only takes one bit of VB that doesn't cope with huge amounts of data and it won't work.

You only need a minor change - change the Read to ReadLine and then process a line at a time (writing each line as you go as well).

If time isn't an issue this should work fine, and you will have it working for any size file.