Reading and writing to files too slow

[semichaud1]

Hi everyone,

In my program I have to read and write large files (100 MB or more) but each time it is taking too long.

I have tried to use settextbuf after reset and rewrite but it makes no difference in speed.

I need to use text files to store large quantities of data that my game engine uses.

Could you please help?

[eny]

Can you tell a bit more about how you do the reading and writing?
Because with for example a TStringList reading and writing of even a 100MB file takes relatively little time.

[semichaud1]

I am using the following commands:

assign
reset
rewrite
readln
writeln

and I am reading and writing ansistring and uint32 variables.

Please see the attachment for a section of my program.

[BitBangerUSA]

Attachments and other options - below message text-box seen when posting a reply.

[semichaud1]

Thank you. I have attached a section of my program.

[eny]

Purely looking at text I/O it should not take that long.
Did a quick test reading and writing a 100MB file (1,000,000 lines) and it takes less than a second.

You are also calling a couple of graphical functions (SDL_xxx and linergba) inside the loops that read the (big?) data.

You could:
- read and write to memory instead of diskfiles (e.g. TStringList).
- cache all graphical functions and do them all áfter you have processed the text.

[BitBangerUSA]

briefly looked at your attachment - but not long enough to follow what you are doing.
i do note that you have only 110 elements in your 3 arrays, so - in general terms - an hour to do your task is... surprising.

again, in general terms, you should look at structuring your data and the files in some way - this can certainly drastically reduce the time spent in the code to process the files.

again, i didn't attempt to understand your code, so that may not be good advice.

[semichaud1]

Thank you for your replies.

The sdl function is drawing the loading bar to indicate how much is left of the process of reading and writing of files to be done. Even without the sdl command it takes a while.

What I am doing essentially is this:

I spent two years programming a game engine from scratch. The game engine uses data in text files. Those text files hold so much data, for example integer values for each tile of each layer in each region in the game I am making. So I programmed what I call a construction tool to add, remove and edit the values in those text files. However, the text files are so large that when I update them, which involves read and write commands, it takes a very long time. I have tried to use the settextbuf command but it made no difference in speed.

[eny]

I'm guessing that you update the data time and time again?
E.g. one tile changes and then you recreate the complete textfile?

I would create a class that caches all that data.
An interface between the physical file and the program logic.
So that you read/parse the big file only once and use the data where required.
Updating the data is then also done in memory and written to disk at a convenient moment, when absolutely necessary.

Or you could abandon the whole textfile idea and put all data into for example a SQLite db.
Gives you a lot of flexibility in storing the data in a more logical way and especially gives you the means to update only data that actually needs updating.
After you've finished the 'construction' you could generate a textfile from the SQL data to be used by your game engine.

Or you could use recordfiles so that you can use direct access to the data and update only the data that needs updating.
(Note that when this gets more elaborate you will quickly reach the point that a SQLite db might be worthwhile.)


<<edit>>
By the way: goto's?? 

[semichaud1]

Thank Eny for the help!

So right now before updating the text file, all this data is actually loaded in arrays that are used to store the values of each tile in the selected region of the game world. What if I updated those variables only during level design and then once I am satisfied with the design of the region and where each tile is, then I updated the text file and let it take the time it needs? It should enable me not to have to wait each time I make an update to tiles in the region. I think I might do this. Why did I not think of this last night when I could not sleep trying to solve this problem in my head?

[eny]

I always have the best ideas the next morning

One thing to be aware of is that you lose all intermediate changes if something goes wrong (program crash or whatever).
I don't know how much work it is to create your config.

Another approach is that you access a textfile as a binary file: you can even use direct access to all data in a textfile.
You only need to make sure that when you update data, you do not go beyond what was originally written at a certain position.

[semichaud1]

I see. I have seen a tutorial about binary files in free pascal but I failed to see the "binary" side of it in the tutorial, lol. Maybe if you have a link to a good tutorial about this.

[BeniBela]

Isn't there some buffer size you need to increase for readln/writeln?

[semichaud1]

Yes Benibela. I actually tried the settextbuf command and that made no difference.

[eny]

I see. I have seen a tutorial about binary files in free pascal but I failed to see the "binary" side of it in the tutorial, lol. Maybe if you have a link to a good tutorial about this.

This is a more complex solution: you have to (binary) index each line of the file and preserve space for text and numbers when writing the file.
The other alternatives are the preferred ones.

Some info in the Wiki: http://wiki.freepascal.org/File_Handling_In_Pascal#Binary_files