Reading and writing to files too slow

[semichaud1]

Sorry guys, I'll have to work on it this weekend as it is getting late and need to work tomorrow. Thanks for the help. I really appreciate it. You are the first to help me like this of all the forums I've been to. I am just too tired from today's work.

[rvk]

That's fine. Get a good nights rest. When you have the time again, please run the code I gave you to test if your system also reads and writes the 100MB files in 6 seconds.

The error you get in test2....  Yeah... that's just the reason why a structured file (or database) would be more suited in this case. As I said... One unexpected value and the whole program crashes 

Good night for now... (Zzzzzzzz)

[Carver413]

you should be useing index maps to store your data. that is an 8bit bitmap or larger if you need. if you have more layers then you can use one bitmap for each layer.
bitmaps can be stored as binary files not text and will reduce file size and increase loading speed.
on the left you see an 8 bit map and on the right you see my world. notice that I am only useing about 5 or six tiles so I still have a lot of room to grow before I would need a 16 bit map.

[semichaud1]

Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!

[rvk]

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast.

Haha, Yeah, I knew it must have been something like that. Because reading and writing alone was real fast. You won't even need a real progress-bar for 7 seconds  (B.T.W you probably won't even need the settextbuf because my guess is it will be just as fast without them)

I would still advice you to look at some alternative-storage method in the future. For now it will work but the method Carver413 suggested looks much more stable. My game-programming knowledge is practically zero but my guess is that he has a lot more experience in that area.

For now... you can sleep tight... Good night... 

[semichaud1]

you should be useing index maps to store your data. that is an 8bit bitmap or larger if you need. if you have more layers then you can use one bitmap for each layer.
bitmaps can be stored as binary files not text and will reduce file size and increase loading speed.
on the left you see an 8 bit map and on the right you see my world. notice that I am only useing about 5 or six tiles so I still have a lot of room to grow before I would need a 16 bit map.

Thank you.

My game engine loads bmp files into a surface array at the start of the program then values are loaded from text files into multidimensional arrays for layers. Then the layers arrays are used depending on coordinates to display the corresponding surfaces from the surface array on the screen. It's basically a tile-based engine for graphics. My engine does much more of course since I am programming a RPG and have worked on it for 2 years during my free time after work. Making the character walk around the map is super fast of course I use getticks to create proper delays for character movement and every animation in the game. The problem I was having was with updating the values of the text files during level design. But now it is solved.

[BitBangerUSA]

Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!

good deal. i'll assume you are now sleeping.

[eny]

Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!

Hm, I remember seeing that suggestion somewhere else.... 

Purely looking at text I/O it should not take that long.

You could:
- cache all graphical functions and do them all áfter you have processed the text.

[JorgeAldo]

drop the assign/reset/readln...

if you need to store strings and integers, TFileStream is perfectly suited and fast.

[semichaud1]

Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!

Hm, I remember seeing that suggestion somewhere else.... 

Purely looking at text I/O it should not take that long.

You could:
- cache all graphical functions and do them all áfter you have processed the text.

Yes, you suggested removing the graphical functions from the loop. I had done so in the past but forgot that I had not tried it with the code using settextbuf. I had codes doing it without setting the buffer. I should have paid more attention! It would have saved me a lot of time and pain. Thanks.

[semichaud1]

drop the assign/reset/readln...

if you need to store strings and integers, TFileStream is perfectly suited and fast.

I did some reading on tfilestream a few days ago. Unfortunately, this would mean changing too many lines of codes and structure. I have been programming this engine for the past two years.

[eny]

I did some reading on tfilestream a few days ago. Unfortunately, this would mean changing too many lines of codes and structure. I have been programming this engine for the past two years.

Maybe something for your next rewrite; shouldn't take two years 
Glad it's working now.

[semichaud1]

Just want to say thanks again everyone for your help! I'll be working on my project and hopefully, one day, I can have others playing it and liking it.

I'll definitely be taking the advice for future projects.

[User137]

If you're into some advanced'ish game tile handling, there's also unit in my game engine called nxData https://code.google.com/p/nxpascal/source/browse/trunk/src/nxData.pas

I've so far only made 1 (unpublished) OpenGL demo with it using Terraria graphics, where tilemap is 4200x1200 in size, and the world can be freely and quickly moved and even modified with mouse at realtime. Each tile is 16x16 in size, so it would make a 67200x19200 bitmap. At fullscreen it renders at 500 fps on my computer, when i disable default frame limiters...

I never get tired praising the underlying system, which is compressing the nonvisible tile-index maps with multiple threads. Actual compressed file it makes about the whole world is 2Mb. Loading times are non-existent, fraction of a second. Of course TFileStream used.