Recent

Author Topic: Reading and writing to files too slow  (Read 27850 times)

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #45 on: April 16, 2015, 09:32:22 pm »
Sorry guys, I'll have to work on it this weekend as it is getting late and need to work tomorrow. Thanks for the help. I really appreciate it. You are the first to help me like this of all the forums I've been to. I am just too tired from today's work. :)

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Reading and writing to files too slow
« Reply #46 on: April 16, 2015, 09:41:19 pm »
That's fine. Get a good nights rest. When you have the time again, please run the code I gave you to test if your system also reads and writes the 100MB files in 6 seconds.

The error you get in test2....  :D Yeah... that's just the reason why a structured file (or database) would be more suited in this case. As I said... One unexpected value and the whole program crashes  :o

Good night for now... (Zzzzzzzz)

Carver413

  • Full Member
  • ***
  • Posts: 119
Re: Reading and writing to files too slow
« Reply #47 on: April 16, 2015, 10:40:45 pm »
you should be useing index maps to store your data. that is an 8bit bitmap or larger if you need. if you have more layers then you can use one bitmap for each layer.
bitmaps can be stored as binary files not text and will reduce file size and increase loading speed.
on the left you see an 8 bit map and on the right you see my world. notice that I am only useing about 5 or six tiles so I still have a lot of room to grow before I would need a 16 bit map.
« Last Edit: April 16, 2015, 10:48:52 pm by Carver413 »

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #48 on: April 16, 2015, 11:20:18 pm »
Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Reading and writing to files too slow
« Reply #49 on: April 16, 2015, 11:26:47 pm »
I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast.
Haha, Yeah, I knew it must have been something like that. Because reading and writing alone was real fast. You won't even need a real progress-bar for 7 seconds  :) (B.T.W you probably won't even need the settextbuf because my guess is it will be just as fast without them)

I would still advice you to look at some alternative-storage method in the future. For now it will work but the method Carver413 suggested looks much more stable. My game-programming knowledge is practically zero but my guess is that he has a lot more experience in that area.

For now... you can sleep tight... Good night...  :)

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #50 on: April 16, 2015, 11:27:31 pm »
you should be useing index maps to store your data. that is an 8bit bitmap or larger if you need. if you have more layers then you can use one bitmap for each layer.
bitmaps can be stored as binary files not text and will reduce file size and increase loading speed.
on the left you see an 8 bit map and on the right you see my world. notice that I am only useing about 5 or six tiles so I still have a lot of room to grow before I would need a 16 bit map.

Thank you.

My game engine loads bmp files into a surface array at the start of the program then values are loaded from text files into multidimensional arrays for layers. Then the layers arrays are used depending on coordinates to display the corresponding surfaces from the surface array on the screen. It's basically a tile-based engine for graphics. My engine does much more of course since I am programming a RPG and have worked on it for 2 years during my free time after work. Making the character walk around the map is super fast of course I use getticks to create proper delays for character movement and every animation in the game. The problem I was having was with updating the values of the text files during level design. But now it is solved.


BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #51 on: April 16, 2015, 11:45:47 pm »
Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!

good deal. i'll assume you are now sleeping.
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

eny

  • Hero Member
  • *****
  • Posts: 1634
Re: Reading and writing to files too slow
« Reply #52 on: April 17, 2015, 12:09:29 am »
Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!
Hm, I remember seeing that suggestion somewhere else....  8)

Purely looking at text I/O it should not take that long.

You could:
- cache all graphical functions and do them all áfter you have processed the text.
All posts based on: Win10 (Win64); Lazarus 2.0.10 'stable' (x64) unless specified otherwise...

JorgeAldo

  • New Member
  • *
  • Posts: 11
Re: Reading and writing to files too slow
« Reply #53 on: April 17, 2015, 10:01:42 am »
drop the assign/reset/readln...

if you need to store strings and integers, TFileStream is perfectly suited and fast.

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #54 on: April 17, 2015, 04:59:54 pm »
Guys thank you all for the great replies and help!

I could not sleep unless I solved this issue.

I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum!
Hm, I remember seeing that suggestion somewhere else....  8)

Purely looking at text I/O it should not take that long.

You could:
- cache all graphical functions and do them all áfter you have processed the text.

Yes, you suggested removing the graphical functions from the loop. I had done so in the past but forgot that I had not tried it with the code using settextbuf. I had codes doing it without setting the buffer. I should have paid more attention! :) It would have saved me a lot of time and pain. Thanks.

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #55 on: April 17, 2015, 05:02:10 pm »
drop the assign/reset/readln...

if you need to store strings and integers, TFileStream is perfectly suited and fast.

I did some reading on tfilestream a few days ago. Unfortunately, this would mean changing too many lines of codes and structure. I have been programming this engine for the past two years.

eny

  • Hero Member
  • *****
  • Posts: 1634
Re: Reading and writing to files too slow
« Reply #56 on: April 17, 2015, 07:48:16 pm »
I did some reading on tfilestream a few days ago. Unfortunately, this would mean changing too many lines of codes and structure. I have been programming this engine for the past two years.
Maybe something for your next rewrite; shouldn't take two years  :D
Glad it's working now.
All posts based on: Win10 (Win64); Lazarus 2.0.10 'stable' (x64) unless specified otherwise...

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #57 on: April 19, 2015, 11:23:23 am »
Just want to say thanks again everyone for your help! I'll be working on my project and hopefully, one day, I can have others playing it and liking it.

I'll definitely be taking the advice for future projects.

User137

  • Hero Member
  • *****
  • Posts: 1791
    • Nxpascal home
Re: Reading and writing to files too slow
« Reply #58 on: April 21, 2015, 12:55:11 am »
If you're into some advanced'ish game tile handling, there's also unit in my game engine called nxData https://code.google.com/p/nxpascal/source/browse/trunk/src/nxData.pas

I've so far only made 1 (unpublished) OpenGL demo with it using Terraria graphics, where tilemap is 4200x1200 in size, and the world can be freely and quickly moved and even modified with mouse at realtime. Each tile is 16x16 in size, so it would make a 67200x19200 bitmap. At fullscreen it renders at 500 fps on my computer, when i disable default frame limiters...

I never get tired praising the underlying system, which is compressing the nonvisible tile-index maps with multiple threads. Actual compressed file it makes about the whole world is 2Mb. Loading times are non-existent, fraction of a second. Of course TFileStream used.

 

TinyPortal © 2005-2018