Recent

Author Topic: Reading and writing to files too slow  (Read 27931 times)

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #15 on: April 15, 2015, 10:46:49 pm »
Ok cool. I've already started implementing my solution of only updating the file once I finish design of a chosen region in the game world. However, I would really like the "increasing the buffer" solution to work as I would not need to change my code much. But I have tried that and it did not increase the speed of reading and writing. Maybe because I had to read value in char type variables, as mentioned in the tutorial, but I used ansistring and longint var instead. If I am obliged to use char var for the settextbuf to work then I don't think I'll use that solution since it will involve too many lines of codes to be rewritten.

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #16 on: April 15, 2015, 10:57:45 pm »
Just a few questions off topic:

How large can a .pas file be?

Is it the same limit for units too?

What would limit how large a .pas can be?

BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #17 on: April 15, 2015, 11:06:23 pm »
good questions.

i spent just a few minutes trying to find the answers - didn't see any.

i seriously doubt that you will encounter a limit on the files mentioned - due to Lazarus/FPC...

for what it's worth, i have a project with about 17 units and 25,000+ lines of code - no issues there.
« Last Edit: April 15, 2015, 11:10:07 pm by BitBangerUSA »
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #18 on: April 15, 2015, 11:25:52 pm »
good questions.

i spent just a few minutes trying to find the answers - didn't see any.

i seriously doubt that you will encounter a limit on the files mentioned - due to Lazarus/FPC...

for what it's worth, i have a project with about 17 units and 25,000+ lines of code - no issues there.

Yeah I was asking because I never found answers to these questions.

I was wandering because my project so far comes to a total of 64521 lines of codes. I have a lot of features in my game engine. Been working on it for two years. I remember when i used turbo pascal a few years back the compiler would give me an error and I had to start putting procedures in units but then it could not compile because it did not have enough memory to compile that many units. Since I switched to Free Pascal 2 years ago I have not had this issue and I'm only using 3 large units and the main program. The only "error" I had was when I had programmed a very large procedure. It did not behave as programmed (did not follow the flow of the commands) but there was no error message.


BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #19 on: April 15, 2015, 11:38:00 pm »
we can hope that a Developer (or other knowledgeable peep) will chime in here.
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #20 on: April 15, 2015, 11:39:47 pm »
we can hope that a Developer (or other knowledgeable peep) will chime in here.

Lol. Would be great.

rvk

  • Hero Member
  • *****
  • Posts: 6163
Re: Reading and writing to files too slow
« Reply #21 on: April 15, 2015, 11:55:46 pm »
However, I would really like the "increasing the buffer" solution to work as I would not need to change my code much. But I have tried that and it did not increase the speed of reading and writing. Maybe because I had to read value in char type variables, as mentioned in the tutorial, but I used ansistring and longint var instead.
No, you don't have to do read(ch); You need to specify a buffer for the read file and a buffer for the write file. FPC will also use that buffer for readln and writeln operations (flushing it to disk when necessary), not just for read(char). You can see this in the source-code at fpc\rtl\inc\text.inc (fpc_ReadLn_End, fpc_Writeln_End and SetTextBuf).
Code: [Select]
var
  Bufin,Bufout : Array[1..60000] of byte;
//....
  SetTextBuf (t[1],Bufin);
  SetTextBuf (t[2],Bufout);

I tried some sample code with a file with 6.000.000 lines (about 100MB) doing readln and writeln in a loop just takes a few seconds (at most) to completely copy the file (even without setting a larger buffer with SetTextBuf). So I'm guessing you're doing something extra that takes this time. You could try commenting out all the writeln's to see if it speeds up your code.

Otherwise you could strip this code to a working example-program and post it somewhere with a sample file so we can properly debug it (to see where the speed-problem lies).

I remember when i used turbo pascal a few years back the compiler would give me an error and I had to start putting procedures in units but then it could not compile because it did not have enough memory to compile that many units.
LOL, I still get this now and then with the Borland Pascal program we still support. Borland Pascal is DOS and 16bits. The total data-segment (global variables of all units) could only be 65536 bytes. There are more limits with Borland Pascal but with FPC you won't hit the limits of 32bit (or even 64bit) that easy. (2GB for 32bit, or even 3GB with the /3GB parameter)

Dividing you code into logical units will help you keep your code manageable.

O yeah... Gotos ??  >:D
« Last Edit: April 16, 2015, 12:00:04 am by rvk »

BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #22 on: April 15, 2015, 11:59:12 pm »
rvk - thanks for the info.

semichaud1:

Turbo Pascal! wasn't that great stuff in it's day?
man, i *really liked* that stuff.
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #23 on: April 16, 2015, 12:13:59 am »
However, I would really like the "increasing the buffer" solution to work as I would not need to change my code much. But I have tried that and it did not increase the speed of reading and writing. Maybe because I had to read value in char type variables, as mentioned in the tutorial, but I used ansistring and longint var instead.
No, you don't have to do read(ch); You need to specify a buffer for the read file and a buffer for the write file. FPC will also use that buffer for readln and writeln operations (flushing it to disk when necessary), not just for read(char). You can see this in the source-code at fpc\rtl\inc\text.inc (fpc_ReadLn_End, fpc_Writeln_End and SetTextBuf).
Code: [Select]
var
  Bufin,Bufout : Array[1..60000] of byte;
//....
  SetTextBuf (t[1],Bufin);
  SetTextBuf (t[2],Bufout);

I tried some sample code with a file with 6.000.000 lines (about 100MB) doing readln and writeln in a loop just takes a few seconds (at most) to completely copy the file (even without setting a larger buffer with SetTextBuf). So I'm guessing you're doing something extra that takes this time. You could try commenting out all the writeln's to see if it speeds up your code.

Otherwise you could strip this code to a working example-program and post it somewhere with a sample file so we can properly debug it (to see where the speed-problem lies).

I remember when i used turbo pascal a few years back the compiler would give me an error and I had to start putting procedures in units but then it could not compile because it did not have enough memory to compile that many units.
LOL, I still get this now and then with the Borland Pascal program we still support. Borland Pascal is DOS and 16bits. The total data-segment (global variables of all units) could only be 65536 bytes. There are more limits with Borland Pascal but with FPC you won't hit the limits of 32bit (or even 64bit) that easy. (2GB for 32bit, or even 3GB with the /3GB parameter)

Dividing you code into logical units will help you keep your code manageable.

O yeah... Gotos ??  >:D

Thanks!

I will make a sample code tomorrow from what I have in my program. Hopefully, I can find out why this is so slow. Maybe it's because I'm reading the file line to ansistring or longint variables?

I use goto because I have to go to different lines of codes in the program. The procedure is quite complex as it handles the world building function when designing the game. Also because I want to stay within the same procedure and not have to make additional procedures. This is a habit I took when programming with turbo pascal because when I made too many procedures it would run out of memory and not compile. I know I could specify the condition for exiting the repeat until loop, but I need to exit the repeat until loop at the start of the loop and not at the end if the conditions are met. Because after exiting the loop it has to read and write the lines of the text file that follow. It's due to how I have structured the data in the text files.

« Last Edit: April 16, 2015, 12:29:43 am by semichaud1 »

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #24 on: April 16, 2015, 12:19:20 am »
rvk - thanks for the info.

semichaud1:

Turbo Pascal! wasn't that great stuff in it's day?
man, i *really liked* that stuff.

Yeah Turbo Pascal was the first programming language I used on the PC in 2000. I had so many projects I developed with this but unfortunately could not have fast graphics rendering. And it's only two years ago that I learned that I needed to use something like SDL to handle graphics, sound and keyboard/mouse.

Wasted so many years and finally I have the knowledge and means to make a video game which is not turn-based, lol, because of limited programming language or lack of api. If I can overcome the problem I have now with slow read and write to file then I'll be all set to finish the game I am making. Hopefully, rvk can help with that after I send a proper sample of my code.

Leledumbo

  • Hero Member
  • *****
  • Posts: 8757
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Reading and writing to files too slow
« Reply #25 on: April 16, 2015, 06:33:40 am »
How large can a .pas file be?

Is it the same limit for units too?

What would limit how large a .pas can be?
No specific limits for a filesize as it's negligible over the complexity of the source code itself. This is the documentation of compiler limits. Data type wise, fize size can be as big as High(Int64) but I believe the compiler would be dead loooooooooong before it reaches that limit due to generated data structure from the unit.

Running:
Code: [Select]
$ find . -type f \( -name "*.pas" -or -name "*.pp" -or -name "*.inc" \) -print0 | xargs -0 ls -ShalIn fpc source dir, here are the top 10 biggest files:
Quote
-rw-r--r-- 1 leledumbo users 5.1M Aug 17  2014 ./packages/rtl-unicode/src/collations/collation_zh_be.inc
-rw-r--r-- 1 leledumbo users 5.1M Aug 17  2014 ./packages/rtl-unicode/src/collations/collation_zh_le.inc
-rw-r--r-- 1 leledumbo users 4.6M May 31  2014 ./packages/rtl-unicode/src/inc/cp936.pas
-rw-r--r-- 1 leledumbo users 4.3M Aug 17  2014 ./packages/rtl-unicode/src/inc/cp949.pas
-rw-r--r-- 1 leledumbo users 4.1M May 31  2014 ./packages/rtl-unicode/src/inc/cp950.pas
-rw-r--r-- 1 leledumbo users 3.8M May 31  2014 ./packages/rtl-unicode/src/inc/cp932.pas
-rw-r--r-- 1 leledumbo users 2.3M May 31  2014 ./packages/rtl-unicode/src/inc/ucadata_be.inc
-rw-r--r-- 1 leledumbo users 2.3M May 31  2014 ./packages/rtl-unicode/src/inc/ucadata_le.inc
-rw-r--r-- 1 leledumbo users 1.3M May 31  2014 ./packages/rtl-unicode/src/collations/collation_ja_be.inc
-rw-r--r-- 1 leledumbo users 1.3M May 31  2014 ./packages/rtl-unicode/src/collations/collation_ja_le.inc
The biggest file is 87054 LOC so your 64521 should be fine.

JorgeAldo

  • New Member
  • *
  • Posts: 11
Re: Reading and writing to files too slow
« Reply #26 on: April 16, 2015, 06:55:15 am »
are you doing read for single chars ?

if yes, it IS slow from turbo pascal era and it is still slow today...

you should use binary files and blockread/blockwrite or Tfilestream.

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #27 on: April 16, 2015, 07:50:15 pm »
Hi everyone, please see the attached doc. Thanks!

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #28 on: April 16, 2015, 07:50:55 pm »
are you doing read for single chars ?

if yes, it IS slow from turbo pascal era and it is still slow today...

you should use binary files and blockread/blockwrite or Tfilestream.

I'm reading and write ansistring and longint one line at a time from the file to another file.

rvk

  • Hero Member
  • *****
  • Posts: 6163
Re: Reading and writing to files too slow
« Reply #29 on: April 16, 2015, 07:54:54 pm »
Hi everyone, please see the attached doc. Thanks!
Do you have the possibility to provide us with a valid tf127.wor file so we can test your code?

 

TinyPortal © 2005-2018