Recent

Author Topic: Reading and writing to files too slow  (Read 27851 times)

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #30 on: April 16, 2015, 08:14:30 pm »
Hi everyone, please see the attached doc. Thanks!
Do you have the possibility to provide us with a valid tf127.wor file so we can test your code?

Unfortunately the site will allow attachments up to 250 KB and tf127.wor is 103 MB

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Reading and writing to files too slow
« Reply #31 on: April 16, 2015, 08:17:32 pm »
Unfortunately the site will allow attachments up to 250 KB and tf127.wor is 103 MB
If you zip the file (with 7zip) it might even be smaller than 250Kb (due to the fact it is text with a lot of repeats).

But if it's still too big... do you have dropbox (so you can share the link to the file) or another file-sharing service?

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #32 on: April 16, 2015, 08:19:40 pm »
Unfortunately the site will allow attachments up to 250 KB and tf127.wor is 103 MB
If you zip the file (with 7zip) it might even be smaller than 250Kb (due to the fact it is text with a lot of repeats).

But if it's still too big... do you have dropbox (so you can share the link to the file) or another file-sharing service?

I have zipped the file. Attached.

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #33 on: April 16, 2015, 08:24:37 pm »
The file takes ages to open even with Notepad

Leledumbo

  • Hero Member
  • *****
  • Posts: 8746
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Reading and writing to files too slow
« Reply #34 on: April 16, 2015, 08:29:03 pm »
The file takes ages to open even with Notepad
Notepad is a naive text editor. You can use text editors with on demand loading support such as whatever uses Scintilla (SciTE, Geany, Notepad++), Vim, Kate on KDE, etc. But a >100 MB text file is indeed HUGE.

EDIT:
Please provide a short, simple compilable example that demonstrates the behavior. Even after modifying your code to compilable one, I still get "EInOutError: Invalid input":
Code: [Select]
constructiontool1lb38:
      readln(t[1],o[3]); // HERE
so we shouldn't even start talking about speed. Apparently either the file format is wrong or the code you provided read it wrong. "EInOutError: Invalid input" will happen when the reader encounter invalid character for the expected variable's type (probably it encounters letters while o[3] is an int32).
« Last Edit: April 16, 2015, 08:42:08 pm by Leledumbo »

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Reading and writing to files too slow
« Reply #35 on: April 16, 2015, 08:32:11 pm »
Yeah. Notepad (and other editors) are not really suited to open a file with 27.658.261 lines. I thought you said it had 936.000 lines:
Quote
this code is reading and writing 640 * 360 * 4 lines from a text file to another but it takes nearly 1 hour to do that}

But it still should take an hour.

But the file you gave can't be read with the code you gave.
In your code you check with the following line and break:
Code: [Select]
    if o[4] = o[47] then
      goto constructiontool1lb38;
Could you explain where o[47] is initialized?

I thought the code you gave should just read the file and write a new file (with some alterations).

BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #36 on: April 16, 2015, 08:33:00 pm »
haven't tried your file, but...
see attachment
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #37 on: April 16, 2015, 08:37:43 pm »
Yeah. Notepad (and other editors) are not really suited to open a file with 27.658.261 lines. I thought you said it had 936.000 lines:
Quote
this code is reading and writing 640 * 360 * 4 lines from a text file to another but it takes nearly 1 hour to do that}

But it still should take an hour.

But the file you gave can't be read with the code you gave.
In your code you check with the following line and break:
Code: [Select]
    if o[4] = o[47] then
      goto constructiontool1lb38;
Could you explain where o[47] is initialized?

I thought the code you gave should just read the file and write a new file (with some alterations).

o[47] holds the value of the selected world or region in the game. The code is part of a much larger unit. Basically I am trying to use text files to store data which my game engine uses to display the tiles. Is there a better way to do this than using text files? I have not studied programming and this was the logical way for me and it works but it is very slow when updating those text files. Reading only to store in variables is faster.
« Last Edit: April 16, 2015, 08:45:24 pm by semichaud1 »

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #38 on: April 16, 2015, 08:48:28 pm »
I'm writing a procedure to create a file with as many lines as tf127 and then read from that and copy to another file and see how long it takes. Then I'll send use the code. Bare with me because I am also doing a workout (weight lifting) session at the same time.

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Reading and writing to files too slow
« Reply #39 on: April 16, 2015, 08:49:22 pm »
Well... before diving in the o[47] issue... I tried the following code.
It reproduced the tf127.wor (with all the 27.658.261 lines) in 6 seconds (on my system):
Code: [Select]
6.18 seconds elapsed

Just to test if it's the same on your system (elimating the source of virusscanner, defraged file, etc) could you put your tf127.wor file in C:\Temp and run the following code:
Code: [Select]
program test_speed2;
uses Windows;
var
  bufin, bufout: array[1..10000] of byte;
  t: array[1..10] of Text;
  o: int32;
  tm: Int64;
  s: ansistring;
begin
  tm := GetTickCount;
  Assign(t[1], 'c:\temp\tf127.wor');
  reset(t[1]);
  Assign(t[2], 'c:\temp\tf0.wor');
  rewrite(t[2]);
  settextbuf(t[1], bufin);
  settextbuf(t[2], bufout);
  repeat
    readln(t[1], s);
    writeln(t[2], s);
  until s = 'endfile';
  Close(t[1]);
  Close(t[2]);
  writeln((GetTickCount - tm)/1000:5:2, ' seconds elapsed');
  readln;
end.

How fast is this on your system?
(This is just to establish a base-line of how fast your system is compared to mine)

(It should also show that the read and write itself isn't the problem)

BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #40 on: April 16, 2015, 08:50:55 pm »
a better way? of course.
as said earlier, you need to structure your data.
that's a broad statement, but the structuring will be dictated by the data *and* how you need to access it.
basically, you have 2 choices - a flat text database with fixed record size, and the data 'grouped' in someway to let your program get to certain data without reading the entire file or using a traditional database.
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #41 on: April 16, 2015, 09:00:02 pm »
huh?

got the file you posted - BTW Textpad opened it in about 5 seconds on my machine.

all i see is

1
8
640
360
1
4
0
0
0
0
0
0
0
0
0
0
0
0
endtotalsprites

with the
1
4
0
0
...
0
0
endtotalsprites

repeated to line #27658261

it would seem you want to use a traditional database.
« Last Edit: April 16, 2015, 09:13:21 pm by BitBangerUSA »
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Reading and writing to files too slow
« Reply #42 on: April 16, 2015, 09:07:24 pm »
It was already established on page 1 a traditional database would be preferable. At this moment we are looking at the possible speed-implication of just reading and writing this file. Not that it could be done better another way. semichaud1 just wanted to know at this moment why it takes so long (over an hour), while for me copying the file with the 27.658.261 lines takes only 6 seconds.

@semichaud1: I take it you get the hint that it is better to alter this code in the future. This code is also without error-checking so any wrong character or unexpected value will completely crash your program.

BitBangerUSA

  • Full Member
  • ***
  • Posts: 183
Re: Reading and writing to files too slow
« Reply #43 on: April 16, 2015, 09:08:19 pm »
as rvk is providing much more specific help, i'll just watch the thread.
Lazarus Ver 2.2.6 FPC Ver 3.2.2
Windows 10 Pro 64-bit

semichaud1

  • New Member
  • *
  • Posts: 38
Re: Reading and writing to files too slow
« Reply #44 on: April 16, 2015, 09:15:37 pm »
It was already established on page 1 a traditional database would be preferable. At this moment we are looking at the possible speed-implication of just reading and writing this file. Not that it could be done better another way. semichaud1 just wanted to know at this moment why it takes so long (over an hour), while for me copying the file with the 27.658.261 lines takes only 6 seconds.

@semichaud1: I take it you get the hint that it is better to alter this code in the future. This code is also without error-checking so any wrong character or unexpected value will completely crash your program.

rvk thanks for the help. I have attached a .txt (please change to .pas, forum does not allow attachment for this type) I made just now to replicate what I am trying to do in my program as close as possible and I get an error which I don't get in my program. Could be another issue. Sorry i need to spend more time on this. But at the moment I am working out and also I am mentally tired from work. Working all day as a credit control analyst.

 

TinyPortal © 2005-2018