Recent

Author Topic: TCSVDataset, Unidirectional, Packetrecords  (Read 1924 times)

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
TCSVDataset, Unidirectional, Packetrecords
« on: May 03, 2020, 05:35:53 pm »
Switching from TSdfDataset to TCSVDataset, in attempt to decrease memory consumption when reading from huge csv files.
1. I tried to use Unidirectional=True, but in that case I cannot open TCSVDataset (with Active:=True) when Unidirectional is set to True. If I switch back to UniDirectional=False, then dataset can be opened. Am I missing something or this is a bug?

2. If I set PacketRecords=1, what does it exactly mean for reading from first to last record, by using Next method? Are previous records kept in memory, buildng up in memory one by one record, or previous records are discarded from memory and only one record is kept in memory?
« Last Edit: May 03, 2020, 10:52:59 pm by tatamata »

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: TCSVDataset, Unidirectional, Packetrecords
« Reply #1 on: May 07, 2020, 12:14:19 pm »
In meantime, I have switched to CSVDocument's TCSVParser and it solved my problems with memory consumption, so I definitely recommend TCSVParser to anybody wanting to import huge CSV files.
However, my original question about TCSVDataset still remains.

LacaK

  • Hero Member
  • *****
  • Posts: 703
Re: TCSVDataset, Unidirectional, Packetrecords
« Reply #2 on: May 11, 2020, 09:10:07 am »
TCSVDataSet uses internally TCSVParser.
TCSVDataSet is derived from TBufDataset.

1. I am not sure, but I think, that UniDirectional should work. And yes it is way how to keep in memory only last readed record ... when we are speaking about buffer of records, which is hold by TBufDataSet

2. PacketRecords in this case will have no impact on memory consumption. It only says how many records will be read at once. But after they are read they will be keept in memory.

 

TinyPortal © 2005-2018