Recent

Author Topic: [SOLVED] Emptying a BufDataset  (Read 8984 times)

iru

  • Sr. Member
  • ****
  • Posts: 321
[SOLVED] Emptying a BufDataset
« on: April 24, 2014, 06:45:24 am »
Gentlefolk,

Somewhere I saw an explanation how to empty all data out of a tBufDataset. I can no longer find it.

It was something along the lines of:

Save off the tFieldDefs.
Somehow empty/clear the data.
Restore the tFieldDefs.

Can anyone explain how to empty a tBufDataset or point me to the explanation?

Thanks, Ian
« Last Edit: May 07, 2014, 10:10:37 am by iru »

mangakissa

  • Hero Member
  • *****
  • Posts: 1131
Re: Emptying a BufDataset
« Reply #1 on: April 24, 2014, 09:41:17 am »
TBufdataset.close
Lazarus 2.06 (64b) / FPC 3.0.4 / Windows 10
stucked on Delphi 10.3.1

janasoft

  • New Member
  • *
  • Posts: 30
Re: Emptying a BufDataset
« Reply #2 on: April 24, 2014, 09:44:56 am »
Perhaps this is what you are looking for:

http://wiki.lazarus.freepascal.org/How_to_write_in-memory_database_applications_in_Lazarus/FPC#Saving_MemDatasets_to_persistent_files

There, you can find the method EmptyMemDataSet(DataSet:TMemDataSet);
Lazarus 2.2.6 (rev lazarus_2_2_6) FPC 3.2.2 x86_64-win64-win32/win64

iru

  • Sr. Member
  • ****
  • Posts: 321
Re: Emptying a BufDataset
« Reply #3 on: April 24, 2014, 02:23:46 pm »
Gentlefolk,

Thank you for the responses.

Janasoft. In the link you provided I see something using tMemdataset which looks like "save the Fielddefs, drop the data, restore the Fielddefs", I will persue this on tBufDataset.

Mangakissa. This does not work. I have 4000+ records in the BufDataset. Close and then open the BufDataset and there are still 4000+ records.
I suppose I could Free/Destroy the BufDataset and then create it again but this seems a touch ugly.

Ian

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Emptying a BufDataset
« Reply #4 on: April 25, 2014, 07:15:27 am »
How about Close and then CreateDataset instead of Open?
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

mangakissa

  • Hero Member
  • *****
  • Posts: 1131
Re: Emptying a BufDataset
« Reply #5 on: April 25, 2014, 09:39:45 am »
Quote from: iru
Mangakissa. This does not work. I have 4000+ records in the BufDataset. Close and then open the BufDataset and there are still 4000+ records.
That's strange. This example show exactly the behavior of TBufdataset. Or you do something else to fill the dataset.
Lazarus 2.06 (64b) / FPC 3.0.4 / Windows 10
stucked on Delphi 10.3.1

iru

  • Sr. Member
  • ****
  • Posts: 321
Re: Emptying a BufDataset
« Reply #6 on: May 04, 2014, 02:31:33 am »
Mangakissa,

Apologies for the delay in responding but other aspects of life get in the way.....

You are correct. I  miss-read/understood what I was looking at. A close does clear the count.Thank you.

However the original problem (and why I miss understood things) is still there.

I have the following code

 
Code: [Select]
Data.Active := true;
 Databuf.Edit;
 for ctr := 0 to <input-record-count> do
 Begin
      Databuf.append;
      <move data from input-data[ctr] to Databuf record fields>.AsString
     <Log time, ctr, etc>
  end;
Databuf.Close;

If I execute the above code the log time stamps show that the loading of 4497 records takes 1-2 seconds. The following  log entries show the first three records with a time per record of 0-2 millie seconds per record.

Code: [Select]
10:13:29:847 AthsDataModule.Loop: 1 0 202549 0
10:13:29:849 AthsDataModule.Loop: 1 1 157224 1
10:13:29:850 AthsDataModule.Loop: 1 2 202501 2

Similar times per record  at the end of the input.

Code: [Select]
10:13:30:718 AthsDataModule.Loop: 1 4494 201967 4494
10:13:30:718 AthsDataModule.Loop: 1 4495 3965 4495
10:13:30:718 AthsDataModule.Loop: 1 4496 203608 4496

If I execute the code a second time  things are different. Note the times!

Code: [Select]
10:14:20:746 AthsDataModule.Loop: 2 0 202549 0
10:14:30:551 AthsDataModule.Loop: 2 1 157224 1
10:14:31:257 AthsDataModule.Loop: 2 2 202501 2

Ten seconds for the first record, about 700 for the second

Things are consistently slow as the record count increases with each record taking around 65 millie seconds compared to around 1 for the first load.

Code: [Select]
10:18:57:970 AthsDataModule.Loop: 2 4490 200786 4490
10:18:58:035 AthsDataModule.Loop: 2 4491 203276 4491
10:18:58:103 AthsDataModule.Loop: 2 4492 203055 4492
10:18:58:169 AthsDataModule.Loop: 2 4493 203599 4493
10:18:58:234 AthsDataModule.Loop: 2 4494 201967 4494
10:18:58:300 AthsDataModule.Loop: 2 4495 3965 4495
10:18:58:361 AthsDataModule.Loop: 2 4496 203608 4496

Over 4 minutes to process 4497 records. So where is all the time going? What resource is not available? Memory?

I will create a test project and see if I can re-create the problem in that, the current project is to large/complex to upload.

Any suggestions, help appreciated?

Ian

iru

  • Sr. Member
  • ****
  • Posts: 321
Re: Emptying a BufDataset
« Reply #7 on: May 05, 2014, 03:01:46 pm »
Gentlefolk,

I have tried to create a simplified version of my program which could be uploaded and tested without special code.

Used a simple text file and a TimeToStrO(now) for logging.
Used string constants to load fields rather than data from a .CSV file.

Checking the results in the logfile "BufLogFile.txt" I HAVE NOT reproduced my problem. Times on the Try-1 and Try-2  Open-load-close Bufdataset are similar.

No great delays on the second Bufdataset load.

I chase on..... Ian

iru

  • Sr. Member
  • ****
  • Posts: 321
Re: Emptying a BufDataset
« Reply #8 on: May 06, 2014, 08:43:00 am »
Gentlefolk,

I am not having any success with the test  program, speeds are good. Disconnecting the DBgrid from the Datasource improved things a lot.

I have turned back to the original program and placed additional log entries througout the main loop that loads the BufDataset.

The long delays are in caused by the  BufDataset.APPEND.

More  chasing, Ian.

cdbc

  • Hero Member
  • *****
  • Posts: 1026
    • http://www.cdbc.dk
Re: Emptying a BufDataset
« Reply #9 on: May 06, 2014, 08:59:15 am »
Hmmmm, thinking beginUpdate -> EndUpdate on dataset and definitely NO visual controls being updated simultanously  :)

Regards Benny
If it ain't broke, don't fix it ;)
PCLinuxOS(rolling release) 64bit -> KDE5 -> FPC 3.2.2 -> Lazarus 2.2.6 up until Jan 2024 from then on it's: KDE5/QT5 -> FPC 3.3.1 -> Lazarus 3.0

LacaK

  • Hero Member
  • *****
  • Posts: 691
Re: Emptying a BufDataset
« Reply #10 on: May 06, 2014, 03:30:48 pm »
The long delays are in caused by the  BufDataset.APPEND.
do you have any Indexes defined on TBufDataset ?
Do you mean only Append ? or Append followed by Post ?

iru

  • Sr. Member
  • ****
  • Posts: 321
[SOLVED] Re: Emptying a BufDataset
« Reply #11 on: May 07, 2014, 09:42:03 am »
Gentlefolk,

I organised things in the IDE/Project so that I could update and compile units DB.pas,Dataset.inc and BufDataset.pas.

I progressively added time stamped logging at selected locations. I could then compare the flow and timing of the first pass through the code  with the second pass.

Wherever I found a delay in the second pass I would add more logging points. The problem was found in the following BufDataset.pas code:

Code: [Select]
function TCustomBufDataset.GetRecord(Buffer: TRecordBuffer; GetMode: TGetMode; DoCheck: Boolean): TGetResult;

var Acceptable : Boolean;
    SaveState : TDataSetState;

begin
  Result := grOK;
  Logger.Send('BufDataset.pas: GetRecord');
  with FCurrentIndex do
    begin
    repeat
    Acceptable := True;
    Logger.Send('BufDataset.pas: GetRecord.Case GetMode=?');
    case GetMode of
      gmPrior : Result := ScrollBackward;
      gmCurrent : Result := GetCurrent;
      gmNext : begin
               if not CanScrollForward and (getnextpacket = 0) then result := grEOF
               else
                 begin
                 result := grOK;
                  Logger.Send('BufDataset.pas: GetRecord.DoScrollForward');
                 DoScrollForward;
                 end;
               end;
    end;

    if Result = grOK then
      begin
      CurrentRecordToBuffer(buffer);

      if Filtered then
        begin
        FFilterBuffer := Buffer;
        SaveState := SetTempState(dsFilter);
         Logger.Send('BufDataset.pas: GetRecord.DoFilterRecord');
        DoFilterRecord(Acceptable);                                         <<<<<<<<<<<<<<<<<<<<<<<<<<<<< Here
        if (GetMode = gmCurrent) and not Acceptable then
          begin
          Acceptable := True;
          Logger.Send('BufDataset.pas: GetRecord.Set acceptable = true');
          Result := grError;
          end;
        RestoreState(SaveState);
        end;
        Logger.Send('BufDataset.pas: GetRecord.Between ends');
      end
    else if (Result = grError) and doCheck then
      DatabaseError('No record');
    Logger.Send('BufDataset.pas: until Acceptable');
    until Acceptable;     
end;


Woops, 'DoFilterRecord', what filter?????

The problem is me!

During the data load to the BufDataset the input is checked for errors and a flag is set in an errored record.
At the end of the load if there are any errors a small form with a grid displays the errors. The user can then cancel the load and discard the input or accept the input. The 'error' form is closed.

A filter is applied to the BufDataset so only the errored records are displayed.

I forgot to set BufDataset.Filtered := False when closing the error display form. Therefore on the second load a filter was doing something while records were being appended to the BufDatset.

Turn the filter off and the second load is as fast as the first load.

So, learnt some things, now to tidy up and get on with the application......

Ian

« Last Edit: May 07, 2014, 10:09:44 am by iru »

 

TinyPortal © 2005-2018