Recent

Author Topic: TCSVDataset duplicates modified records on Post  (Read 547 times)

Martin Lowry

  • New Member
  • *
  • Posts: 17
TCSVDataset duplicates modified records on Post
« on: September 26, 2022, 04:25:20 pm »
Hi All,

As the subject says :)  Somewhere in the call to Post the record is duplicated and thus two copies appear in the output file, the unmodified original AND a modified copy. TCSVDataset is a TBufDataset descendent so can't trace where it's happening as the write logic is quite complicated. Also the sources are part of FCL.

EDIT: Actually Post does not write to file since the dataset components are disabled. Updates are postponed until Close is called, and then the modified records are duplicated.

Can somebody suggest what I'm missing?

Cheers
« Last Edit: September 26, 2022, 07:31:54 pm by Martin Lowry »

egsuh

  • Hero Member
  • *****
  • Posts: 1009
Re: TCSVDataset duplicates modified records on Post
« Reply #1 on: September 27, 2022, 01:02:51 pm »
Please post some codes.

Thaddy

  • Hero Member
  • *****
  • Posts: 12138
Re: TCSVDataset duplicates modified records on Post
« Reply #2 on: September 27, 2022, 02:34:27 pm »
Plz provide real code.
Your edit is correct: TCsvDataSet, - most if not all datasets - only persist when closed or forced to update.
« Last Edit: September 27, 2022, 02:35:59 pm by Thaddy »
"I am almost 15! You don't have to pick me up every time I go to a party" Well lady, I do... and fix your bike lights.. You know how to do that...

Martin Lowry

  • New Member
  • *
  • Posts: 17
Re: TCSVDataset duplicates modified records on Post
« Reply #3 on: September 27, 2022, 06:30:50 pm »
Okay, I eventually traced the issue into a procedure within TBufDataset from which TCSVDataset descends.

Here's the codestack:
Code: Text  [Select][+][-]
  1. #0 GETDATASETPACKET(0x153bb10, 0xa925010) at ..\..\Extras\SQLITE\bufdataset.pas:3423
  2. #1 SAVETOSTREAM(0x153bb10, 0x14f3030, DFDEFAULT) at ..\..\Extras\SQLITE\bufdataset.pas:3490
  3. #2 SAVETOFILE(0x153bb10, 0x15031c8 'C:\Users\Martin\Documents'..., DFDEFAULT) at ..\..\Extras\SQLITE\bufdataset.pas:3522
  4. #3 DOBEFORECLOSE(0x153bb10) at ..\..\Extras\SQLITE\bufdataset.pas:1440
  5. #4 SETACTIVE(0x153bb10, false) at ..\..\Extras\SQLITE\dataset.inc:1102
  6. #5 CLOSE(0x153bb10) at ..\..\Extras\SQLITE\dataset.inc:1544
  7. #6 FORMCLOSE(0x1546a90, 0x1546a90, CAFREE) at main.pas:168
  8.  

... and the procedure extracted from bufdataset.pas. See my embedded comments marked "//ML". Commenting out line 76 gives my expected behaviour - only the modified data is stored.

Code: Pascal  [Select][+][-]
  1. procedure TCustomBufDataset.GetDatasetPacket(AWriter: TDataPacketReader);
  2.  
  3.   procedure StoreUpdateBuffer(AUpdBuffer : TRecUpdateBuffer; var ARowState: TRowState);
  4.   var AThisRowState : TRowState;
  5.       AStoreUpdBuf  : Integer;
  6.   begin
  7.     if AUpdBuffer.UpdateKind = ukModify then
  8.       begin
  9.       AThisRowState := [rsvOriginal];
  10.       ARowState:=[rsvUpdated];
  11.       end
  12.     else if AUpdBuffer.UpdateKind = ukDelete then
  13.       begin
  14.       AStoreUpdBuf:=FCurrentUpdateBuffer;
  15.       if GetRecordUpdateBuffer(AUpdBuffer.BookmarkData,True,False) then
  16.         repeat
  17.           if CurrentIndexBuf.SameBookmarks(@FUpdateBuffer[FCurrentUpdateBuffer].NextBookmarkData, @AUpdBuffer.BookmarkData) then
  18.             StoreUpdateBuffer(FUpdateBuffer[FCurrentUpdateBuffer], ARowState);
  19.         until not GetRecordUpdateBuffer(AUpdBuffer.BookmarkData,True,True);
  20.       FCurrentUpdateBuffer:=AStoreUpdBuf;
  21.       AThisRowState := [rsvDeleted];
  22.       end
  23.     else // ie: UpdateKind = ukInsert
  24.       ARowState := [rsvInserted];
  25.  
  26.     FFilterBuffer:=AUpdBuffer.OldValuesBuffer;
  27.     // OldValuesBuffer is nil if the record is either inserted or inserted and then deleted
  28.     if assigned(FFilterBuffer) then
  29.       FDatasetReader.StoreRecord(AThisRowState,FCurrentUpdateBuffer);
  30.   end;
  31.  
  32.   procedure HandleUpdateBuffersFromRecord(AFindNext : boolean; ARecBookmark : TBufBookmark; var ARowState: TRowState);
  33.   var StoreUpdBuf1,StoreUpdBuf2 : Integer;
  34.   begin
  35.     if not AFindNext then ARowState:=[];
  36.     if GetRecordUpdateBuffer(ARecBookmark,True,AFindNext) then
  37.       begin
  38.       if FUpdateBuffer[FCurrentUpdateBuffer].UpdateKind=ukDelete then
  39.         begin
  40.         StoreUpdBuf1:=FCurrentUpdateBuffer;
  41.         HandleUpdateBuffersFromRecord(True,ARecBookmark,ARowState);
  42.         StoreUpdBuf2:=FCurrentUpdateBuffer;
  43.         FCurrentUpdateBuffer:=StoreUpdBuf1;
  44.         StoreUpdateBuffer(FUpdateBuffer[StoreUpdBuf1], ARowState);
  45.         FCurrentUpdateBuffer:=StoreUpdBuf2;
  46.         end
  47.       else
  48.         begin
  49.         StoreUpdateBuffer(FUpdateBuffer[FCurrentUpdateBuffer], ARowState);
  50.         HandleUpdateBuffersFromRecord(True,ARecBookmark,ARowState);
  51.         end;
  52.       end
  53.   end;
  54.  
  55. var ScrollResult   : TGetResult;
  56.     SavedState     : TDataSetState;
  57.     ABookMark      : PBufBookmark;
  58.     ATBookmark     : TBufBookmark;
  59.     RowState       : TRowState;
  60.  
  61. begin
  62.   FDatasetReader := AWriter;
  63.   try
  64.     //  CheckActive;
  65.     ABookMark:=@ATBookmark;
  66.     FDatasetReader.StoreFieldDefs(FAutoIncValue);
  67.  
  68.     SavedState:=SetTempState(dsFilter);
  69.     ScrollResult:=CurrentIndexBuf.ScrollFirst;
  70.     while ScrollResult=grOK do
  71.       begin
  72.       RowState:=[];
  73.       CurrentIndexBuf.StoreCurrentRecIntoBookmark(ABookmark);
  74.       // updates related to current record are stored first
  75.   // ML - This actually causes the ORIGINAL record to be saved  (see line 26-29 above)
  76.       HandleUpdateBuffersFromRecord(False,ABookmark^,RowState);
  77.       // now store current record
  78.   // ML - Now the modified data is stored
  79.       FFilterBuffer:=CurrentIndexBuf.CurrentBuffer;
  80.       if RowState=[] then
  81.         FDatasetReader.StoreRecord([])
  82.       else
  83.   // ML - TCSVDataPacketReader.StoreRecord ignores parameters  (see csvdataset.pp line 271)
  84.         FDatasetReader.StoreRecord(RowState,FCurrentUpdateBuffer);
  85.  
  86.       ScrollResult:=CurrentIndexBuf.ScrollForward;
  87.       if ScrollResult<>grOK then
  88.         begin
  89.         if getnextpacket>0 then
  90.           ScrollResult := CurrentIndexBuf.ScrollForward;
  91.         end;
  92.       end;
  93.     // There could be an update buffer linked to the last (spare) record
  94.     CurrentIndexBuf.StoreSpareRecIntoBookmark(ABookmark);
  95.     HandleUpdateBuffersFromRecord(False,ABookmark^,RowState);
  96.  
  97.     RestoreState(SavedState);
  98.  
  99.     FDatasetReader.FinalizeStoreRecords;
  100.   finally
  101.     FDatasetReader := nil;
  102.   end;
  103. end;
  104.  

Having discovered the source it makes me think the duplication was intended as there doesn't appear to be a work around without affecting other functionality.

wp

  • Hero Member
  • *****
  • Posts: 10248
Re: TCSVDataset duplicates modified records on Post
« Reply #4 on: September 27, 2022, 06:56:03 pm »
Without seeing your code, I do not believe that there is such a bug in TBufDataset/TCSVDataset. Please run the attached demo project which I posted for some other thread. There are no duplicated records. Find out what you are doing differently. Modify the demo to show the bug and post it here again.

Martin Lowry

  • New Member
  • *
  • Posts: 17
Re: TCSVDataset duplicates modified records on Post
« Reply #5 on: September 27, 2022, 07:39:00 pm »
Hi wp,

I was able to reproduce the 'bug' by adding the following method to the form:

Code: Pascal  [Select][+][-]
  1. procedure TForm1.FormClose(Sender: TObject; var CloseAction: TCloseAction);
  2. begin
  3.   CSVDataset1.Close;
  4.   CloseAction:=caFree;
  5. end;
  6.  

I ran the test, modified 2 fields in record 2, and closed the form without doing anything further. The original file was rewritten and now has an extra record duplicating the one I modified.
Code: Text  [Select][+][-]
  1. FirstName,LastName,City,Address,Birthdate
  2. Walter,Mellon,Oklahoma City,"1261, Main Street",01/01/1980
  3. Mario,Speedwagon,Hollywood,"1500, Hollywood Blvd",17/12/1982
  4. Mario2,Speedwagon2,Hollywood,"1500, Hollywood Blvd",17/12/1982
  5. Anna,Mull,Los Angeles,"2202, Capitol Square",17/12/1982
  6.  

This appears to happen because the update buffers are not cleared by calling MergeChangeLog as is done in TCustomCSVDataset.SaveToCSVStream. As a workaround the duplication can prevented by adding the MergeChangeLog call in a TCustomCSVDataset.OnBeforeClose event.



wp

  • Hero Member
  • *****
  • Posts: 10248
Re: TCSVDataset duplicates modified records on Post
« Reply #6 on: September 27, 2022, 10:33:25 pm »
Now that I know what to do I can also reproduce it even without your change. Each time when the app is closed after a record has been edited and posted, the un-edited record remains in the database file. Condition is that the database file already exists when the application starts; the way my application is written this is true at the second start, or later. No idea, why this makes a difference...

Replaced the TCSVDataset by a TBufDataset, but I do not see the issue here (attached as "bufdataset_duplicate_records.zip")

Could you file a bug report? Add the test project, describe the issue as well as the steps to reproduce it. Mention the results of your own analysis of the issue.

wp

  • Hero Member
  • *****
  • Posts: 10248
Re: TCSVDataset duplicates modified records on Post
« Reply #7 on: September 27, 2022, 10:49:59 pm »
Here is a simpler test project which shows the issue in a pure FPC program without user interaction: it creates the dataset with a few records, edits and posts one record, closes the dataset and reopens it - after reopening the recordcount has increased by one, and the original version of the edited record is still there.

Martin Lowry

  • New Member
  • *
  • Posts: 17
Re: TCSVDataset duplicates modified records on Post
« Reply #8 on: September 28, 2022, 03:28:27 pm »

 

TinyPortal © 2005-2018