Recent

Author Topic: tbufdataset.CopyfromDataset - strange behaviour in Lazarus trunk version  (Read 5441 times)

kjteng

  • Full Member
  • ***
  • Posts: 211
I noticed that some my code could not copy data properly using copyfromdataset method.
The same code works perfectly if I compile it using the stable version of Lazarus but not in the trunk version.  Please see attached pix and sample code.
What are the new changes/requirement for tBufdataset?
« Last Edit: November 01, 2021, 10:12:04 am by kjteng »

wp

  • Hero Member
  • *****
  • Posts: 10060
Please see attached pix and sample code.
Sample code seems to be missing. OK - you added it.

Since the code of TBufDataset belongs to the FCL it is probably the FPC version which makes the difference. Please specify your Lazarus and, most of all, FPC versions.

I do not understand the screenshots. What is their difference?

At first trying your project with Laz/main+fpc3.2.2 and with Laz 2.0.12/fpc3.2 I saw significant differences between the left and right dbgrids, just like in your second screenshot. But then I noticed that the left grid, upon copying the data, has scrolled to the end while the right grid is at the top of the data. Bringing them in sync removes the differences.
« Last Edit: November 01, 2021, 10:49:57 am by wp »

kjteng

  • Full Member
  • ***
  • Posts: 211
Sorry for not being clear in my previous post.

The first pix (screenshot (27)) is the result from the trunk version whereas the other pix is from the stable version.

In the trunk version, 'PART' column does not display the correct numbers. For example, the original part number was 0 (in the querydataset and table), but shown as -72057594037927934 in the bufdataset.

I am using the following:
   Trunk version2.3.0 date 2021-10-30, FPC ver 3.3.1
   Stable version2.0.12 date 2021-02-21,  FPC ver 3.2.0, SVN 64642

wp

  • Hero Member
  • *****
  • Posts: 10060
Moving to FPC 3.3.1 I see the issue now, and you should report the issue. But the sample project is still too complex for a report, it should be simplified: Remove the dependence on ZEOS, or maybe even move to a simple memory-based dataset as data source. Ideally even the dependence on Lazarus should be removed. Normally, when doing so, additional information about the bug is revealed.

wp

  • Hero Member
  • *****
  • Posts: 10060
I now replaced ZEOS by TSQLite3Connection/TSQLTransaction/TSQLQuery which come with Lazarus, and - voila - the error is gone...

Therefore I'd say that the issue is not an issue of TBufDataset.CopyFromDataset, but an issue somewhere in ZEOS. Please report it to the ZEOS forum.

kjteng

  • Full Member
  • ***
  • Posts: 211
Thank you for your reply.

Your answer led me to further test using other memorydataset. The results are  as follows:
1. zmQueryDataSet (a descendant of TBufDataset) - same issue
2. rxmemdataset - No issue
3. zMemTable (new component in the zeoslib) - No issue.

Based on the above, I think the problem comes from TBufDataset. Is anyone maintaining this component (to whom we can highlight this issue)  currently?


« Last Edit: November 03, 2021, 02:57:56 pm by kjteng »

wp

  • Hero Member
  • *****
  • Posts: 10060
Is anyone maintaining this component (to whom we can highlight this issue)  currently?
Most of the database-related bugs are handled by Michel Van Canneyt. But I am rather sure that he will not feel responsible for the issue when it can only be demonstated on the basis of a ZEOS database...

I tried to further simplify the problem and modified your demo so that it creates a simple ZEOS-DB with a single ftLargeInt column. And this one cannot be copied to a TBufDataset either, the field does appear but is empty; and when the program ends, there is a memory access error. These are indications that the buffers are not set up correctly in the BufDataset.

Two observations:

(1) When I replace the ZQuery by a TMemoryDataset the issue is gone.

(2) My program can use the TBufDataset of FPC 3.2.2 or FPC 3.3.1 (I copied the sources to subdirs of the project). Both versions show the same error with FPC 3.3.1. But, when I compile with FPC 3.2 the error does not appear, neither with the BufDataset of 3.3.1 nor with 3.2.2. This seems to indicate that the issue does not originate in the TBufdataset sources, but somewhere else, probably in unit db or related, or even completely somewhere else in FPC.

wp

  • Hero Member
  • *****
  • Posts: 10060
I think I found the issue. It's the FieldNo property. It seems to be that ZEOS is counting the FieldNo starting at 0, but LCL documentation says that it is a 1-based property (https://www.freepascal.org/docs-html/fcl/db/tfield.fieldno.html), and in fact, there are lots of lines in the FPC database code in which Field.FieldNo-1 is calculated.

Please copy the following code into your test program and replace the  bf1.CopyFromDataset(ZReadOnlyQuery1,true) by CopyFromDataset(ZReadOnlyQuery1, bf1, true). This modified code detects whether the FieldNo is 0-based and increments it in this case (the changed lines are highlighted).

Code: Pascal  [Select][+][-]
  1. procedure CopyFromDataset(ASource, ADest: TDataSet; CopyData: Boolean);
  2. const
  3.   UseStreams = ftBlobTypes;
  4. var
  5.   I: Integer;
  6.   F,F1,F2: TField;
  7.   L1,L2: TList;
  8.   N: String;
  9.   OriginalPosition: TBookMark;
  10.   S: TMemoryStream;
  11.   minFieldNo: Integer;
  12.   fieldNo: Integer;
  13. begin
  14.   ADest.Close;
  15.   ADest.Fields.Clear;
  16.   ADest.FieldDefs.Clear;
  17.  
  18.   minFieldNo := MaxInt;
  19.   for I:=0 to ASource.FieldCount-1 do
  20.   begin
  21.     F := ASource.Fields[i];
  22.     if F.FieldNo < minFieldNo then
  23.       minFieldNo := F.FieldNo;
  24.   end;
  25.  
  26.   for I:=0 to ASource.FieldCount-1 do
  27.   begin
  28.     F := ASource.Fields[I];
  29.     fieldNo := F.FieldNo;
  30.     if minFieldNo = 0 then inc(fieldno);
  31.     TFieldDef.Create(ADest.FieldDefs, F.FieldName, F.DataType, F.Size, F.Required, fieldno);
  32.   end;
  33.   if (ADest is TBufDataset) then
  34.     TBufDataset(ADest).CreateDataset
  35.   else
  36.     raise Exception.Create('Destination dataset type not supported.');
  37.   L1 := nil;
  38.   L2 := nil;
  39.   S := nil;
  40.   if CopyData then
  41.     try
  42.       L1:=TList.Create;
  43.       L2:=TList.Create;
  44.       ADest.Open;
  45.       for I:=0 to ADest.FieldDefs.Count-1 do
  46.       begin
  47.         N := ADest.FieldDefs[I].Name;
  48.         F1 := ADest.FieldByName(N);
  49.         F2 := ASource.FieldByName(N);
  50.         L1.Add(F1);
  51.         L2.Add(F2);
  52.         If (ADest.FieldDefs[I].DataType in UseStreams) and (S=Nil) then
  53.           S := TMemoryStream.Create;
  54.       end;
  55.       OriginalPosition := ASource.GetBookmark;
  56.       try
  57.         ASource.Open;
  58.         ASource.First;
  59.         while not ASource.EOF do
  60.         begin
  61.           ADest.Append;
  62.           for I:=0 to L1.Count-1 do
  63.           begin
  64.             F1 := TField(L1[i]);
  65.             F2 := TField(L2[I]);
  66.             if not F2.IsNull then
  67.               case F1.DataType of
  68.                 ftFixedChar,
  69.                 ftString   : F1.AsString:=F2.AsString;
  70.                 ftFixedWideChar,
  71.                 ftWideString : F1.AsWideString:=F2.AsWideString;
  72.                 ftBoolean  : F1.AsBoolean:=F2.AsBoolean;
  73.                 ftFloat    : F1.AsFloat:=F2.AsFloat;
  74.                 ftAutoInc,
  75.                 ftSmallInt,
  76.                 ftInteger  : F1.AsInteger:=F2.AsInteger;
  77.                 ftLargeInt : F1.AsLargeInt:=F2.AsLargeInt;
  78.                 ftDate     : F1.AsDateTime:=F2.AsDateTime;
  79.                 ftTime     : F1.AsDateTime:=F2.AsDateTime;
  80.                 ftTimestamp,
  81.                 ftDateTime : F1.AsDateTime:=F2.AsDateTime;
  82.                 ftCurrency : F1.AsCurrency:=F2.AsCurrency;
  83.                 ftBCD,
  84.                 ftFmtBCD   : F1.AsBCD:=F2.AsBCD;
  85.               else
  86.                 if (F1.DataType in UseStreams) then
  87.                 begin
  88.                   S.Clear;
  89.                   TBlobField(F2).SaveToStream(S);
  90.                   S.Position := 0;
  91.                   TBlobField(F1).LoadFromStream(S);
  92.                 end
  93.                 else
  94.                   F1.AsString := F2.AsString;
  95.               end;
  96.           end;
  97.           try
  98.             ADest.Post;
  99.           except
  100.             ADest.Cancel;
  101.             raise;
  102.           end;
  103.           ASource.Next;
  104.         end;
  105.       finally
  106.         ASource.GotoBookmark(OriginalPosition); //Return to original record
  107.       end;
  108.     finally
  109.       L2.Free;
  110.       l1.Free;
  111.       S.Free;
  112.     end;
  113. end;

After fixing the FieldNo this way the ZEOS file can be copied successfully.
« Last Edit: November 04, 2021, 10:11:49 pm by wp »

wp

  • Hero Member
  • *****
  • Posts: 10060
Reading some comments in the ZEOS code I found that they intentionally switch from 1-based to 0-based indexes. I reported this thread to the ZEOS forum as an example that this change still has issues (https://sourceforge.net/p/zeoslib/tickets/539/).

kjteng

  • Full Member
  • ***
  • Posts: 211
wp, Thank you for the fix. Now it copydata correctly. I have removed the line 'if CopyData then...'
because it wont compile with this line. What would be the effect without this line?
   

wp

  • Hero Member
  • *****
  • Posts: 10060
Re: tbufdataset.CopyfromDataset - strange behaviour in Lazarus trunk version
« Reply #10 on: November 08, 2021, 09:58:35 am »
What's wrong with it? It compiles here...

Anyway, when "if copyData" is removed you have the same effect as if CopyData were true, i.e. you will copy the fielddefs of the dataset as well as the data. If CopyData were false you would only copy the the fielddefs and have an empty table.

kjteng

  • Full Member
  • ***
  • Posts: 211
Re: tbufdataset.CopyfromDataset - strange behaviour in Lazarus trunk version
« Reply #11 on: November 08, 2021, 12:50:47 pm »
Ok ok. I didn't notice that "copyfromdataset" is a new procedure with parameter. I just copied the body to an event handler. My bad.

 

TinyPortal © 2005-2018