Recent

Author Topic: Tdbf is raising exception "out of memory"  (Read 1320 times)

penpen

  • New Member
  • *
  • Posts: 23
Tdbf is raising exception "out of memory"
« on: March 14, 2019, 04:02:08 pm »
Hi,
when I connect a .dbf database file and try to read a memo field that is empty I always get an outofmemory exception because it just keeps reading "forever".

This is the command I use:

MyString := dbf2.FieldByName('MemoField').AsString;

How can I check if the field is empty, without reading it ?
dbf2.FieldByName('MemoField').isNull doesnt work unfortunately.

Hope you can help

penpen
« Last Edit: March 14, 2019, 04:14:36 pm by penpen »

Handoko

  • Hero Member
  • *****
  • Posts: 3130
  • My goal: build my own game engine using Lazarus
Re: Tdbf is raising exception "out of memory"
« Reply #1 on: March 14, 2019, 06:43:09 pm »
I cannot reproduce the issue. I've just made a test and S := MyDbf.FieldByName('MemoField').AsString; worked correctly even the field is empty. So I'm sure the bug you got was not on the line you showed us.

I've been using TDbf for years on a small program I wrote and use almost everyday, so far it's working good.

Please show us the whole source code so we can inspect it and tell you where you did it wrong.

penpen

  • New Member
  • *
  • Posts: 23
Re: Tdbf is raising exception "out of memory"
« Reply #2 on: March 14, 2019, 08:56:35 pm »
Thank you for your input.

I figured the problem is with the database. I cant share that with you, but it has some Memo Fields where dbf lib just wont stop reading till the "out of memory" exception is raised. It also does that when I call  bytes := MyDbf.FieldByName('MemoField').AsBytes

Some other empty memo fields in the same .dbf file however return NIL.

PS:
I can kind of workaround that using
try
 S := MyDbf.FieldByName('MemoField').AsString;
except
end;
But once these outofmemory memo fields come the program slows down dramatically because its reading many MB of nothing.

Is there a decent way I can debug this ?
« Last Edit: March 14, 2019, 08:59:23 pm by penpen »

sstvmaster

  • Full Member
  • ***
  • Posts: 117
Re: Tdbf is raising exception "out of memory"
« Reply #3 on: March 14, 2019, 09:34:25 pm »
Did you tried a Third party software to test your dbf?

You can check using this:

- https://sourceforge.net/projects/gtkdbfeditor/files/gtkdbfeditor/gtkdbfeditor-1.0.4/
- http://www.alexnolan.net/software/dbf.htm

Maybe this editor can read the memo field
Lazarus 2.0.4 x32
Lazarus 2.1.0 Trunk x32
OS Win 7 32bit

Cyrax

  • Hero Member
  • *****
  • Posts: 745
Re: Tdbf is raising exception "out of memory"
« Reply #4 on: March 14, 2019, 09:52:02 pm »
Thank you for your input.

I figured the problem is with the database. I cant share that with you, but it has some Memo Fields where dbf lib just wont stop reading till the "out of memory" exception is raised. It also does that when I call  bytes := MyDbf.FieldByName('MemoField').AsBytes

Some other empty memo fields in the same .dbf file however return NIL.

PS:
I can kind of workaround that using
try
 S := MyDbf.FieldByName('MemoField').AsString;
except
end;
But once these outofmemory memo fields come the program slows down dramatically because its reading many MB of nothing.

Is there a decent way I can debug this ?

You need to rebuild whole FPC suite with debug info options enabled : -gw2 -godwarfsets -godwarfmethodclassprefix -gl -O-

penpen

  • New Member
  • *
  • Posts: 23
Re: Tdbf is raising exception "out of memory"
« Reply #5 on: March 14, 2019, 11:02:37 pm »
Did you tried a Third party software to test your dbf?

You can check using this:

- https://sourceforge.net/projects/gtkdbfeditor/files/gtkdbfeditor/gtkdbfeditor-1.0.4/
- http://www.alexnolan.net/software/dbf.htm

Maybe this editor can read the memo field

Yes they dont crash at these fields. Except for one DBF editor that is written in lazarus, that one also crashes when reading the memos. Its this one: http://mydbfstudio.altervista.org/
Thats why I believe there is something wrong with the lib.

PS:
Actually gtkdbfeditor just shows numbers or nothing where there should be a memo.

And the other editor opens the memos fine and doesnt crash on the empty ones where lazarus crashes.

PPS:
I did select "count all" in dbfplus and got an outofmemory error as well.
http://prntscr.com/my0y90
« Last Edit: March 14, 2019, 11:25:14 pm by penpen »

Handoko

  • Hero Member
  • *****
  • Posts: 3130
  • My goal: build my own game engine using Lazarus
Re: Tdbf is raising exception "out of memory"
« Reply #6 on: March 15, 2019, 03:35:35 am »
Have you tried to pack and rebuild the dbf using any available tools or this code below?
http://wiki.freepascal.org/Lazarus_Tdbf_Tutorial#Packing_and_rebuilding_the_tables

If the issue was caused by the damage of the dbf, packing it may solve the problem.

Or maybe you can create a new dbf with similar fields and write some code to import the data from the old dbf.

penpen

  • New Member
  • *
  • Posts: 23
Re: Tdbf is raising exception "out of memory"
« Reply #7 on: March 15, 2019, 10:33:04 am »
Have you tried to pack and rebuild the dbf using any available tools or this code below?
http://wiki.freepascal.org/Lazarus_Tdbf_Tutorial#Packing_and_rebuilding_the_tables

If the issue was caused by the damage of the dbf, packing it may solve the problem.

Or maybe you can create a new dbf with similar fields and write some code to import the data from the old dbf.


Just tried it. It also raises this exception "out of memory" on this line Dbf2.PackTable;
http://prntscr.com/my703p

The db is only 900kb.

BrunoK

  • Full Member
  • ***
  • Posts: 174
  • Retired programmer
Re: Tdbf is raising exception "out of memory"
« Reply #8 on: March 15, 2019, 02:08:52 pm »
It is very likely that a blobfield (TMemoField is a TBlobField descendant) has been damaged in the data file during an update operation.
You ABSOLUTELY need to recreate a NEW clean table with the data that can be recovered.

You can "steal" dbf.pas TDbf.CopyFrom procedure and paste it to a father-son program where you will have to do a bit of editing.
Code: Pascal  [Select]
  1. equivalent to lines lines 1720-1733 of dbf.pas
  2.  
  3.         if not lSrcField.IsNull then
  4.         begin
  5.           if lSrcField.DataType = ftDateTime then
  6.           begin
  7.             if FCopyDateTimeAsString then
  8.             begin
  9.               lDestField.AsString := lSrcField.AsString;
  10.               if Assigned(FOnCopyDateTimeAsString) then
  11.                 FOnCopyDateTimeAsString(Self, lDestField, lSrcField)
  12.             end else
  13.               lDestField.AsDateTime := lSrcField.AsDateTime;
  14.           end
  15.           else begin
  16.             if lSrcField.DataType = ftMemo then begin // <-- begin of patch
  17.               try
  18.                lDestField.Assign(lSrcField);
  19.               except
  20.                // maybe log a message, a counter for dropped data or something
  21.                // identifying the damage record
  22.               end
  23.             end
  24.           else
  25.             lDestField.Assign(lSrcField);
  26.         end;                                                              // <-- end of patch
  27.  

If that works, Keep a backup of the old file and try using the regenerated file.
Lazarus trunk r. 59978/03.01.2019 (+/- patches regarding enabled, TScrollBar, TCursorImage). FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903)

penpen

  • New Member
  • *
  • Posts: 23
Re: Tdbf is raising exception "out of memory"
« Reply #9 on: March 17, 2019, 12:12:00 am »
Hey. thanks for the help.

I personally dont think the records are damaged but its more of a bug in the lazarus dbf lib.
I think so since other programmes (unless written in lazarus' dbf lib) open the db just fine and read the memos with no problem.

I can "restore" it to fix it like you said and I probably will since its not very  good if my programme uses like 500mb of ram till the exception gets raised after 15 secs.

wp

  • Hero Member
  • *****
  • Posts: 6154
Re: Tdbf is raising exception "out of memory"
« Reply #10 on: March 17, 2019, 12:42:29 am »
Can you post the dbf file? You will have to zip it to squeeze it into the 250 KB upload limit and to bypass the filetype restriction. If it remains too big upload it to some cloud server and share the link. Or send me a PM with the link if you don't want to publish the file.
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10