Lazarus

Programming => Packages and Libraries => FPSpreadsheet => Topic started by: kjteng on May 06, 2018, 05:06:54 am

Title: user define function
Post by: kjteng on May 06, 2018, 05:06:54 am
Is it possible to create user defined function for fpSpread?  If so, whare can I get any sample code or help document on this subject?
Title: Re: user define function
Post by: wp on May 06, 2018, 09:54:48 am
Read this: http://wiki.lazarus.freepascal.org/FPSpreadsheet#Extending_FPSpreadsheet_by_user-defined_formulas. And there is a sample project in example/other, project "demo_formula_func".
Title: Re: user define function
Post by: kjteng on May 08, 2018, 06:00:33 am
Thanks for the pointer.  I have managed to create a simple formula and it seems working.  Now I have three more question relating to user defined formula:-

1.  What number should I use for the AExcelCode (parameter) in the  RegisterFunction if  there is no similar function in excel. I have tried using 0 and which seems ok but I wish to know what is the correct number to use ?

2. The wiki page refers to sample project demo_ignore_formula. The code in the sample project add boIgnoreFormula to worksheet options.  However boIgnoreFormula  is not a valid identifier.  Shall boReadFormulas be used instead ?

Thanks in advance.
 
Title: Re: user define function
Post by: wp on May 08, 2018, 09:18:55 am
1.  What number should I use for the AExcelCode (parameter) in the  RegisterFunction if  there is no similar function in excel. I have tried using 0 and which seems ok but I wish to know what is the correct number to use ?
I fear I don't understand... fpspreadsheet is a library mainly for reading and writing spreadsheet files. If you implement a completely new function then Excel and certainly also Calc will not be able to open the file any more. Or maybe you are talking of a "macro"? Macros are not supported by fpspreadsheet. Or if you just want to make a calculation without saving the formula: Why don't you make the calculation in Pascal within your program?

2. The wiki page refers to sample project demo_ignore_formula. The code in the sample project add boIgnoreFormula to worksheet options.  However boIgnoreFormula  is not a valid identifier.  Shall boReadFormulas be used instead ?
This is a feature of the development version on svn. The fpspreadsheet wiki refers to this version. If you need the wiki for the version which you actually are using refer to the fpspreadsheet-wiki.chm file in the folder docs of you fpspreashseet installtion.
Title: Re: user define function
Post by: kjteng on May 08, 2018, 10:57:18 am

... If you implement a completely new function then Excel and certainly also Calc will not be able to open the file any more. Or maybe you are talking of a "macro"? Macros are not supported by fpspreadsheet. Or if you just want to make a calculation without saving the formula: Why don't you make the calculation in Pascal within your program?


BasicaIly what I am trying to make a link to other worksheet in the same workbook.  So I created a formula call LINK(address).For example if a cell A1 in Sheet1 contain the formula Link('Sheet2!A2) + Link('Sheet3!B3), the call back function will sum up the two values from Sheet2!A2 and Sheet3!B3 and then store the result in Sheet1!A1.
The function seems to work but I need know the effect of those parameters (AExcelCode, boReadFormulas...).
 
Title: Re: user define function
Post by: wp on May 09, 2018, 12:39:02 am
The latest addition to fpspreadsheet trunk will make this obsolete. Now you can enter in formulas references to single cells in other sheets of the same workbook, such as '=Sheet2!A1+Sheet3!B3. Cell ranges are not yet supported. Reading and writing to xlsx and ods are working, You can also write to xls (Excel 8 only) , but you cannot read xls at the moment.

See this demo:
Code: Pascal  [Select][+][-]
  1. program project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   fpspreadsheet, fpstypes, fpsutils, fpsallformats;
  7.  
  8. const
  9.   FILE_NAME_ODS = 'test.ods';
  10.   FILE_NAME_XLSX = 'test.xlsx';
  11.   FILE_NAME_XLS = 'test.xls';
  12. var
  13.   book: TsWorkbook;
  14.   sheet: TsWorksheet;
  15.   cell: PCell;
  16.   i: Integer;
  17. begin
  18.   book := TsWorkbook.Create;
  19.   try
  20.     book.Options := [boCalcBeforeSaving];
  21.  
  22.     sheet := book.AddWorksheet('Sheet1');
  23.     sheet.WriteFormula(1, 1, 'Sheet2!A3+Sheet3!C1');
  24.     sheet := book.AddWorksheet('Sheet2');
  25.     sheet.WriteNumber(2, 0, 1.0);
  26.     sheet := book.AddWorksheet('Sheet3');
  27.     sheet.WriteNumber(0, 2, 2.0);
  28.  
  29.     book.WriteToFile(FILE_NAME_XLS, true);
  30.     book.WriteToFile(FILE_NAME_XLSX, true);
  31.     book.WriteToFile(FILE_NAME_ODS, true);
  32.   finally
  33.     book.Free;
  34.   end;
  35.  
  36.   book := TsWorkbook.Create;
  37.   try
  38.     book.Options := [boReadFormulas];
  39.     book.ReadFromFile(FILE_NAME_XLSX);  // xls not yet working for reading
  40.     for i:=0 to book.GetWorksheetCount-1 do
  41.     begin
  42.       sheet := book.GetWorksheetByIndex(i);
  43.       WriteLn('Sheet "' + sheet.Name + '":');
  44.       for cell in sheet.Cells do begin
  45.         Write('  ', GetCellString(cell^.Row, cell^.Col), ': Value ', sheet.ReadAsText(cell));
  46.         if HasFormula(cell) then
  47.           Write(', Formula: ', cell^.Formulavalue);
  48.         WriteLn;
  49.       end;
  50.     end;
  51.   finally
  52.     book.Free;
  53.   end;
  54.  
  55.   ReadLn;
  56. end.
Title: Re: user define function
Post by: kjteng on May 09, 2018, 11:24:31 am
Thanks for the new feature. The following problems were encounter after I download and install the SVN:

1. My existing codes couldn't complie any more. (Error message:
fpspread1.pas(173,43) Error: Incompatible type for arg no. 1: Got "AVL_TREE.TAVLTreeNode", expected "LAZ_AVL_TREE.TAVLTreeNode"
laz_avl_tree.pp(1067,19) Hint: Found declaration: FindSuccessor (TAVLTreeNode): TAVLTreeNode;)
Have I missed out something?

2. I managed to start a simple application from scratch without problem. However I notice that the link formula does not get the correct value if it only refers to single cell in another sheet eg. Sheet2!A1 .However Sheet2!A1+0 works! (see attached screen shot);

3. It would be nice if the formula can automatically update if the position of the linked cells changed.  As it is now the linked formula would remains if we insert or delete row/col in the reference sheet.  Perhaps to implement such feature would take time ?
Title: Re: user define function
Post by: kjteng on May 09, 2018, 12:40:42 pm
OK the first question in my previous post is solved as follows:
i. change avl_tree (in uses clause) to avglvltree as the trunk version is using the latter;
ii. change TAVLTreeNode to TAVGLVLTreeNode in all the variable declaration.

Title: Re: user define function
Post by: wp on May 09, 2018, 01:33:24 pm
1. My existing codes couldn't complie any more. (Error message:
fpspread1.pas(173,43) Error: Incompatible type for arg no. 1: Got "AVL_TREE.TAVLTreeNode", expected "LAZ_AVL_TREE.TAVLTreeNode"
laz_avl_tree.pp(1067,19) Hint: Found declaration: FindSuccessor (TAVLTreeNode): TAVLTreeNode;)
OK the first question in my previous post is solved as follows:
i. change avl_tree (in uses clause) to avglvltree as the trunk version is using the latter;
ii. change TAVLTreeNode to TAVGLVLTreeNode in all the variable declaration.
What is your Lazarus/FPC version?
Title: Re: user define function
Post by: kjteng on May 09, 2018, 01:58:20 pm
1.8.2
Title: Re: user define function
Post by: wp on May 09, 2018, 03:30:39 pm
Strange, fpspreadsheet does compile with Laz 1.8.2 without any problems. What is this file fpspread1.pas? I guess this is your file, isn't it? This would explain the issue. The AvlTree units have been rearranged in Lazarus recently. Why do you need them at all?
Title: Re: user define function
Post by: kjteng on May 09, 2018, 04:30:18 pm
Strange, fpspreadsheet does compile with Laz 1.8.2 without any problems. What is this file fpspread1.pas? I guess this is your file, isn't it? This would explain the issue. The AvlTree units have been rearranged in Lazarus recently. Why do you need them at all?
As I have mentioned above, problem 1 has been resolved.  My code could not compile becos I used avl_tree unit (also useds in the fpSpreadsheet.pas which comes with lazarus 1.8.1).  The problem is solved by changing all avl_tree to avglvltree in all my *.pas files.

Yes, fpSpread1.pas is just my file.   In this file I have some test procedure which traverse through all the nodes of worksheet.hyperlinks  (i.e. findfirst,  findsuccessor ...)

Please help me on the second and third issues.
Title: Re: user define function
Post by: wp on May 09, 2018, 07:18:22 pm
In this file I have some test procedure which traverse through all the nodes of worksheet.hyperlinks  (i.e. findfirst,  findsuccessor ...)
The Hyperlinks already have an enumerator in the WorkSheet, like all cell-based lists (Cells, Hyperlinks, Comments, MergedCells); there's no need to dive into the AVLTree:
Code: Pascal  [Select][+][-]
  1. var
  2.   hyperlink: TsHyperlink;
  3.   comment: TsComment;
  4. ...
  5.   for hyperlink in Worksheet.Hyperlinks do ...
  6.   for comment in Worksheet.Comments do ...

I notice that the link formula does not get the correct value if it only refers to single cell in another sheet eg. Sheet2!A1 .However Sheet2!A1+0 works! (see attached screen shot);
Should be fixes along with some other bugs in the current svn version. Now fpspreadsheet can also read xls files (Excel 8 only) with 3d-formulas.

It would be nice if the formula can automatically update if the position of the linked cells changed.  As it is now the linked formula would remains if we insert or delete row/col in the reference sheet.  Perhaps to implement such feature would take time ?
Updating of links when rows/columns are inserted/deleted is already contained, however, operates only on the active sheet. But I think it should be feasible to adjust the routine for any other sheet.

But let me finish the file-related stuff first, i.e. reader/writer for xls/Excel 5, support of 3d cell ranges (Sheet1!A1:B3, Sheet1!A1:Sheet2!B3).
Mo
Title: Re: user define function
Post by: kjteng on May 10, 2018, 01:07:59 am
Thanks. I wasnt aware that
'for... in' is available in free pascal (i came from delphi 5).
Btw please let me know once you have completed the amendment.
Title: Re: user define function
Post by: kjteng on May 10, 2018, 07:06:06 am

The Hyperlinks already have an enumerator in the WorkSheet, like all cell-based lists (Cells, Hyperlinks, Comments, MergedCells); there's no need to dive into the AVLTree:
Code: Pascal  [Select][+][-]
  1. var
  2.   hyperlink: TsHyperlink;
  3.   comment: TsComment;
  4. ...
  5.   for hyperlink in Worksheet.Hyperlinks do ...
  6.   for comment in Worksheet.Comments do ...


should the above code be ?
var
  hyperlink: psHyperlink;
  comment: psComment;
 ...
Title: Re: user define function
Post by: wp on May 10, 2018, 09:36:52 am
Your are right.
Title: Re: user define function
Post by: kjteng on May 11, 2018, 02:13:51 pm
I have installed the latest version of fpSpread. Encountered the following problem for single cell linking. eg Sheet2!B2  Specifically when rows/cols are inserted/deleted in sheet2 and when the new B2 cell is empty, the formula in sheet1 would not be recalculated.  Attached is my test code.

 
Title: Re: user define function
Post by: kjteng on May 14, 2018, 06:03:52 pm
Hi, is  revision 6411 the latest version?

In this revision, additional sheets would be automatically created when I try to open my  xlsx file.

For example, in my book1.xlsx file (uploaded in my previous post) it consists of Sheet1, Sheet2  and Sheet3 with some formulas. When open the file with:-
 
Code: Pascal  [Select][+][-]
  1.    wbSrc1.Workbook.Options := wbSrc1.Workbook.Options + [boReadFormulas];    
  2.    wbSrc1.Workbook.ReadFromFile(OpenDialog1.FileName);  

I get 5 sheets i.e. Sheet1, Sheet2(blank, new sheet), sheet3 (blank, new sheet), sheet21  (which is the original sheet2) and Sheet 31 (which is the original sheet3).

Any thing wrong with my code or did I miss out something?
Title: Re: user define function
Post by: wp on May 15, 2018, 06:45:47 pm
Fixed.

fpspreadsheet now supports "3d references", i.e. cell addresses an other sheets of the same workbook. Cell ranges can be in a single sheet or in a range of sheets.

The syntax is that of Excel, e.g. Sheet3!A1 (single cell), Sheet3!A1:B3 (cell range in single sheet), Sheet3:Sheet6!A1 (the same cell in a range of sheets, Sheet3:Sheet6!A1:B3 (the same cell range in a range of sheets). The syntax of LibreOffice Calc (e.g. Sheet3.A1:Sheet6.B3 is supported internally for reading and writing .ods files, but is not allowed in the TsWorksheet's method WriteFormula. Maybe this will change - I don't know...

The readers and writers of the formats Excel 95, 97 (BIFF5 and BIFF8, .xls) and 2007+ (.xlsx), as well as OpenDocument (.ods) are supporting the new feature. Excel 2 is not supported because it serves only a single worksheet anyway.

References to other files still are not allowed. I don't know whether I will implement this. Use the workaround with the workbook option [boIgnoreFormulas] if you must read such files and write them back -- but you must be aware that the formula engine is disabled with this option.

All provided unit tests are passed, but of course I cannot exclude that some bugs are still hidden somewhere. So please test.

The new feature has consequences on inserting/deleteing of rows and columns - the required adaptions are have not been implemented yet. Moreover I expect some speed pentaly for large workbooks with many sheets and formulas when the workbook option boAutoCalc is set because now the entire workbook has to be recalculated with every change of any cell - the old version only had to recalculate the sheet containing the changed cell.
Title: Re: user define function
Post by: kjteng on May 16, 2018, 08:31:07 am
SVN 6419 - test :
1. Reference to single cell in another sheet wont recalculate when the cell is deleted (with DEL key) or moved - same issue as previous version;
2. sum(B3:B1) does not work;
3. formula SUM(Sheet1:Sheet2!A1) causes access violation error.
Please refer  to attached screen shot
Title: Re: user define function
Post by: kjteng on May 17, 2018, 03:35:19 pm
I have problems when rename worksheet. I have sheet1, sheet2 ...sheet5 in a workbook and sheet5 contains several cells with 3d reference.  (see attached image)
If I rename sheet1 to sht1, or sheet5 to sht5, some formulas will changed and my program crash if I try to key in data into the worksheet.
Please advise.
Title: Re: user define function
Post by: wp on May 21, 2018, 10:20:44 am
If I rename sheet1 to sht1, or sheet5 to sht5, some formulas will changed and my program crash if I try to key in data into the worksheet.
fpspreadsheet, at the moment, stores the string formulas in the cell records. The formulas are parsed whenever needed, the parsed formulas are not stored.

When a sheet is renamed all formulas are parsed to find out the sheetname, and this is where the program crashes because the old sheet in the formula does not exist any more.

Of course the crash could be avoided by some changes in the formula engine. But I think the real problem is that the formulas are not stored in the parsed state. This slows down formula calculation, and adding columns, renaming sheets, copying cells etc. becomes unnecessarily complicated.

Therefore, I decided not to fix this issue at the moment. Instead, I plan to redo the formula handling by fpspreadsheet. I want to set up another tree to store the parsed formulas outside of the cell records. This will speed up calculation and make correction of formulas easier.
Title: Re: user define function
Post by: kjteng on May 26, 2018, 12:00:27 pm
Noted and thanks for you reply.
Title: Re: user define function
Post by: wp on June 02, 2018, 12:02:05 am
Done. I put a lot of effort into restructuring the formula handling. The formulas now are longer stored as strings in the cells (the string format caused successive parsing of the formula), but they are put into a separate tree (Formulas) as string plus parser tree. This makes corrective actions on changes in the workbook structure much easier. The cases "rename worksheet" and "delete worksheet" should work now. Moving of worksheets would be another candidate but this is not yet implemented at all, so I'll postpone this a bit until dust has settled.

Inserting and deleting rows and columns was another issue that you mentioned. But I tested with Excel: If sheet1 contains a formula "SUM(Sheet2:Sheet3!C3)" and I insert a row in sheet2 before the summed cell C3, then the formula in sheet1 does not change (the sum, of course, does). So, I think there's nothing to do here with the formula. Already earlier, recalculation has been changed to operate on the full workbook instead of the worksheet only. Therefore, the formula result should update upon changes in the referenced cells of another sheet.

Please test.

Side note: Since the formula is no longer part of the cell record reading a formula by "cell^.FormulaValue" is no longer valid now. Instead, use the function ReadFormula(cell) or ReadFormulaAsString(cell...) of the worksheet containing the formula (the latter variant can be used to return a localized formula).
Title: Re: user define function
Post by: taazz on June 02, 2018, 02:44:34 am
Done. I put a lot of effort into restructuring the formula handling. The formulas now are longer stored as strings in the cells (the string format caused successive parsing of the formula), but they are put into a separate tree (Formulas) as string plus parser tree. This makes corrective actions on changes in the workbook structure much easier. The cases "rename worksheet" and "delete worksheet" should work now. Moving of worksheets would be another candidate but this is not yet implemented at all, so I'll postpone this a bit until dust has settled.

Inserting and deleting rows and columns was another issue that you mentioned. But I tested with Excel: If sheet1 contains a formula "SUM(Sheet2:Sheet3!C3)" and I insert a row in sheet2 before the summed cell C3, then the formula in sheet1 does not change (the sum, of course, does). So, I think there's nothing to do here with the formula. Already earlier, recalculation has been changed to operate on the full workbook instead of the worksheet only. Therefore, the formula result should update upon changes in the referenced cells of another sheet.

Please test.

Side note: Since the formula is no longer part of the cell record reading a formula by "cell^.FormulaValue" is no longer valid now. Instead, use the function ReadFormula(cell) or ReadFormulaAsString(cell...) of the worksheet containing the formula (the latter variant can be used to return a localized formula).
wow, that must have been at least a moths effort, grats! One stupid question though, is it hard to call readformula(cell) from the cell^.FormulaValue getter?
Title: Re: user define function
Post by: kjteng on June 02, 2018, 05:17:48 am
Thanks for the good work.
Append below are my preliminary test results:

1. formula=SUM(B1:B7)
delete first row: Crash (except when B1 is empty)
delete end row (row 7):  formula changed to SUM(B1:B6) - ok
2. formula=Sheet1!A1   
   formula=Sheet1!A1+Sheet4!A2   
   formula=Sheet4!A2+Sheet1!A1

Rename Sheet1 to sht1 - Formula changed correctly

Delete Sheet1, formula changed to #REF
formula=Sheet1!A1    became #REF
formula=Sheet1!A1+Sheet4!A2    became #REF+Sheet4!A2
formula=Sheet4!A2+Sheet1!A1    became Sheet4!A2+Sheet2!A1

Delete sheet4
formula=Sheet1!A1+Sheet4!A2    became Sheet1!A1+ #REF
formula=Sheet4!A2+Sheet1!A1    became #REF +Sheet1!A1
Title: Re: user define function
Post by: kjteng on June 02, 2018, 08:34:43 am
Done. I put a lot of effort into restructuring the formula handling. The formulas now are longer stored as strings in the cells (the string format caused successive parsing of the formula), but they are put into a separate tree (Formulas) as string plus parser tree. This makes corrective actions on changes in the workbook structure much easier. The cases "rename worksheet" and "delete worksheet" should work now. Moving of worksheets would be another candidate but this is not yet implemented at all, so I'll postpone this a bit until dust has settled.

Inserting and deleting rows and columns was another issue that you mentioned. But I tested with Excel: If sheet1 contains a formula "SUM(Sheet2:Sheet3!C3)" and I insert a row in sheet2 before the summed cell C3, then the formula in sheet1 does not change (the sum, of course, does). So, I think there's nothing to do here with the formula. Already earlier, recalculation has been changed to operate on the full workbook instead of the worksheet only. Therefore, the formula result should update upon changes in the referenced cells of another sheet.

Please test.

Side note: Since the formula is no longer part of the cell record reading a formula by "cell^.FormulaValue" is no longer valid now. Instead, use the function ReadFormula(cell) or ReadFormulaAsString(cell...) of the worksheet containing the formula (the latter variant can be used to return a localized formula).
You are right: for 3D formula like "SUM(Sheet2:Sheet3!C3)", C3 should remain even if new row are inserted in the linked sheet. However, for  non 3D formula "SUM(Sheet2!C1:Sheet2!C3)"  insertion/deletion between the range from C1 to C3 would affect on the formula.
Title: Re: user define function
Post by: wp on June 02, 2018, 09:38:47 am
Thanks for the good work.
Append below are my preliminary test results:

1. formula=SUM(B1:B7)
delete first row: Crash (except when B1 is empty)
delete end row (row 7):  formula changed to SUM(B1:B6) - ok
2. formula=Sheet1!A1   
   formula=Sheet1!A1+Sheet4!A2   
   formula=Sheet4!A2+Sheet1!A1

Rename Sheet1 to sht1 - Formula changed correctly

Delete Sheet1, formula changed to #REF
formula=Sheet1!A1    became #REF
formula=Sheet1!A1+Sheet4!A2    became #REF+Sheet4!A2
formula=Sheet4!A2+Sheet1!A1    became Sheet4!A2+Sheet2!A1

Delete sheet4
formula=Sheet1!A1+Sheet4!A2    became Sheet1!A1+ #REF
formula=Sheet4!A2+Sheet1!A1    became #REF +Sheet1!A1
Please give more complete test conditions. Besides the tested formula you should provide this information:
- In which cell is the formula, and in which sheet?
- If a row/column is deleted, in which sheet is it?
Title: Re: user define function
Post by: wp on June 02, 2018, 09:45:39 am
For 3D formula like "SUM(Sheet2:Sheet3!C3)", C3 should remain even if new row are inserted in the linked sheet. However, for  non 3D formula "SUM(Sheet2!C1:Sheet2!C3)"  insertion/deletion between the range from C1 to C3 would affect on the formula.
Again, I don't know in which sheet these formulas are.

Did you test this? I think fpspreadsheet should reject these formulas because they are no valid Excel syntax. Excel always has "Range of sheets!Range of cells", in this case "SUM(Sheet2:Sheet3!C2:C3)"
Title: Re: user define function
Post by: kjteng on June 02, 2018, 10:39:58 am
Sorry, there are typo in the formula. Please see attached excel file and screen shot for your reference.
Title: Re: user define function
Post by: wp on June 02, 2018, 10:54:04 am
Thanks for the detailed information. Still unclear: In the test case "sum(H1:J10)" you delete rows and columns. In which sheet are they deleted?
Title: Re: user define function
Post by: kjteng on June 02, 2018, 11:56:58 am
Thanks for the detailed information. Still unclear: In the test case "sum(H1:J10)" you delete rows and columns. In which sheet are they deleted?

the formula is in sheet5. Deleted row 10 of sheet5 - program crash. 

Deleted column J of the same sheet - Not crash but formula changed to sum(H1:I10) ..sorry for the wrong information.
Title: Re: user define function
Post by: wp on June 02, 2018, 02:44:36 pm
The first failed test case (delete sheet1) is fixed in trunk.

But I cannot reproduce the other two (delete columns/rows in sheet5) - see attached test program. Could you provide your test program?
Title: Re: user define function
Post by: kjteng on June 02, 2018, 04:20:54 pm
The first failed test case (delete sheet1) is fixed in trunk.

But I cannot reproduce the other two (delete columns/rows in sheet5) - see attached test program. Could you provide your test program?
attached as per your  request
Title: Re: user define function
Post by: wp on June 02, 2018, 06:35:30 pm
Should be fixed now.
Title: Re: user define function
Post by: kjteng on June 03, 2018, 05:03:27 am
Should be fixed now.
Almost there.

1. Refer to the attached PNG, the results for sum(Sheet1!A1:C5) (when row/col are inserted/deleted in sheet1)  are not quite consisitent with that of  sum(H1:J10) <sum within same sheet) and also different from ms-Office. (You can test this with the test program in my previous post.)

2. Reference to invalid sheetname in formula are removed in most cases but not in the case of  sum(Sheet1!A1:C5).  The program crash when I set the formula to sum(Sheet11!A1:C5)  (if Sheet11 does not exist).  See attached test program.
Title: Re: user define function
Post by: kjteng on June 03, 2018, 11:08:20 am
CopyCellsToClipboard does not work now (if it is formula cell), I think this is because formulas are stored in a separate tree now.
PasteCellsFromClipboard may be affected too but I am unable to test it now.
Title: Re: user define function
Post by: wp on June 04, 2018, 09:08:24 pm
Yes, getting closer...
- Fixed clipboard issue
- Fixed update of formulas when rows/cols are inserted/deleted
- Fixed crashes when reading Excel files which contain shared formulas (i.e. formula written into one cell and then dragged into the other cells of a range).

What is left?
Title: Re: user define function
Post by: kjteng on June 05, 2018, 08:31:48 am
Yes, getting closer...
- Fixed clipboard issue
- Fixed update of formulas when rows/cols are inserted/deleted
- Fixed crashes when reading Excel files which contain shared formulas (i.e. formula written into one cell and then dragged into the other cells of a range).

What is left?
Copycells function has some problem. Please see attached images.
Title: Re: user define function
Post by: kjteng on June 06, 2018, 06:11:26 am
Test results: 1. CutCellsToClipboard  2. Installing desiggtime package
Title: Re: user define function
Post by: wp on June 06, 2018, 12:45:43 pm
The compilation error is fixed

I'm not sure about the other one. My concern is
I don't like the behavior of Excel here, sometimes changing, sometimes not. This is confusing. Calc is clear: Never change.
Title: Re: user define function
Post by: kjteng on June 06, 2018, 01:40:25 pm
The compilation error is fixed

I'm not sure about the other one. My concern is
  • this feature has never been implemented in fpspreadsheet and I fear to open another box of Pandora...
  • the behavior of the office applications is inconsistent
    • Excel changes the cell addresses referenced by a formula when the referemced cells are dragged to another place. But this seems to be true only within the current sheet, Excel does not do this in a 3d-Formula, i.e. when the referenced and moved cell is in another sheet, not in the sheet with the formula.
    • Libre/OpenOffice Calc never changes the cell addresses when referenced cells are dragged
    • Excel does not change the cell address when a referenced cell is cut and pasted somewhere else (Your observation in line 18 says: "Formula in B18 became Sheet1!A1+Sheet2°A1", but this is exactly the initial formula - no change).
I don't like the behavior of Excel here, sometimes changing, sometimes not. This is confusing. Calc is clear: Never change.
Should be Ok so long as the user of the component know about this behaviour.

I've just tested Libre office/Calc  (v5.2.7.2) does changed the formula when we move/cut paste the reference cell.
Title: Re: user define function
Post by: wp on June 06, 2018, 03:11:38 pm
LibreOffice v5.4.1.2 (Windows) does not, OpenOffice v4.1.3 neither.

What exactly are you doing?

I do this:
** DRAG AND DROP **
- Cell B3: type value 1
- Cell B5: type vlaue 2
- Cell D3: type formula "=SUM(B3:B5)". ENTER --> value 3 is displayed
- with the mouse I select cells B5 and B6, click somewhere in any of these two cells and drag both cells to a different location
- The sum value in D3 changes to 1. And when I select D3 I still see the old formula

** CUT AND PASTE **
- repeat the first 3 steps of above
- Select cell B5.
- Press CTRL+X --> sum cell value jumps to 1 (because value in B5 is erased)
- Click some other cell
- Press CTRL+V to paste the value back in --> sum cell value remains at 1. The formula in D3 is still SUM(B3:B5)

[EDIT]
if the formula cell C3 contains the expression "=B3+B5" then the behavior is like you describe. But this is even worse: Even within the same application the behavior is inconsistent. Why should "SUM(B3:B5)" behave differently from "B3+B5"?
Title: Re: user define function
Post by: kjteng on June 06, 2018, 05:50:13 pm
LibreOffice v5.4.1.2 (Windows) does not, OpenOffice v4.1.3 neither.

What exactly are you doing?

I do this:
** DRAG AND DROP **
- Cell B3: type value 1
- Cell B5: type vlaue 2
- Cell D3: type formula "=SUM(B3:B5)". ENTER --> value 3 is displayed
- with the mouse I select cells B5 and B6, click somewhere in any of these two cells and drag both cells to a different location
- The sum value in D3 changes to 1. And when I select D3 I still see the old formula

** CUT AND PASTE **
- repeat the first 3 steps of above
- Select cell B5.
- Press CTRL+X --> sum cell value jumps to 1 (because value in B5 is erased)
- Click some other cell
- Press CTRL+V to paste the value back in --> sum cell value remains at 1. The formula in D3 is still SUM(B3:B5)

[EDIT]
if the formula cell C3 contains the expression "=B3+B5" then the behavior is like you describe. But this is even worse: Even within the same application the behavior is inconsistent. Why should "SUM(B3:B5)" behave differently from "B3+B5"?

OIC you were talking about formula like sum(B2:B5) but I was testing A3+B3 and sheet1.A1+Sheet1.A3 (where both cut+paste  and drag move operation would result in change in the formula).
I understand and accept your reasoning for this. For me it's just a matter of getting use to it (because I have been using Excel for decades :-)
Title: Re: user define function
Post by: wp on June 06, 2018, 06:18:44 pm
I would not have thought that the Excel formula engine would make a difference whether it gets a sum as a sequence of binary operations or as a function call. This is really hard to understand and very confusing. Therefore, I'll go the easy way and will not implement this behavior, at least for the time being. So, the rule is for fpspreadsheet:
BTW, thank you for the help in debugging.
Title: Re: user define function
Post by: kjteng on June 09, 2018, 08:28:21 am
New issue:
I get #REF! when I enter formula
                     sheet1!A1    -  in cell A1 of sheet2 (or sheet 3),
                     sheet2!A2    -  in cell A2 of other sheets
but entering    sheet1!A2 in cell A1 of sheet 2 is OK.

If I try to save the workbook, I get circular reference error (see attached)
Title: Re: user define function
Post by: kjteng on June 09, 2018, 08:51:25 am
Another issue:
(i)
I accidentally keyed in an invalid formula (=BB)  into a cell say A1 of sheet3, an error message pop up (see e1.png attached) and the invlaid formula would not be shown in the worksheet.

After that, whenever I key in something to the worksheet (in any cell), I will get the same error message.

However, if i key in a valid 3D formula in cell A1 eg Sheet1!B12, the worksheet would be back to normal (error message wont popup anymore).


(ii)
Similar ourcome if I key in an invalid 3D formula such as Sheet1!AA in cell A1 of sheet 3
Title: Re: user define function
Post by: wp on June 14, 2018, 12:18:49 pm
I accidentally keyed in an invalid formula (=BB)  into a cell say A1 of sheet3, an error message pop up (see e1.png attached) and the invlaid formula would not be shown in the worksheet.

After that, whenever I key in something to the worksheet (in any cell), I will get the same error message.

However, if i key in a valid 3D formula in cell A1 eg Sheet1!B12, the worksheet would be back to normal (error message wont popup anymore).
This is hard stuff. I tried several approaches, but none of them worked out, mostly things are even getting worse...

There are two elemental issues:
Title: Re: user define function
Post by: kjteng on June 20, 2018, 09:37:03 am
This is hard stuff. I tried several approaches, but none of them worked out, mostly things are even getting worse...

There are two elemental issues:
  • While TCustomGrid has a ValidateEntry method which can be used to prevent leaving an erroneous cell, TEdit does not have such a thing. It does have OnEditingDone but this is too late, it is called when the other control is already focused. Does anybody know a cross-platform way to hook into the TEdit method flow to call some kind of "ValidateEntry" method before the control loses focus?
  • Even if this can be solved there is the fundamental question how an error message can be displayed in a messagebox if the error control is not allowed to become unfocused. Or how the application can be closed in the error case by a click on a TButton which also requires a focus change.
Noted your comment above.
I also noticed that you have amended the code in fpspreadsheetgrid.pas and the test result are follows:
Now if I entered =AA  in TsCellEdit box, I will get 'Unknown identifier' error message, and I have two choices: OK or Abort. If I choose OK, the cursor will be back to the tsCellEdit box for me to make changes----this seems to be ok.

However, if I key in =AA directly in the wsGrid (editor on the cell), I will get 'Unknown identifier' error message with an OK button. I am stuck in this error message window because the same message will be display if I click ok button. The only way to get out is to kill the program from task manager.  Please look into this. TQ
 
Title: Re: user define function
Post by: kjteng on June 20, 2018, 12:52:05 pm
Sorry. Just installed trunk 6513. Now I can get error message with ok and abort button in all cases.
However, when program is closed, I got a long message from heaptrc. See attached pix.  I dont really know what it means (though normally I got a shorter message saying that there is 0 unfreed memory block).
Is it ok to ignore this message?
Title: Re: user define function
Post by: wp on June 20, 2018, 07:05:56 pm
Is it ok to ignore this message?
This indicates a lot of memory leaks. Please report the EXACT steps how to reproduce.

Maybe it is the same which I can reproduce with these steps, but which I could not fix so far:
- Use your demo program (the one posted recently in the "Workbook.ReadFromFile" thread"
- Activate HeapTrc in the Project Options.
- Compile and run.
- Click in cell A1
- Enter '1', press ENTER --> the active cell jumps to A2
- Click in the TsCellEdit box
- Enter '=A1', press ENTER --> Cell A2 displays '1'
- Click in any cell, e.g. B2
- Close the program --> a long list of memory leaks appears.
Title: Re: user define function
Post by: wp on June 20, 2018, 07:22:08 pm
This is hard stuff. I tried several approaches, but none of them worked out, mostly things are even getting worse...

There are two elemental issues:
  • While TCustomGrid has a ValidateEntry method which can be used to prevent leaving an erroneous cell, TEdit does not have such a thing. It does have OnEditingDone but this is too late, it is called when the other control is already focused. Does anybody know a cross-platform way to hook into the TEdit method flow to call some kind of "ValidateEntry" method before the control loses focus?
  • Even if this can be solved there is the fundamental question how an error message can be displayed in a messagebox if the error control is not allowed to become unfocused. Or how the application can be closed in the error case by a click on a TButton which also requires a focus change.
Noted your comment above.
I also noticed that you have amended the code in fpspreadsheetgrid.pas and the test result are follows:
Now if I entered =AA  in TsCellEdit box, I will get 'Unknown identifier' error message, and I have two choices: OK or Abort. If I choose OK, the cursor will be back to the tsCellEdit box for me to make changes----this seems to be ok.

However, if I key in =AA directly in the wsGrid (editor on the cell), I will get 'Unknown identifier' error message with an OK button. I am stuck in this error message window because the same message will be display if I click ok button. The only way to get out is to kill the program from task manager.  Please look into this. TQ
I fixed a lot of the navigation issues in the error case. This is what should work:
- Use the in-place editor of the grid to type an erroneous formula (e.g. '=A')
- Navigate to another cell by click, arrow keys, or complete editing by ENTER key --> the incorrect formula is detected, the parser's error message is displayed, and navigation is reverted, the grid is in the erroneous cell in edit mode again.

This is what is not working:
(1)
-  Use the in-place editor of the grid, type an erroneous formula
- Click into the TsCellEdit box. The error message appears, but the cell is erased - this is not good.

(2)
- Use the TsCellEdit box to type an erroneous formula
- When the error message comes up, press ENTER to close it (don't click "OK") -> the box comes up again and again (However, if you click OK with the mouse, the message box does not appear again.

(3)
- Use the TsCellEdit box to type an erroneous formula
- After typiing don't press ENTER, but click into the cell. The formula is erased. The error message box comes up. Press ENTER or click OK to close the error message. A selection rectangle appears in the grid which follows the mouse. You must click anywhere to return to normal operation.
Title: Re: user define function
Post by: kjteng on June 21, 2018, 11:40:59 am
(0)
Quote
- Use the in-place editor of the grid to type an erroneous formula (e.g. '=A')
- Navigate to another cell by click, arrow keys, or complete editing by ENTER key --> the incorrect formula is detected, the parser's error message is displayed, and navigation is reverted, the grid is in the erroneous cell in edit mode again.

- I got the same result here


(1)
Quote
-  Use the in-place editor of the grid, type an erroneous formula
- Click into the TsCellEdit box. The error message appears, but the cell is erased - this is not good.

I got the same result here

Note that the cell is not erased but reverted to its original value. If the cell contains 123 before the errorneous formula is entered,  it will revert to 123 after the error message appears. i.e. the user will be editing the cell containing 123 now.  If the cell was empty, it would revert to empty cell (thus the formula appeared to be erased).

This above test result is consistent with the following cases:
(1a)
Goto cell A1, in tsEdit key in =aa then click into cell A1 in the grid. Click OK when error message appears. The cell will revert to the original value of A1. 

(1b)
Goto cell A1, in tsEdit key in =aa then click into cell C1 in the grid. Click OK when error message appears. The cell will revert to the original value of C1.
 
I think the above behaviour is acceptable.

(2)
Quote
- Use the TsCellEdit box to type an erroneous formula
- When the error message comes up, press ENTER to close it (don't click "OK") -> the box comes up again and again (However, if you click OK with the mouse, the message box does not appear again.

I got the same result here


(3)
Quote
- Use the TsCellEdit box to type an erroneous formula
- After typiing don't press ENTER, but click into the cell. The formula is erased. The error message box comes up. Press ENTER or click OK to close the error message. A selection rectangle appears in the grid which follows the mouse. You must click anywhere to return to normal operation.
Same result but see my comment in (1) above


(4) Goto cell A1, do steps in (0) above. When the erroneous cell is back to edit mode, change the formula to =B1
Now we can see the formula B1 in tsEdit whenever we are in cell A1, however cell A1 would not show any value even if B1 contains a valid number.

Same thing would happen if you key in erroneous formula in TsCellEdit then changed it to a valid formula.

(5) At times, heaptrc shows there is unfreed memory block, but I am yet to find the way to reproduce this.
Title: Re: user define function
Post by: wp on June 21, 2018, 12:50:18 pm
At times, heaptrc shows there is unfreed memory block, but I am yet to find the way to reproduce this.
I fixed a memory leak related to the new handling of formulas.
Title: Re: user define function
Post by: kjteng on June 22, 2018, 08:15:01 am
(Y)
Title: Re: user define function
Post by: wp on June 25, 2018, 12:04:24 am
I accidentally keyed in an invalid formula (=BB)  into a cell say A1 of sheet3, an error message pop up (see e1.png attached) and the invlaid formula would not be shown in the worksheet.

After that, whenever I key in something to the worksheet (in any cell), I will get the same error message.

However, if i key in a valid 3D formula in cell A1 eg Sheet1!B12, the worksheet would be back to normal (error message wont popup anymore).
This is hard stuff. I tried several approaches, but none of them worked out, mostly things are even getting worse...
Maybe I've got an operating solution now - it based on the old Borland article https://community.embarcadero.com/article/technical-articles/149-tools/12766-validating-input-in-tedit-components and looks ok to me. Now it is no longer possible to exit the CellEdit or edited WorksheetGrid cell while an entered formula is not valid.
Title: Re: user define function
Post by: kjteng on June 25, 2018, 04:21:26 am
Thanks for the info. Congratulation.
Title: Re: user define function
Post by: kjteng on June 25, 2018, 12:43:40 pm
Quote
Maybe I've got an operating solution now - it based on the old Borland article https://community.embarcadero.com/article/technical-articles/149-tools/12766-validating-input-in-tedit-components and looks ok to me. Now it is no longer possible to exit the CellEdit or edited WorksheetGrid cell while an entered formula is not valid.

Tested. Works.
One  minor comment: on returning to celledit or inplace editor (after closing the error message), sometime the whole edit text is selected, something it is not selected but the cursor remains at its original position.  It seems that this is dependant on (1) how I ended the edit (by pressing enter key, click on grid cell or click on cellEdit); and (ii) how I responded to the error message (press enter key or click OK). 

I tried to but still have not figured out the behaviour pattern.
Title: Re: user define function
Post by: wp on June 25, 2018, 01:58:53 pm
I know. The current revision seems to have a more consistent editor behavior.
Title: Re: user define function
Post by: kjteng on June 26, 2018, 03:29:54 pm
TsCellEdit accept edited text only if the user press enter key and not if the user click on grid. Is there a property need to be set to change this behaviour?
Title: Re: user defined function
Post by: kjteng on June 26, 2018, 04:03:50 pm
Problem with 'built-in' function:

ROUND(1.2345,2)  returns 1.23 but
ROUND(a1,b1) returns 1.00 where a1=1.2345 and b1=2

ISBLANK(" ") returns TRUE but
ISBLANK(A1) returns FALSE even if A1 contains only space/spaces

FACT(n) returns correct result if n is a number constant
FACT(A1) returns #VALUE!


3D formula:
countif(sheet2!A1: A5,">10") return same result as countif(A1: A5,">10")
Same problem with countif, countblank etc


I have not tested all the function. Just tested on a few that I used frequently.
Title: Re: user defined function
Post by: wp on June 26, 2018, 07:22:55 pm
ROUND(1.2345,2)  returns 1.23 but
ROUND(a1,b1) returns 1.00 where a1=1.2345 and b1=2

FACT(n) returns correct result if n is a number constant
FACT(A1) returns #VALUE!
Fixed along with some similar ones.

ISBLANK(" ") returns TRUE but
ISBLANK(A1) returns FALSE even if A1 contains only space/spaces
The correct result must be FALSE, only if the cell is not allocated or contains only formatting the result is TRUE. Fixed the cases ISBLANK(" ") and ISBLANK("") which both return FALSE in Excel/Calc.

countif(sheet2!A1: A5,">10") return same result as countif(A1: A5,">10")
Fixed. It did not find an exact specification, but my experiments with Excel/Calc show that this type of formulas does not support a sheet range here, i.e. '=COUNTIF(Sheet1:Sheet2!A1:B10, ">10")' is not allowed. Therefore I am returning an error in such a case. A single sheet like in your example, however, is correct.
Title: Re: user define function
Post by: kjteng on July 01, 2018, 08:14:38 am
When autocalc is off :-
WorkSheetGrid would display blank cell after a cell formula is keyed in /edited.

In similar situation, ExcelCalc would show the calculated value (but not updated subsequently when the related cells is modified).
Title: Re: user define function
Post by: wp on July 01, 2018, 10:19:26 am
I never understood what this partial calculation mode would be good for. It will display inconsistent results somewhere in the end after all.

No, I will not do this. But your post gave me the idea to display the formula in the cells when AutoCalc is off.
Title: Re: user define function
Post by: kjteng on July 01, 2018, 02:55:08 pm
I never understood what this partial calculation mode would be good for. It will display inconsistent results somewhere in the end after all.

No, I will not do this. But your post gave me the idea to display the formula in the cells when AutoCalc is off.

Though 'partial calculation' may not show the correct value, it serves two purposes to end user:
1. It let me know that the cell is not a blank cell (without navigating to the particular cell);
2. It helps to me check if I have entered a wrong formula. For example if A1=1 and B1=2, if I keyed into A1+B1 into C1, I would expect 3 to appear in C1. If C1 shows other value, immediately I would know I have keyed in wrong formula.
Also users may be confused if some formula cells (thosed created before the autocalc is turned off)  shows value and some show blank.
I was also confused as the formula cell (with value) suddenly became blank cell when I accidentally entered into edit mode and then exit edit mode (by clicking on other cell).
IMHO showing blank is not a good choice. Your idea of showing formula might be a workable alternative (and may also sovle my next question below).

When autorecalc is off with goAlwayShowEditor option,  the inplace editor will always show blank (or the value displayed in cell) . To edit the formula, I have to press ESC follwed by F2 key?
Title: Re: user define function
Post by: wp on July 01, 2018, 03:29:23 pm
When autorecalc is off with goAlwayShowEditor option,  the inplace editor will always show blank (or the value displayed in cell) . To edit the formula, I have to press ESC follwed by F2 key?
I don't understand. Did you already try the new version? I never use goAlwaysShowEditor, even without it you can start typing immediately once the grid is focused. The only important option is goEditing which must be active to allow editing at all. The annoying feature of the LCL grids, however, is, that the left/right arrow keys end edit mode and move to the previous/next cell if the cursor is at the first/last character of the edited cell.
Title: Re: user define function
Post by: kjteng on July 01, 2018, 05:26:54 pm
When autorecalc is off with goAlwayShowEditor option,  the inplace editor will always show blank (or the value displayed in cell) . To edit the formula, I have to press ESC follwed by F2 key?
I don't understand. Did you already try the new version? I never use goAlwaysShowEditor, even without it you can start typing immediately once the grid is focused. The only important option is goEditing which must be active to allow editing at all. The annoying feature of the LCL grids, however, is, that the left/right arrow keys end edit mode and move to the previous/next cell if the cursor is at the first/last character of the edited cell.
Just install SVN6542.  Works as expected. Thanks.
Title: Re: user define function
Post by: kjteng on July 02, 2018, 10:44:07 am
I cannot WriteNumber(r1, c1, 123.45) if cell R1C1 already contain a formula. 

However, GetCell(r1, c1).^.NumberValue := 123.45 can overwrite the existing formula with 123.45


WriteFormular(r1, c1,'');  //or WriteBlank(r1, c1)
WriteNumber(r1, c1, 123.45)  works too.
 
(Same for WriteText method).

May I know what is the correct way to change the cell content ?

I also tried to clear the cell content first by EraseCell method but found that erasecell only take effect after I clicked on another cell.  Is this the normal behaviour?
Title: Re: user define function
Post by: wp on July 05, 2018, 02:57:17 pm
I cannot WriteNumber(r1, c1, 123.45) if cell R1C1 already contain a formula. 
Should be fixed in current trunk.
Title: Re: user define function
Post by: wp on July 05, 2018, 03:12:05 pm
I also tried to clear the cell content first by EraseCell method but found that erasecell only take effect after I clicked on another cell.  Is this the normal behaviour?
EraseCell is a low-level function and does not perform notification. Use DeleteCell instead. I know it is hard to tell which one is the correct method...
Title: Re: user define function
Post by: kjteng on July 06, 2018, 02:41:18 pm
EraseCell is a low-level function and does not perform notification. Use DeleteCell instead. I know it is hard to tell which one is the correct method...
Thank you master WP for the fix.

I tried to install laz_fpspreadsheet_crypto.lpk but got this 'packages not found' error (see attached)
May I ask where can I find the dcpcyt package ?
Title: Re: user define function
Post by: wp on July 06, 2018, 02:58:56 pm
The easiest way is to install dcpcrypt via Online Package Manager.

You only need the crypto package if you want to read Excel 2007 files (exactly this version, not just xlsx in general) in which passwords to remove workbook/worksheet protection are used (http://wiki.lazarus.freepascal.org/FPSpreadsheet#Passwords). It does not open generally encrypted files! So, its use is rather limited.

You may want to read https://forum.lazarus.freepascal.org/index.php/topic,36075.msg240803.html to learn about the background why it is there. I did not put any further work into the encryption thing, thus I am not 100% sure if the information provided is correct.
Title: Re: user define function
Post by: kjteng on July 06, 2018, 04:11:11 pm
The easiest way is to install dcpcrypt via Online Package Manager.

You only need the crypto package if you want to read Excel 2007 files (exactly this version, not just xlsx in general) in which passwords to remove workbook/worksheet protection are used (http://wiki.lazarus.freepascal.org/FPSpreadsheet#Passwords). It does not open generally encrypted files! So, its use is rather limited.

You may want to read https://forum.lazarus.freepascal.org/index.php/topic,36075.msg240803.html to learn about the background why it is there. I did not put any further work into the encryption thing, thus I am not 100% sure if the information provided is correct.
Thanks for the link and useful info.  After reading and with better understanding of the package, I think I do not need to install crypto.
TinyPortal © 2005-2018