TString grid copy and paste UTF-8 Ansi issue

[tr_escape]

Hello LCL team,

I am trying to paste some datas into TStringGrid of lazarus but Turkish characters is not pasting.

I tried to change some code page etc but not working. But if I paste in the cell editing mode it is working.

Is there any another setting?

Best regards

Lazarus 1.9.0 Fpc 3.1.1 (trunk)

[wp]

I just tried with some Turkish text copied from a Turkish website, and it is working here without any problems (Win 10/64 bit, Laz trunk / fpc 3.04 (32 bit)).

Please describe in more detail: "Turkish characters is not pasting" - does this mean that normal ASCII characters (< #128) are pastring correctly? How do the Turkish characters appear in the grid? Upload a screenshot. Are you sure that the text that you paste is UTF8-encoded? (Although: it should be since pasting works for you in edit mode).

Or is nothing pasted at all? How do you paste? By code? Or by keyboard? In the latter case you should know that SHIFT-INSERT is working only in edit mode, use CTRL-V instead.

[tr_escape]

I am using windows 7 64 bit but lazarus is 32 bit.
(I didn't try in Linux for now)

I used ctrl+v only but one of is I select only a cell not in edit mode other is copied one-by-one for each cell with editing mode.

[tr_escape]

Additional info:

I tried in Lubuntu copied from web site it is working but same text in windows 7 not working.

That copied:

Türkçe ya da Türk dili, batıda Balkanlar’dan başlayıp doğuda Hazar Denizi sahasına kadar konuşulan Altay dillerinden biridir. Yaşı, en eski hesaplara göre 8500 olan Türkçe, bugün yaşayan Dünya dilleri arasında en eski yazılı belgelere sahip olan dildir.

https://tr.wikipedia.org/wiki/Türkçe

Edit:
Also I checked diff both sources of lazarus's lcsvutils.pas that LoadFromCSVStream function is same.

[JuhaManninen]

I tried in Lubuntu copied from web site it is working but same text in windows 7 not working.

Copy/paste from a web page does not work either?
Maybe you use -dDisableUTF8RTL. See:
 http://wiki.freepascal.org/Lazarus_with_FPC3.0_without_UTF-8_mode
Otherwise it is difficult to explain.

[wp]

Could you try this experiment: Add a button to the form with this OnClick event handler:

Code: Pascal  [Select][+][-]

1. uses
2.   Lcltype, Lclintf, clipbrd;
3.  
4. procedure TForm1.Button1Click(Sender: TObject);
5. var
6.   i: Integer;
7.   rs: RawByteString;
8.   s: String;
9. begin
10.   StringGrid1.Cells[1, 1] := 'AsText';
11.   StringGrid1.Cells[1, 2] := 'AsHTML';
12.   StringGrid1.Cells[1, 3] := 'Format';
13.   StringGrid1.Cells[1, 4] := 'Raw';
14.  
15.   SetLength(rs, 100);
16.   Clipboard.GetTextBuf(PChar(rs), 100);
17.   s := '';
18.   for i:=1 to Length(rs) do
19.     if rs[i] < #128 then
20.       s := s + rs[i] else
21.       s := s + '.';
22.  
23.   StringGrid1.Cells[2, 1] := Clipboard.AsText;
24.   StringGrid1.cells[2, 2] := Clipboard.GetAsHtml(true);
25.   StringGrid1.Cells[2, 3] := IntToStr(PredefinedClipboardFormat(pcfText));
26.   StringGrid1.Cells[2, 4] := s;
27. end;  

This code pastes the text into cell (2,1), reads the clipboard format identifier used when pasting the text (cell 2,3), and displays the clipboard's raw bytes as ASCII characters in cell (2, 4). If the clipboard contains UTF8-encoded text then two or three dots should be shown in place of the "ü", "ç" and other Turkish characters. The format identifier is 13 in my case.

See my screenshot as an example when pasteing is working correctly. I copied the text to be pasted from your previous post. Please try this also, maybe Excel is doing something weird.

Cell (2,2) is pasted as HTML. Does this work for you? For me it does.

BTW, Windows7 in my VM is behaving correctly either.

[tr_escape]

thank you for your support.

I tried it worked for me also but if I paste from ctrl+v it is not working again.

I think I will not accept ctrl+v on the stringgrid for now. I'll create a popup menu for paste.

[balazsszekely]

What if you do an explicit typecast just before paste?

Code: Pascal  [Select][+][-]

1. procedure TForm1.StringGrid1KeyDown(Sender: TObject; var Key: Word;
2.   Shift: TShiftState);
3. var
4.   Str: String;
5. begin
6.   if (Key = VK_V) and (ssCtrl in Shift) then
7.   begin
8.     if Clipboard.HasFormat(CF_Text) then
9.     begin
10.       Str := ClipBoard.AsText; //this line should do the job(FPC 3.0.0+)
11.       StringGrid1.Cells[StringGrid1.Col, StringGrid1.Row] := Str;
12.     end;
13.   end;
14. end;

[tr_escape]

What if you do an explicit typecast just before paste?

Code: Pascal  [Select][+][-]

1. procedure TForm1.StringGrid1KeyDown(Sender: TObject; var Key: Word;
2.   Shift: TShiftState);
3. var
4.   Str: String;
5. begin
6.   if (Key = VK_V) and (ssCtrl in Shift) then
7.   begin
8.     if Clipboard.HasFormat(CF_Text) then
9.     begin
10.       Str := ClipBoard.AsText; //this line should do the job(FPC 3.0.0+)
11.       StringGrid1.Cells[StringGrid1.Col, StringGrid1.Row] := Str;
12.     end;
13.   end;
14. end;

No it is not changed behaving like as manuel paste as in the picture of posted by mine.

The other issue is Tstringgrid.SavetoFile(filename); function is taking a lot of time because of my table is little bit big.
5000*4 datas take 1 minute 14 sec in debug mode.

I think xml format the reason of slowness so I changed to Tstringgrid.SaveToCSVFile(filename); csv format it takes about 1 sec.

xml data size 792KB csv is 200KB this means data size isn't main reason of this slowness.

Is there any way to use fast this xml file?

[wp]

I tried it worked for me also but if I paste from ctrl+v it is not working again.

These are the sources of TStringGrid.DoPasteFromClipboard (which is called when you press Ctrl-V):

Code: Pascal  [Select][+][-]

1. procedure TCustomStringGrid.DoPasteFromClipboard;
2. begin
3.   // Unpredictable results when a multiple selection is pasted back in.
4.   // Therefore we inhibit this here.
5.   if HasMultiSelection then
6.     exit;
7.  
8.   if EditingAllowed(Col) and Clipboard.HasFormat(CF_TEXT) then begin
9.     SelectionSetText(Clipboard.AsText);
10.   end;
11. end;

This is getting the clipboard by calling Clipboard.AsText - which is working correctly for you (if I understood you correctly).

Then the text is passed to SelectionSetText:

Code: Pascal  [Select][+][-]

1. procedure TCustomStringGrid.SelectionSetText(TheText: String);
2. var
3.   StartCol,StartRow: Integer;
4.   Stream: TStringStream;
5.  
6.   procedure LoadTSV(Fields: TStringList);
7.   var
8.     i, aCol, aRow: Integer;
9.     NewValue: string;
10.   begin
11.     if StartRow<RowCount then begin
12.       aRow := StartRow;
13.       for i := 0 to Fields.Count-1 do begin
14.         aCol := StartCol + i;
15.         if (aCol<ColCount) and not GetColumnReadonly(aCol) then begin
16.           NewValue := Fields[i];
17.           if ValidateOnSetSelection and not ValidateEntry(aCol,aRow,Cells[aCol,aRow],NewValue) then
18.             break;
19.           DoCellProcess(aCol, aRow, cpPaste, NewValue);
20.           Cells[aCol, aRow] := NewValue;
21.         end;
22.       end;
23.       inc(StartRow);
24.     end;
25.   end;
26.  
27. begin
28.   Stream := TStringStream.Create(TheText);
29.   try
30.     StartCol := Selection.left;
31.     StartRow := Selection.Top;
32.     LCSVUtils.LoadFromCSVStream(Stream, @LoadTSV, #9);
33.   finally
34.     Stream.Free;
35.     if ValidateOnSetSelection then
36.       EditingDone;
37.   end;
38. end;

This is done to extract the text into separate fields in case that a cell range is to be pasted. It is calling some CSV handling routine in unit LCSVUtils where, of course, something could go wrong.

Do you observe the same issue when loading a CSV file? Please prepare a CSV file with some Turkish text and load it via StringGrid.LoadFromCSVFile

[tr_escape]

It is worked correctly for normal ways I will explain:

I used two files one of is created with notepad++ UTF8 formatted other is created by excel's csv export file is ANSI.

So if I use excel exported file it isn't working and other created manuelly file is working.

Because excel exporting as ANSI and If I copy from excel result is same.

I used that strings:

TÜRKÇE   türkçe   BÜYÜK   büyük   SESLİ   sesli
SAKİN   sakin   DAĞ   dağ   ŞARKI   şarkı
SAKIN   sakın   TAŞ   taş   ÇÖZÜM   çözüm
MÂN   mânâ   ÖŞÜR   öşür   PİŞİR   pişir
ŞAŞI   şaşı   ÖĞRETMEN   öğretmen   ÇÖĞÜR   çöğür
ŞASE   şase   ÇİĞ   çiğ   ÇÖİŞÜĞI   çöişüğı

[wp]

Strange. You are saying that the grid imports UTF8 text correctly. And some post earlier you said that the text pasted into my test program shows the correct number of dots, i.e. the text provided by the clipboard is UTF8 as well. So, where is the problem?

Can you post a little demo which shows the error? Maybe the bug is somewhere else in your code.
(Only .pas, .lfm, .lpi and .lpr files packed into a common zip).

[tr_escape]

Yes you are correct thats why I create this topic.

I added my codes as attachment. I removed binary files.

[wp]

I don't know how you can paste in there anything at all because the combobox is active in every cell except for the first column. Or do you want to paste into the first column only? In this case you must activate Option goEditing.

[tr_escape]

Wrong project delivered sorry about that I added real project.

Because I tried with few other clean(!) project.