Lazarus

Programming => General => Topic started by: pixelink on May 16, 2019, 01:48:51 am

Title: [SOLVED] SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 01:48:51 am
When I create a *.bat file in Notepad, and use ascii characters or special character to make lines of a box
I run the .BAT file it looks right

If I paste the same code from the .*bat file into SynEdit, SAVE IT as a *.bat file
Then I run the *.bat file, none of the boxes in the bat file look right.

Is this a UTF8 issue??

Here is my good box code in the good working BAT file.

Code: Pascal  [Select]
  1.  
  2. Echo ÉÍ Double-lined ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»
  3. Echo º                                                                                                º
  4. Echo º                                                                                                º
  5. Echo º                                                                                                º
  6. Echo º                                                                                                º
  7. Echo ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ
  8.  
  9.  
  10.  

Here is a screen between the good and bad
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 01:50:56 am
So, basically it looks like it converts my special characters to "?"'s
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 02:13:56 am
What ever the solution may be, my end-game is to save the correct text from the SynFile

I am  using this code for saving...
sDlg means Dialog component

Code: Pascal  [Select]
  1.  
  2. synEdit.Lines.SaveToFile(sDlg.Filename);
  3.  
  4.  
Title: Re: SynEdit for Batch Files
Post by: jamie on May 16, 2019, 03:12:47 am
maybe you could assign the LINES to another TStringList and then use UTF8ToWinCP(YourString);

Lets see if I can vision this..

MyStringList.Assign(UTF8toWinCP(SynEdit.Lines.Text));

MySringList.SaveTofile(….

UTF8ToWInCP is in LazUTF8 unit.
The above is not tested.
Title: Re: SynEdit for Batch Files
Post by: engkin on May 16, 2019, 05:10:30 am
UTF8ToConsole instead of UTF8ToWinCP?
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 01:27:06 pm
Okay... i will look into these suggestions.

Just one q:
Why or can  the SynEdit be setup to except UTF8 characters by default??

Is there some code i can put in Form.Create to make the editor UTF8 compatible before it is even used?
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 02:12:17 pm
After some research it seems this is more of a UniCode issues than just UTF8... not sure though
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 02:32:17 pm
My first initial "Simple" test worked...
Now, I can now put it in a TStringList for SynEdit conversion.

Code: Pascal  [Select]
  1.  
  2. var
  3.  uStr: String;
  4. begin
  5.   uStr:='ßßßßßß';
  6.   showmessage(ConvertEncoding(uStr, 'Ansi', 'UTF-8'));
  7.  
  8.  
  9.  
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 03:07:29 pm
Okay... got the conversion code in place (see below)

It saves it correctly visually at least.

Follow carefully...

1) The *bat code typed in Notepad working correctly in CMD  (see screenshot)
2) Copy the working code to SynEdit and it saves the exact text at least visually perfect. (see screenshot)
3) The "Good" code (from notepad) works when I run the *BAT file.  (see screenshot)
4) The "Bad" code from SynEdit, saved to bat file, and then RUn BAT file... doesn't work  (see screenshot)

So, even if the file saves visually correct, it still doesn't work.

There is something about SynEdit that is not saving to a file just like Notepad saves to a file... what's the difference??

CONVERSION CODE (which visually works):

Code: Pascal  [Select]
  1.  
  2. This code has been removed, I need to fix it.
  3.  
  4.  

SCREENSHOT
Title: Re: SynEdit for Batch Files
Post by: garlar27 on May 16, 2019, 04:03:49 pm
Didn't read any thing but your code:
Code: Pascal  [Select]
  1.  
  2. uStr:=TStringList.Create;
  3.    try
  4.       uStr.Assign(SynEdit.Lines);
  5.    finally
  6.       uStr.Free;
  7.    end;
  8.    uStr.SaveToFile(sDlg.Filename);
  9.  
The code is wrong and should give you an Access Violation. It should be like this:
Code: Pascal  [Select]
  1. uStr:=TStringList.Create;
  2.    try
  3.       uStr.Assign(SynEdit.Lines);
  4.       uStr.SaveToFile(sDlg.Filename);
  5.    finally
  6.       uStr.Free;
  7.    end;
  8.  
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 04:20:47 pm
Didn't read any thing but your code:
Code: Pascal  [Select]
  1.  
  2. uStr:=TStringList.Create;
  3.    try
  4.       uStr.Assign(SynEdit.Lines);
  5.    finally
  6.       uStr.Free;
  7.    end;
  8.    uStr.SaveToFile(sDlg.Filename);
  9.  
The code is wrong and should give you an Access Violation. It should be like this:
Code: Pascal  [Select]
  1. uStr:=TStringList.Create;
  2.    try
  3.       uStr.Assign(SynEdit.Lines);
  4.       uStr.SaveToFile(sDlg.Filename);
  5.    finally
  6.       uStr.Free;
  7.    end;
  8.  

Actually I realized before you posted that I didn't put the conversion part in.
BTW, that code, does work... I get no errors, it just not the conversion code from ANSi to UTF-8 that is missing

Which I am fixing as we "post"
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 04:28:04 pm
So, here is my attempt at converting the SynEdit text from Ansi to UTF-8

I get error on this line...
S:=ConvertEncoding(SynEdit.Lines.Text, 'Ansi', 'UTF-8');

Saying...
Incompatible types: got "AnsiString" expected "TStrings"

CODE THAT DOESN'T WORK
Code: Pascal  [Select]
  1.  
  2. var
  3.   uStr, S : TStrings;
  4.   i: Integer = 0;
  5.  
  6. uStr:=TStringList.Create;
  7. try
  8.    uStr.Assign(SynEdit.Lines);
  9.      for i:= 0 to SynEdit.Lines.Count - 1 do
  10.         S:=ConvertEncoding(SynEdit.Lines.Text[i], 'Ansi', 'UTF-8');
  11.         uStr.Add(S);
  12.      finally
  13.         uStr.Free;
  14.      end;
  15.  
  16.  
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 04:32:18 pm
Found my error issue.

I had "S" declare as a TString and not a string

CORRECTED VAR
Code: Pascal  [Select]
  1.  
  2. var
  3.   uStr: TStrings;
  4.   i: Integer = 0;
  5.   S: String;
  6.  
  7.  
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 04:36:30 pm
Okay... my attempt to convert code compiled and saved the text exactly as it should, but it still doesn't display the double-line properly in the CMD prompt when I run the *.BAT file.

Now, I don't know where to go from here unless I can know why SynEdit doesn't save the exact format as Notepad does.

Because only notepad written BAT files work.
Title: Re: SynEdit for Batch Files
Post by: korba812 on May 16, 2019, 04:57:39 pm
Just one q:
Why or can  the SynEdit be setup to except UTF8 characters by default??
Because the whole LCL is utf8-aware.
btw engkin and jamie were good suggestions, please read:
http://wiki.freepascal.org/Unicode_Support_in_Lazarus (http://wiki.freepascal.org/Unicode_Support_in_Lazarus)
http://wiki.freepascal.org/FPC_Unicode_support (http://wiki.freepascal.org/FPC_Unicode_support)
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 05:29:06 pm
Okay... read the two articles... lot of complexion, very little code examples except one.
This is the one I am trying.

I get error on this line...
ustr.Assign(UTF8ToConsole(SynEdit.Lines.Text));

Saying...
Identifier not found "UTF8ToConsole"

The only uses reference I could find is...
LConvEncoding
I put in my uses, but still get error,
I don't know what other uses I need, the two articles don't help.

Code: Pascal  [Select]
  1.  
  2. uses
  3.    LConvEncoding
  4.  
  5. uStr:=TStringList.Create;
  6. try
  7.    ustr.Assign(UTF8ToConsole(SynEdit.Lines.Text));
  8. finally
  9.    uStr.Free;
  10. end;
  11. uStr.SaveToFile(sDlg.Filename);
  12.  
  13.  
  14.  
Title: Re: SynEdit for Batch Files
Post by: wp on May 16, 2019, 05:42:31 pm
Besides lconvencoding there is another utf8-related unit which you should memorize: lazutf8...

Similarly, for utf16-related functions you have a good chance in unit lazutf16.

And for unicode (=utf8 or utf16) you should know that there is a lazunicode.

Difficult to remember?
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 05:53:53 pm
Besides lconvencoding there is another utf8-related unit which you should memorize: lazutf8...

Similarly, for utf16-related functions you have a good chance in unit lazutf16.

And for unicode (=utf8 or utf16) you should know that there is a lazunicode.

Difficult to remember?

I have all 3 uses (LConvEncoding,lazutf8,lazunicode), and now get this error
Error: Incompatible type for arg no. 1: Got "AnsiString", expected "TPersistent"

Title: Re: SynEdit for Batch Files
Post by: Bart on May 16, 2019, 06:08:55 pm
Code: Pascal  [Select]
  1.  
  2. uses
  3.    LConvEncoding
  4.  
  5. uStr:=TStringList.Create;
  6. try
  7.    ustr.Assign(UTF8ToConsole(SynEdit.Lines.Text));
  8. finally
  9.    uStr.Free;
  10. end;
  11. uStr.SaveToFile(sDlg.Filename);
  12.  

You cannot assign a String (which the SynEdit.Lines.Text is) to a TStrings (which is waht TStringList is).
You can however do
Code: Pascal  [Select]
  1.   uStr.Text := UTF8ToConsole(SynEdit.Lines.Text)

Furthermore, your code would instantly crash, since you call uStr.SaveToFile() after you have just freed uStr.

This might work:
Code: Pascal  [Select]
  1. uses
  2.   LazUtf8; //for Utf8ToConsole function
  3. ...
  4.   uStr:=TStringList.Create;
  5.   try
  6.     ustr.Text := UTF8ToConsole(SynEdit.Lines.Text);
  7.     uStr.SaveToFile(sDlg.Filename);  
  8.   finally
  9.     uStr.Free;
  10.   end;

Bart
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 07:11:50 pm
Okay... makes sense.

Adjusted code... Still don't display right when i run the BAT file
BUT, NO compile errors in code!!!

Code: Pascal  [Select]
  1.  
  2. uStr:=TStringList.Create;
  3. try
  4.    uStr.Text := UTF8ToConsole(SynEdit.Lines.Text)
  5. finally
  6.    uStr.SaveToFile(sDlg.Filename);
  7.    uStr.Free;
  8. end;
  9.  
  10.  
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 16, 2019, 08:10:39 pm
Well, unless someone has a real solution to why SynEdit doesn't save exactly like Notepad, then this project is basically dead for now, at least as far as using SynEdit.

%)
Title: Re: SynEdit for Batch Files
Post by: wp on May 16, 2019, 10:43:59 pm
Stop lamenting, don't blame SynEdit for not using your brain:

SynEdit is a Lazarus control, it uses UTF-8 encoding. Your batch file is encoded as ANSI, but not in one of the Windows code pages because they don't have the line-draw characters. Open the Lazarus character table ("Edit" > "Insert from character map") and switch to tab "ANSI". On my system the combobox shows that the character table is encoded as CP1252, yours is maybe different, but most probably you will not see the double line character in the cell at row 192 and column 13 - this is where the 'Í' sits. But when you scroll the combobox down to item "CP 437 - Original IBM PC Hardware" the line characters will be there, and the 'Í' will have converted to a '═'!

From this you learn that the line characters are on CP437.

So, all you have to do is: after reading the batch file you must convert the text from CP437 to UTF8 (use function CP437ToUTF8 in unit LConvEncoding), and SynEdit (or any other Lazarus control) will show the double-line characters.

Of course, after you edited the text in SynEdit and want to save the changed version you must convert back to CP437 before saving (UTF8ToCP437)

What I want to tell you is that when there are encoding issues for a particular file the first thing you have to do is to find out the encoding of the file instead of blindly trying code that you've seen somewhere. Use your brain!
Title: Re: SynEdit for Batch Files
Post by: jamie on May 17, 2019, 02:08:03 am
@WP,

 Thanks for mentioning those TABLES, I've never investigated that option in Lazarus. I generally only  look for
options I need at the time but that is a great little converter util  :)
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 17, 2019, 02:15:13 am
@WP... thanks for your help.  ::)

Unfortunately I am treading on uncharted waters with all this UTF-8/Ansi/Unicode and stuff.
Remember, I have only been LAZ'in for two years now, and I have mostly just been doing regular characters.

I read the two wiki pages that were suggested, but it was so complicated and all over the place that I couldn't figure what I was supposed to do. Let alone know what to code without example code.

I didn't make SynEdit, so I do not know what it supports or don't... and don't forget... SynEdits documentation plain out rots because there is none.

Its hard to use my brain, when there ain't good info to learn.

Its the same concept as me trying to fly a plane without learning how the airplane flies.

Can't be done unless you have something to read and learn and be taught.

So, I downloaded your ZIp file, but it is missing Form1

If you don't mind, I would appreciate it if I could get that

 :D

Title: Re: SynEdit for Batch Files
Post by: jamie on May 17, 2019, 03:38:22 am
Hmm
Looks simple to me, I ran some test here...
Code: Pascal  [Select]
  1. // Include "LconvEncoding" in the uses list.
  2. procedure TForm1.Button1Click(Sender: TObject);
  3. Var
  4.   S:TStringList;
  5. begin
  6.    S := TStringList.Create;
  7.    S.Text := UTF8ToCP437(SynEdit1.Lines.Text);
  8.    Caption := S.Text;
  9.    //S.SaveTofile(....); // Save here.
  10.    S.Free;
  11. end;                                
  12.  

That is Code page 437, the old IBM page but I am not so sure your console is showing this?
to load the file you do the opposite.

 To get the IBM chars, use the ALT+KeyCode in the synEDit, that will generate  a UTF8..

Code: Pascal  [Select]
  1. procedure TForm1.Button2Click(Sender: TObject);
  2. var
  3.   S:TStringList;
  4. begin
  5.   S := TstringList.Create;
  6.   //S.LoadFromFIle(....);
  7.   S.Text := 'Test Line'+#185; //Fake a load from a CP437 file
  8.   SynEdit1.Text := CP437ToUTF8(S.Text); // a Test line.
  9.   S.free;
  10. end;                                      
  11.  
Title: Re: SynEdit for Batch Files
Post by: lucamar on May 17, 2019, 04:26:51 am
That is Code page 437, the old IBM page but I am not so sure your console is showing this?

The OP is in "Sabattus, ME", which I assume means "Sabattus, Maine, USA", so it's not only possible but very probable that his console is set to CP437.

Anyway, ConsoleToUTF8() (and viceversa, from LazUTF8) should be doing the trick nicely; if it's not then there is a bug somewhere, either in his code or in Lazarus'.

ETA: I couldn't sleep (again! :() so I've made a small test/demo of how it may be done.

The "meat" of it is in the unit batfiles.pas which takes care of loading/saving any text file encoded in the console code-page. The rest (in the main form) is mostly window dressing ... except the part that shows (in the status bar) what exactly Lazarus thinks is your console's CP.

I couldn't test if it works because I'm in Linux (i.e. my console is UTF8 so nothing happens) but you can use it at least to see if the proposed solution works for you.

HTH!
Title: Re: SynEdit for Batch Files
Post by: wp on May 17, 2019, 09:38:19 am
So, I downloaded your ZIp file, but it is missing Form1
Sorry. I work with Laz trunk which uses a different file format in some places, and I always forget to save projects in the old format. In the attachment, there is the corrected project.

Execuse my arrogant tone in the previous message.
Title: Re: SynEdit for Batch Files
Post by: pixelink on May 17, 2019, 12:43:05 pm
Okay.. will give this a shot.

So, yes... my code is cp1252

Title: Re: SynEdit for Batch Files
Post by: pixelink on May 17, 2019, 01:22:41 pm
Okay... it all works!!!

Wow... this was a big learning experience.
I didn't know about the "edit/insert char map, the whole code page based on location thing."

A lot to take in.

It may seem simple, but to a newbie there is so many little parts to sort out.

THANK YOU TO EVERYONE FOR YOUR HELP!!!
 O:-)