Minimum code for implementing WebView (deprecated) or WKWebView

[MISV]

One thing I noticed and tried...

I just tried to rename
/Users/%name%/LazarusCocoa64/fpcsrc/packages/cocoaint/units/x86_64-darwin
(got prompted which is kinda odd since I am logged in as %name%)

then create new empty folder called
/Users/%name%/LazarusCocoa64/fpcsrc/packages/cocoaint/units/x86_64-darwin
(got prompted which is kinda odd since I am logged in as %name%)

.......

New files got generated with the *make* command and I thought success...

.......

Inside Lazarus I can now navigate to  *WKNavigationDelegateProtocol* declaration (CTRL + mouse click on the identifier in my own source) and I am taken to WKNavigationDelegate.inc (this did not work before)

But I still get compiler error "Identifier not found "WKNavigationDelegateProtocol" 

I have tried in Lazarus "Clean and Build all"

Open to ideas... I feel a bit puzzled right now as how this combination can happen....

[trev]

You have permission and/or ownership issues with your source files which I suspect is due to the way you installed FPC. You need to solve them.

[MISV]

As mentioned, I successfully edited fprsrc source files. I successfully the relevant fpcsrc recompiled them (by manually deleting the old compiled files first)

So as such it would seem I solved those parts. (Or do you disagree? I would have considered this part fully solved otherwise)

At least I have freshly edited fpcsrc source files (where Lazarus editor can navigate around and fine declarations) and freshly compiled fprsrc files (since I had deleted them before the make/install)

...

So I *think* the permission issue got fully solved by that.

Why it is still not working is beyond me.

just thought I could scratch off "permission" issue

[trev]

If there are no more permission errors emitted when you recompile, then yes, that is solved. Sorry, it wasn't clear to me from your post that those issues had been resolved.

[MISV]

Quick note: I believe I solved them specifically for the WKWebView related files by deleting the compiled files. (i.e. by deleting the old compiled files and running new make/install, I saw new compiled files being ... compiled/made ... hence I know they got successfully recompiled. I hope that makes sense. If my logic is wrong and I am missing something with the behavior of make please let me know.)

Anyway, still won't won't in Lazarus so success is minimal at best. Anyhow, I kinda expected to run into all sorts of trouble. I always do on my Mac

[MISV]

Anyway going further in experiment. Maybe helpful for other using FPCDeluxe needing to recompile FPC source:


1)
In finder I select "Get info" for "packages folder" and I see "staff" user does not have read + write ... 

I decide to fix this for the entire "LazarusCocoa64" directory because it is a problem *ALLOVER* that "staff" is write protected

And I recall from earlier ls -l that "staff" was mentioned (I am not into Mac intrincsic)

I do this like
https://support.apple.com/guide/mac-help/change-permissions-for-files-folders-or-disks-mchlp1203/mac
"Apply permissions to all items in a folder or a disk"


2)
I go into
/Users/%name%/LazarusCocoa64/fpcsrc/packages/cocoaint/units/x86_64-darwin
and make sure all is deleted so *new* files can be generated

3)
make clean all FPC=/Users/%name%/LazarusCocoa64/fpc/lib/fpc/3.3.1 OS_TARGET=darwin CPU_TARGET=x86_64 OPT="-XR/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/"
(now gives drastically fewer permission errors, but they are not all solved oddly. But the errors not related to the files in "cocoaint")

4)
we check and this folder is empty
/Users/%name%/LazarusCocoa64/fpcsrc/packages/cocoaint/units/x86_64-darwin


5)
For testing everything is getting compiled correctly inside *ifdef*, we introduce *temporary* compile time bugs in:
/Users/%name%/LazarusCocoa64/fpcsrc/packages/cocoaint/src/webkit/WKNavigationDelegate.pas
followed by running
sudo make install FPC=$PWD/compiler/ppcx64 OS_TARGET=darwin CPU_TARGET=x86_64
so we are 100% sure everything gets compiled correctly.


6)
When ready having tested evertyhing above carefully we run final:
sudo make install FPC=$PWD/compiler/ppcx64 OS_TARGET=darwin CPU_TARGET=x86_64

7)
we check and see new files have been successfully created in
/Users/%name%/LazarusCocoa64/fpcsrc/packages/cocoaint/units/x86_64-darwin


Thus we can completely entirely positively state new compiled units have been created

...

That Lazarus somehow still broken / out of sync I do not understand how can be.

Inside Lazarus I can now navigate to  *WKNavigationDelegateProtocol* declaration (CTRL + mouse click on the identifier in my own source) and I am taken to WKNavigationDelegate.inc

But I still get compiler error "Identifier not found "WKNavigationDelegateProtocol" 

I have tried in Lazarus "Clean and Build all" + manually delete "Lib" folders anywhere related to my own code.


Conclusion sofar: Lazarus broken deep within.
Probability of being correct 10%

Please if anyone has ideas let me know.

[trev]

If you attach what you're trying to compile, I'll give it a whirl with a patched FPC. Let me know which FPC version you would like me to use.

(Note: My attempt may be delayed by Christmas festivities,  my tomorrow being Christmas Eve.)

[MISV]

@trev @Zoë

I can see you both redeclare some delegate methods (Zoë in her code and Trev elsewhere) that use OpaqueCBlock

e.g.

webView_decidePolicyForNavigationAction_decisionHandler

Do I need to rewrite/fix WKNavigationProtocol to replace:

Code: Pascal  [Select][+][-]

1. procedure webView_decidePolicyForNavigationAction_decisionHandler(webView: WKWebView; navigationAction: WKNavigationAction; decisionHandler: OpaqueCBlock); message 'webview:decidePolicyForNavigationAction:decisionHandler:')
2.  

with this

Code: Pascal  [Select][+][-]

1. procedure webView_decidePolicyForNavigationAction_decisionHandler(webView: WKWebView; navigationAction: WKNavigationAction; decisionHandler: WKNavigationDelegateProtocol_actionPolicy_decisionHandler); message 'webview:decidePolicyForNavigationAction:decisionHandler:')
2.  

I think Trev does this. (In other thread where he helps med with CBLOCK documentation) https://forum.lazarus.freepascal.org/index.php/topic,57724.0.html

But in Zoe code the delegate implementor simply defines the implemention like this

Code: Pascal  [Select][+][-]

1. procedure webView_decidePolicyForNaviagtaionAction_decisionHandler(webView: WKWebView; navigationAction: WKNavigationAction; decisionHandler: WKNavigationDelegateProtocol_actionPolicy_decisionHandler);

[MISV]

Another problem Zoe probably fixed in FPC source is WKNavigationDelegateProtocol_actionPolicy_decisionHandler is not defined anywhere I can find. Hmmm.

I guess * OpaqueCBlock* is and can somehow be used as a filler when something is not yet implemented in FPC

[trev]

You can see my OpaqueCBlock approach in this Wiki example which at first glance seems different to Zoe's approach.

[Zoë]

Ah yeah, I must have either missed that or fixed it after I submitted the patch.  Here's the relevant lines from WKNavigationDelegate.inc:

Code: Pascal  [Select][+][-]

1. type
2.   WKNavigationDelegateProtocol_actionPolicy_decisionHandler =
3.     reference to procedure(policy: WKNavigationActionPolicy); cdecl; cblock;
4.  
5. type
6.   WKNavigationDelegateProtocol = objcprotocol external name 'WKNavigationDelegate' (NSObjectProtocol)
7.   optional
8.     procedure webView_decidePolicyForNavigationAction_decisionHandler (webView: WKWebView; navigationAction: WKNavigationAction; decisionHandler: WKNavigationDelegateProtocol_actionPolicy_decisionHandler); message 'webView:decidePolicyForNavigationAction:decisionHandler:';
9.     procedure webView_decidePolicyForNavigationResponse_decisionHandler (webView: WKWebView; navigationResponse: WKNavigationResponse; decisionHandler: OpaqueCBlock); message 'webView:decidePolicyForNavigationResponse:decisionHandler:';
10.     procedure webView_didStartProvisionalNavigation (webView: WKWebView; navigation: WKNavigation); message 'webView:didStartProvisionalNavigation:';
11.     procedure webView_didReceiveServerRedirectForProvisionalNavigation (webView: WKWebView; navigation: WKNavigation); message 'webView:didReceiveServerRedirectForProvisionalNavigation:';
12.     procedure webView_didFailProvisionalNavigation_withError (webView: WKWebView; navigation: WKNavigation; error: NSError); message 'webView:didFailProvisionalNavigation:withError:';
13.     procedure webView_didCommitNavigation (webView: WKWebView; navigation: WKNavigation); message 'webView:didCommitNavigation:';
14.     procedure webView_didFinishNavigation (webView: WKWebView; navigation: WKNavigation); message 'webView:didFinishNavigation:';
15.     procedure webView_didFailNavigation_withError (webView: WKWebView; navigation: WKNavigation; error: NSError); message 'webView:didFailNavigation:withError:';
16.     procedure webView_didReceiveAuthenticationChallenge_completionHandler (webView: WKWebView; challenge: NSURLAuthenticationChallenge; completionHandler: OpaqueCBlock); message 'webView:didReceiveAuthenticationChallenge:completionHandler:';
17.   end;
18.  

OpaqueCBlock is the way cblock callbacks were declared in the headers before the compiler supported them, and is basically just a plain pointer.  You can still use them with the existing headers, but you need to declare the callback elsewhere and do casts as appropriate.

[MISV]

Thank you Zoë and Trev

[MISV]

What is left is:

FWebView_CocoaWKWebView.customUserAgent
is not defined (but I think I can do this myself if neccessary by moving the entire declaration into the unit?)

I need to get
evaluateJavaScript_completionHandler('document.documentElement.outerHTML', ...OpaqueCBlock....)
working so I can retrieve the HTML of the document (probably hardest... anyway to cast this otherwise? Anyway this is by far most important since you can then get content*HTML *after* e.g. AJAX)

I need to get content filter working... Still need to research. But it is not always I want to download images, videos etc. Usuaully I only want HTML/AJAX

[MISV]

@Zoe


Just to understand you correctly concerning CBlock and casting

So I have this

Code: Pascal  [Select][+][-]

1. myWKWebView.evaluateJavaScript_completionHandler('document.documentElement.outerHTML', ...OpaqueCBlock....)

In Lazarus WEKWebView.inc defined as

Code: Pascal  [Select][+][-]

1. procedure evaluateJavaScript_completionHandler(javaScriptString: NSString; completionHandler: OpaqueCBlock)

In Apple help defined as this:

Code: Pascal  [Select][+][-]

1. func evaluateJavaScript(_ javaScriptString: String, completionHandler: ((Any?, Error?) -> Void)? = nil)

I would need to translate this part completionHandler: ((Any?, Error?) -> Void) to Lazarus.

And my new code would then be (using casting)

Code: Pascal  [Select][+][-]

1. myWKWebView.evaluateJavaScript_completionHandler('document.documentElement.outerHTML', OpaqueCBlock(PtrFuncToCompletionHandlingMatchingTranslatedDefinition))

And ]myWKWebView.evaluateJavaScript_completionHandler would call into my completetion handly asynchroneously


.... Kinda a shame WKWebview does not yet contain properly translate the calls suing CBlock

And also not synchroneous versions of calls, e.g.
func evaluateJavaScript(_ javaScriptString: String) async throws -> Any

(Since I run WKWebView for AJAX crawling it is already in its own thread)

[MISV]

If anyone has any idea on how to get evaluateJavascript to work (necessary to grab content and many other things) let me know

I am not sure what is easier, implementing the sync or async version. But the async version has at least been defined in WK in Lazarus

Code: Pascal  [Select][+][-]

1. procedure evaluateJavaScript_completionHandler(javaScriptString: NSString; completionHandler: OpaqueCBlock)
2.  
3. func evaluateJavaScript(_ javaScriptString: String, completionHandler: ((Any?, Error?) -> Void)? = nil)
4.  

So we have this code

Code: Pascal  [Select][+][-]

1.       FWebBrowser.FWebView_CocoaWKWebView.evaluateJavaScript_completionHandler(
2.         'document.documentElement.outerHTML',
3.         nil // need to pass in a completionhandler here that will process output and then through a mechanishm get that returned to our code
4.       );
5.  

Now, I am not too much into Apple API but I guess a try at CBlock callback would be

Code: Pascal  [Select][+][-]

1.   reference to procedure(AAnyPtr: idPtr; AErrorPtr: NSErrorPtr); cdecl; cblock;
2.  

But that is for sure wrong...

Ideally it would be great if the completionHandler code could be closure like... Or a sub procecedure? But I do not think CBlock allows for this?

But then to actually use with the non-fixed FPC source casting would be necessary?

Code: Pascal  [Select][+][-]

1. evaluateJavaScript_completionHandler('document.documentElement.outerHTML', OpaqueCBlock(PtrFuncToCompletionHandlingMatchingTranslatedDefinition))
2.