Recent

Author Topic: Synapse, OAuth2, Imap & Gmail  (Read 7422 times)

Frank

  • Jr. Member
  • **
  • Posts: 69
Synapse, OAuth2, Imap & Gmail
« on: January 22, 2019, 08:55:18 pm »
Hi All,
  Has anyone had any success with Synapse(svn), OAuth2, Imap & Gmail?
  I stumbled across a great jump point by rvk (google_oauth2.pas & example), but that only deals with SMTP. As well I found a Delphi/Win32/Indy10 example. I am looking for a Win/Mac solution using Lazarus/Synapse. I can get Imap to work IF I turn on "Allow Less Secure App Access", but I'd rather not do that... OAuth2 seems to be the way to go.

I really like rvk's approach, it would be nice to continue that, if possible.

Any Ideas? Frank

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: Synapse, OAuth2, Imap & Gmail
« Reply #1 on: January 22, 2019, 09:15:40 pm »
My google_oauth2.pas can also be used to access imap and even google drive.

I use it to put a concept mail via imap into the drafts-folder of gmail, before the user sends it off.

You can use the access_token retrieved via that unit to gain access.

Frank

  • Jr. Member
  • **
  • Posts: 69
Re: Synapse, OAuth2, Imap & Gmail
« Reply #2 on: January 22, 2019, 09:20:40 pm »
@rvk : Thanks, is there a chance I'm using an old example ? Where might I find the most up-to-date? Frank

EDIT: Last recorded edit in 'google_oauth2.pas' was, my version, 2016-12-12.
         I am able to make SMTP/Google drive work with google_oauth2.pas.
« Last Edit: January 22, 2019, 09:40:55 pm by Frank »

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: Synapse, OAuth2, Imap & Gmail
« Reply #3 on: January 22, 2019, 09:49:51 pm »
Yeah, it's been a while since I did anything with the original source.
The latest is at https://github.com/rvk01/google-oauth2

In there I showed with a TmySMTPSend how you should do the AUTH XOAUTH2 with the GetXOAuth2Base64 (which was a combination of EMail and the Access_token) via Gmail SMTP.

With Gmail IMAP it's a similar process. Below is my TmyIMAPSend you can use (which is inherited from the Synapse one).

Code: Pascal  [Select][+][-]
  1. // =============================================================================
  2. type
  3.   TmyIMAPSend = class(TIMAPSend)
  4.   public
  5.     OAuth2: String;
  6.     function Login: Boolean;
  7.     function DoXOAuth2(const Value: string): Boolean;
  8.     function ChallengeError(): string;
  9.   end;
  10.  
  11. function TmyIMAPSend.Login: Boolean;
  12. var
  13.   S: string;
  14. begin
  15.   FSelectedFolder := '';
  16.   FSelectedCount := 0;
  17.   FSelectedRecent := 0;
  18.   FSelectedUIDvalidity := 0;
  19.   Result := false;
  20.   FAuthDone := false;
  21.   if not Connect then
  22.       Exit;
  23.   S := string(FSock.RecvString(FTimeout));
  24.   if Pos('* PREAUTH', S) = 1 then
  25.       FAuthDone := true
  26.   else
  27.     if Pos('* OK', S) = 1 then
  28.       FAuthDone := false
  29.   else
  30.       Exit;
  31.   if Capability then
  32.   begin
  33.     // * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1
  34.     // XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
  35.     // Showmessage(FullResult.Text);
  36.     if Findcap('IMAP4rev1') = '' then
  37.         Exit;
  38.     if FAutoTLS and (Findcap('STARTTLS') <> '') then
  39.       if StartTLS then
  40.           Capability;
  41.   end;
  42.  
  43.   Result := Self.DoXOAuth2(OAuth2);
  44.   // Result := AuthLogin;
  45.  
  46. end;
  47.  
  48. function TmyIMAPSend.DoXOAuth2(const Value: string): Boolean;
  49. var
  50.   S: string;
  51. begin
  52.   S := IMAPcommand('AUTHENTICATE XOAUTH2 ' + Value);
  53.   Result := S = 'OK';
  54.   // Showmessage(S);
  55.   // x := StrToIntDef(Copy(S, 1, 3), 0);
  56.   // Result := (x = 235);
  57. end;
  58.  
  59. function TmyIMAPSend.ChallengeError(): string;
  60. var
  61.   S: string;
  62. begin
  63.   Result := '';
  64.   Sock.SendString('' + CRLF);
  65.   repeat
  66.     S := string(Sock.RecvString(FTimeout));
  67.     if Sock.LastError <> 0 then
  68.         break;
  69.     if Result <> '' then
  70.         Result := Result + CRLF;
  71.     Result := Result + S;
  72.   until Pos('-', S) <> 4;
  73. end;
  74. // =============================================================================

Then you can use this to access IMAP from Google.

Here is an example from my program to push a mail into the Drafts-folder. (It's a subroutine for a larger procedure to send mail via Gmail IMAP, Gmail SMTP, Microsoft Office365 and regular Windows Mapi, so some variables might be unknown in this subroutine standalone)

Code: Pascal  [Select][+][-]
  1.   function SendItViaGMailImap: Cardinal;
  2.   const
  3.     google_client_id = '1111111111-xxxxxxxxxxxxxx.apps.googleusercontent.com';
  4.     google_client_secret = 'xxxxxxxxxxxxxxxxxxxxxxx';
  5.   var
  6.     gOAuth2: TGoogleOAuth2;
  7.     Scopes: GoogleScopeSet;
  8.   var
  9.     Imap: TmyIMAPSend;
  10.     ts: TStringList;
  11.     Uid: int64;
  12.     UidStr: string;
  13.     Ok: Boolean;
  14.     Match: TMatch;
  15.     Url: String;
  16.   begin
  17.     Result := MAPI_E_FAILURE;
  18.     ts := TStringList.Create;
  19.     Imap := TmyIMAPSend.Create;
  20.     gOAuth2 := TGoogleOAuth2.Create(google_client_id, google_client_secret);
  21.     try
  22.  
  23.       Scopes := [GoogleScope.goMail];
  24.       gOAuth2.refresh_token := CipherString(ReadRegistry('GMail', 'refresh_token', ''), 'my_app_name', false);
  25.       gOAuth2.access_token := CipherString(ReadRegistry('GMail', 'access_token', ''), 'my_app_name', false);
  26.       gOAuth2.GetAccess(Scopes, false); // <- DO NOT get from file
  27.       if gOAuth2.email <> '' then
  28.       begin
  29.         DeleteRegistry('GMail', 'Gebruikersnaam');
  30.         DeleteRegistry('GMail', 'Wachtwoord');
  31.         WriteRegistry('GMail', 'refresh_token', CipherString(gOAuth2.refresh_token, 'my_app_name', true));
  32.         WriteRegistry('GMail', 'access_token', CipherString(gOAuth2.access_token, 'my_app_name', true));
  33.       end
  34.       else
  35.       begin
  36.         Result := MAPI_E_INVALID_SESSION;
  37.         Exit;
  38.       end;
  39.  
  40.       // https://developers.google.com/gmail/imap_extensions?csw=1
  41.       Imap.AutoTLS := false;
  42.       // https://www.limilabs.com/blog/oauth2-gmail-imap-service-account
  43.       Imap.Username := '';
  44.       Imap.Password := '';
  45.       Imap.OAuth2 := gOAuth2.GetXOAuth2Base64;
  46.       Imap.TargetHost := 'imap.gmail.com';
  47.       Imap.TargetPort := '993';
  48.       Imap.FullSSL := true;
  49.       Imap.Sock.SSL.SSLType := LT_all;
  50.       // Imap.Sock.SSLDoConnect();
  51.       if not Imap.Login() then
  52.       begin
  53.         ShowMessage('Fout: ' + Imap.ChallengeError);
  54.         Result := MAPI_E_LOGIN_FAILURE;
  55.         Exit;
  56.       end;
  57.  
  58.       (*
  59.         Imap.List('*', ts);
  60.         ShowMessage(ts.Text);
  61.         ShowMessage(Imap.SelectedFolder);
  62.       *)
  63.  
  64.       Ok := Imap.SelectFolder('[Gmail]/Concepten');
  65.       if not Ok then Ok := Imap.SelectFolder('[Gmail]/Drafts');
  66.       if not Ok then
  67.       begin
  68.         Imap.Logout();
  69.         Result := MAPI_E_INVALID_SESSION;
  70.         Exit;
  71.       end;
  72.  
  73.       ts.Text := CreatePlainEML;
  74.       Imap.AppendMess(Imap.SelectedFolder, ts);
  75.       // S5 OK [APPENDUID 628624526 53] (Success)
  76.       Match := TRegEx.Match(Imap.ResultString, '\[APPENDUID (.*?) (.*?)\]');
  77.       if Match.Success and (Match.Groups.Count = 3) then
  78.       begin
  79.         UidStr := Match.Groups[2].Value;
  80.  
  81.         if Imap.IMAPcommand('UID FETCH ' + UidStr + ' (X-GM-MSGID)') = 'OK' then
  82.         begin
  83.           // * 13 FETCH (X-GM-MSGID 1590941195342179459 UID 61)
  84.           Match := TRegEx.Match(Imap.FullResult.Text, '\(X-GM-MSGID (.*?) UID (.*?)\)');
  85.           if Match.Success and (Match.Groups.Count = 3) then
  86.           begin
  87.             UidStr := Match.Groups[1].Value;
  88.             Uid := StrToInt64(UidStr);
  89.  
  90.             Url := 'https://mail.google.com/mail/#drafts?compose=' + lowercase(IntToHex(Uid, 16));
  91.             Url := 'https://mail.google.com/mail?authuser=' + gOAuth2.email + '#drafts/' + lowercase(IntToHex(Uid, 16));
  92.  
  93.             // Url := 'https://mail.google.com/mail/#drafts/' + lowercase(IntToHex(Uid, 16));
  94.             // https://mail.google.com/mail/u/0/#drafts/%23thread-a%3ammiai-xxxx
  95.             // https://mail.google.com/mail/u/0/#drafts?compose=xxx
  96.  
  97.             BrowseURL(Url);
  98.             Result := SUCCESS_SUCCESS;
  99.  
  100.           end;
  101.         end;
  102.       end;
  103.  
  104.       Imap.Logout();
  105.  
  106.     finally
  107.       Imap.Free;
  108.       ts.Free;
  109.       gOAuth2.Free;
  110.     end;
  111.  
  112.   end;

Hope you can get it working.
(source is from my Delphi application but it should work in Lazarus too)

If you can't get it to work I could expand the example on github to include IMAP (when I get some time).

Frank

  • Jr. Member
  • **
  • Posts: 69
Re: Synapse, OAuth2, Imap & Gmail
« Reply #4 on: January 22, 2019, 10:14:36 pm »
@rvk : And presto, It is working !!! Thank you for your help !!! Frank

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: Synapse, OAuth2, Imap & Gmail
« Reply #5 on: January 22, 2019, 10:43:58 pm »
Great.

One note: I got an e-mail recently that Google is updating their policies regarding the scopes you can use and what scopes are needing their approval. The https://mail.google.com/ scope (full access) needs extra approval and you need to provide a privacy policy to include on the authentication screen for the user. You could use the gmail.compose scope (for just accessing drafts and sending mail) but that also need their approval. If you don't ask approval the user gets an extra security screen. I think this will apply from 15 februari onwards. You can see in the authentication-screen-designer if you need such approval.

Quote
In October 2018, we announced that, in January 2019, new Gmail API policies for restricted scopes will go into effect. We want to let you know that, starting today, you can submit your app(s) that use restricted scopes for verification. Please review the full policy and OAuth FAQ for more information including the secure handling requirement.
https://developers.googleblog.com/2018/10/elevating-user-trust-in-our-api.html

Frank

  • Jr. Member
  • **
  • Posts: 69
Re: Synapse, OAuth2, Imap & Gmail
« Reply #6 on: January 23, 2019, 11:59:34 am »
@rvk : Once again thanks, my project is aimed at a closed audience, but it's nice to know in advance.
           By the way, is pop3 possible with a similar approach?

  Frank

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: Synapse, OAuth2, Imap & Gmail
« Reply #7 on: January 23, 2019, 12:20:39 pm »
I'm not sure about POP3. You might not even need to set "allow less secure apps" to access Gmail via POP3.

B.T.W. Instead of setting "allow less secure apps" when accessing Gmail via SMTP you can also create a specific password for your application. You can do that here: https://myaccount.google.com/apppasswords. That way you can use SMTP without the fancy authentication method shown in the github example. It's just what you prefer.

So you have:
Sending mail:
*) SMTP with setting "allow less secure apps" (not advised anymore)
*) SMTP with special created password for specific app (slightly better solution)
*) SMTP with authentication via authorize screen and access_token (see github example)
*) IMAP with authentication via authorize screen and access_token (see above example)
*) Directly pushing mail via Gmail API-calls (see below example)

Reading mail:
*) POP3 (not sure if you need any special setting)
*) IMAP with authentication via authorize screen and access_token
*) Directly reading mail via Gmail API-calls (see documentation)


Direct Gmail API-call (JSON-requests):
The method I've shown earlier used IMAP. For IMAP you need the full access scope (https://mail.google.com/). You can however also use API-calls and push a JSON to the Gmail server. In that case you don't need full access but can use gmail.compose. Here is an example. (I'm not sure if it works completely because it's been a while since I tried this but you see how it's done)

(You see here that a header "Authorization: Bearer" is used for authentication)

Code: Pascal  [Select][+][-]
  1.   function SendItViaGMailAPI: Cardinal;
  2.   const
  3.     google_client_id = '1111111111-xxxxxxxxxxxxxx.apps.googleusercontent.com';
  4.     google_client_secret = 'xxxxxxxxxxxxxxxxxxxxxxx';
  5.   var
  6.     gOAuth2: TGoogleOAuth2;
  7.     Scopes: GoogleScopeSet;
  8.   var
  9.     HTTP: THTTPSend;
  10.     Response: TStringList;
  11.     obj: ISuperObject;
  12.     json: ISuperObject;
  13.     Url: STring;
  14.  
  15.   begin
  16.     Result := MAPI_E_FAILURE;
  17.     gOAuth2 := TGoogleOAuth2.Create(google_client_id, google_client_secret);
  18.     try
  19.  
  20.       Scopes := [GoogleScope.goMail];
  21.       gOAuth2.refresh_token := CipherString(ReadRegistry('GMail', 'refresh_token', ''), 'my_app_name', false);
  22.       gOAuth2.access_token := CipherString(ReadRegistry('GMail', 'access_token', ''), 'my_app_name', false);
  23.       gOAuth2.GetAccess(Scopes, false); // <- DO NOT get from file
  24.       if gOAuth2.email <> '' then
  25.       begin
  26.         DeleteRegistry('GMail', 'Gebruikersnaam');
  27.         DeleteRegistry('GMail', 'Wachtwoord');
  28.         WriteRegistry('GMail', 'refresh_token', CipherString(gOAuth2.refresh_token, 'my_app_name', true));
  29.         WriteRegistry('GMail', 'access_token', CipherString(gOAuth2.access_token, 'my_app_name', true));
  30.       end
  31.       else
  32.       begin
  33.         Result := MAPI_E_INVALID_SESSION;
  34.         Exit;
  35.       end;
  36.  
  37.       // https://developers.google.com/gmail/api/v1/reference/users/drafts/create?hl=nl
  38.       Url := 'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email + '/drafts';
  39.       Response := TStringList.Create;
  40.       HTTP := THTTPSend.Create;
  41.       try
  42.         json := SO;
  43.         // json.S['auth'] := 'oauth2Client';
  44.         // json.S['userId'] := 'me';
  45.         json.S['message.raw'] := CreatePlainEML;
  46.  
  47.         WriteStrToStream(HTTP.Document, ansistring(CreatePlainEML));
  48.         json := nil;
  49.  
  50.         HTTP.MimeType := 'message/rfc822';
  51.         HTTP.Headers.Clear;
  52.         HTTP.Headers.Add('Authorization: Bearer ' + gOAuth2.access_token);
  53.         HTTP.Headers.Add('Accept: application/json');
  54.         if HTTP.HTTPMethod('POST', Url) then
  55.         begin
  56.           Response.LoadFromStream(HTTP.Document);
  57.  
  58.           // gaat momenteel mis
  59.           ShowMessage(Response.Text);
  60.  
  61.           obj := SO(Response.Text);
  62.           Url := 'https://mail.google.com/mail?authuser=' + gOAuth2.email + '#drafts/' + urlencode(obj.S['message.id']);
  63.           BrowseURL(Url);
  64.  
  65.           Result := SUCCESS_SUCCESS;
  66.         end;
  67.  
  68.       finally
  69.         HTTP.Free;
  70.         Response.Free;
  71.       end;
  72.  
  73.     finally
  74.         gOAuth2.Free;
  75.     end;
  76.  
  77.   end;

You can look at the API-documentation at https://developers.google.com/gmail/api/guides/
You see in the example I used upload/gmail/v1/users/userId/drafts as per example here:
https://developers.google.com/gmail/api/v1/reference/users/drafts/create

I'm sure it's possible to read mail too (but in that case you would again need the full access scope).


Frank

  • Jr. Member
  • **
  • Posts: 69
Re: Synapse, OAuth2, Imap & Gmail
« Reply #8 on: January 23, 2019, 12:57:22 pm »
@rvk Thank you for the information, I believe an application specific password might be just the ticket.  I have written "Google Drive API" specific code (& use it in my project), but the temptation to use Synapse/imap/pop3 was there (I have legacy projects that could use it as well)...

Unfortunately I have a day job to attend to, but I'll attempt your suggestions tonight.

And again, Thanks 
     Frank

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Synapse, OAuth2, Imap & Gmail
« Reply #9 on: January 23, 2019, 07:54:17 pm »
I believe an application specific password might be just the ticket.

I use app-specific passwords when accessing Gmail via Indy and it works just fine.  Will work fine in Synapse, too, and you don't need OAuth to use it.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: Synapse, OAuth2, Imap & Gmail
« Reply #10 on: January 23, 2019, 08:01:43 pm »
One reason for using OAuth over app specific password would be user-convenience.
Na usernane and no password to enter in the application.
A app specific password can.only be created by the user manually (as far as I know).

But both methods should work fine.

Frank

  • Jr. Member
  • **
  • Posts: 69
Re: Synapse, OAuth2, Imap & Gmail
« Reply #11 on: January 23, 2019, 08:08:34 pm »
@Remy Lebeau : Thanks. Just setup apppassword for my legacy project & now it works with the code that is probably 10 years old (had to use "Allow Less Secure" before)(personal email program).

@rvk : Thanks, as above... Now you've given me '3' possibilities - pop3/apppassword, imap/OAuth2 & GMail API...  Again, Thank you for your time & expertise.
   
  Frank

 

TinyPortal © 2005-2018