Recent

Author Topic: Check the windows desktop to send mouse clicks  (Read 10269 times)

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Check the windows desktop to send mouse clicks
« on: August 28, 2014, 11:43:25 pm »
Due to my job, i need make many mouse clicks in some predefined places of my screen every few seconds.

The task is easy: when a label appears in the app i use to work, i must click that label. Im trying to imagine if i could create a lazarus app which detects when the label appears and makes auto-clicks.

If i could get a screenshot of my desktop every 20 seconds, i could check if the pixel at x,y have the color of the label and then send a mouse click to that position. My app could be run minimized in the system tray.

What i  need help is:

1- How i get screenshots from my whole desktop? Even better if i could get the pixel color in a coordinate x,y directly

2- How i could send mouse clicks to the desktop?

Any help is really apreciated.
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Check the windows desktop to send mouse clicks
« Reply #1 on: August 29, 2014, 12:02:33 am »
2- How i could send mouse clicks to the desktop?

See the Simulating Mouse section of http://wiki.lazarus.freepascal.org/LCL_Tips

Alguien

  • New Member
  • *
  • Posts: 14
Re: Check the windows desktop to send mouse clicks
« Reply #2 on: August 29, 2014, 01:56:15 am »
1- How i get screenshots from my whole desktop?
Have a look here http://wiki.lazarus.freepascal.org/LCL_Internals#TBitmap.LoadFromDevice_for_screenshot_taking
If you prefer to get a capture of a window, pass the window handle as a parameter to GetDC()

You can play with TBitmap class in order to detect pixel changes.

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Check the windows desktop to send mouse clicks
« Reply #3 on: August 30, 2014, 08:09:16 am »
thanks a lot

im able to read the pos of the cursor without problem (nice)

I added one timer

Code: [Select]
procedure TForm1.Timer2Timer(Sender: TObject);
var control: tcontrol;
begin
  control  := tcontrol.create(nil);
  LCLSendMouseDownMsg(control,692,534,mbLeft);
  LCLSendMouseupMsg(control,692,534,mbLeft);
  control.free;
end;

It is supossed to send a left click to the screen pos 692,534 everytime the timer elapsed (5 seconds) but is not working. It compiles fine but nothings happens. What im missing?
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

Alguien

  • New Member
  • *
  • Posts: 14
Re: Check the windows desktop to send mouse clicks
« Reply #4 on: August 30, 2014, 05:43:06 pm »
It seems you are trying to send clicks outside your application.

If that is the case, those LCL functions won't help at all since they expect a TControl as a parameter and will only work (I assume, I haven't tested them) inside your application scope.
Another thing to note in your code is that if it works you wouldn't see a result since you are creating a non-visual control, which is not attached to the UI, then you send the mouse messages to it, the control ignores them because it has nothing to do with them, and then you free the control, all this in a fraction of time.

Probably that code could work if you use a TButton for instance, which was previously created and assigned to the UI (created from the form editor) and then send the messages to the button coordinates to see what happens.


Now going back with your goal, sending a global mouse click in Windows is easy, but keep in mind that this solution is not multiplatform.
To "simulate" user's mouse input in windows you should:
  • Use the Windows unit
  • Move the mouse to the screen position where you want the click to be sent (see SetCursorPos() function at msdn.microsoft.com)
  • Send a mouse click event (see mouse_event() function and its parameters at msdn.microsoft.com)

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Check the windows desktop to send mouse clicks
« Reply #5 on: August 30, 2014, 06:05:31 pm »
use http://wiki.lazarus.freepascal.org/MouseAndKeyInput which is multi platform (linux & windows at least).
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Check the windows desktop to send mouse clicks
« Reply #6 on: August 31, 2014, 05:21:35 am »
ok, i have a good improvement now

Code: [Select]
procedure TForm1.Timer2Timer(Sender: TObject);
var punto : TPoint;
begin
  getcursorpos(punto);
  setcursorpos(692,534);
  Mouse_Event(MOUSEEVENTF_ABSOLUTE or MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0) ;;
  Mouse_Event(MOUSEEVENTF_ABSOLUTE or MOUSEEVENTF_LEFTUP, 0, 0, 0, 0) ;;
  setcursorpos(punto.x,punto.y);
end;

This is working, making clicks outside my lazarus app. Until now i cant figure if it is possible to make those click without the need to move the cursor to the desired position.

I will now work with pixel detection. thanks for help until now :)
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

Never

  • Sr. Member
  • ****
  • Posts: 409
  • OS:Win7 64bit / Lazarus 1.4
Re: Check the windows desktop to send mouse clicks
« Reply #7 on: August 31, 2014, 11:35:45 am »
hold the original mouse place
Move the mouse to the desired place
exec your event
return the mouse back to the original place
================================
alternative change the mouse icon and reset it when you v done
Νέπε Λάζαρε λάγγεψων οξωκά ο φίλοσ'ς αραεύσε

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Check the windows desktop to send mouse clicks
« Reply #8 on: August 31, 2014, 04:35:20 pm »
Windows version!

1- ... Even better if i could get the pixel color in a coordinate x,y directly
Code: [Select]
uses
..Windows;
..
var
..
  dc: HDC;
  clr: COLORREF;
..
  dc := GetWindowDC(0);
  clr := GetPixel(dc, x, y);  //<---- You need to provide the correct x and y coordinates here
  ReleaseDC(0, dc);

//Do what you want with the color: clr
  if clr = target_clr then ...

Until now i cant figure if it is possible to make those click without the need to move the cursor to the desired position.
I believe it is possible. Use SendMessage with WM_LBUTTONDOWN and WM_LBUTTONUP. The coordinates pass through LParam.
« Last Edit: August 31, 2014, 04:58:01 pm by engkin »

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Check the windows desktop to send mouse clicks
« Reply #9 on: August 31, 2014, 09:46:09 pm »
thanks a lot, pixel detection works perfect.

Please, could you be a little more specific about this:

Quote
I believe it is possible. Use SendMessage with WM_LBUTTONDOWN and WM_LBUTTONUP. The coordinates pass through LParam.
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Check the windows desktop to send mouse clicks
« Reply #10 on: August 31, 2014, 10:08:17 pm »
Drop a button and a timer on a form. Change the timer period to 5000 and use the following code:
Code: [Select]
uses
  Windows;

var
  counter:integer = 0;

procedure TForm1.Button1Click(Sender: TObject);
begin
  inc(counter);
  ShowMessage(Format('Button got clicked! (%d times)',[counter]));
end;

procedure TForm1.Timer1Timer(Sender: TObject);
var
  Coordinates: TSmallPoint;
begin
  Coordinates.X := Button1.ClientWidth div 2;
  Coordinates.Y := Button1.ClientHeight div 2;
  SendMessage(Button1.Handle, WM_LBUTTONDOWN, 0, LParam(Coordinates));
  SendMessage(Button1.Handle, WM_LBUTTONUP, 0, LParam(Coordinates));
end;


engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Check the windows desktop to send mouse clicks
« Reply #11 on: September 01, 2014, 08:17:03 am »
Replacing the previous code with this one, would produce a "click" at the coordinates you provide:
Code: [Select]
procedure TForm1.Timer1Timer(Sender: TObject);
var
  Coordinates: TSmallPoint;
  aPt: POINT;
  aWnd: HWND;
  aRect: TRect;
begin
  aPt.x := ??;  //<--- provide the correct values
  aPt.y := ??;

  aWnd := WindowFromPoint(aPt);
  GetWindowRect(aWnd, aRect);

  Coordinates.X := (aRect.Right-aRect.Left) div 2;
  Coordinates.Y := (aRect.Bottom-aRect.Top) div 2;

  SendMessage(aWnd, WM_LBUTTONDOWN, 0, LParam(Coordinates));
  SendMessage(aWnd, WM_LBUTTONUP, 0, LParam(Coordinates));
end;

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Check the windows desktop to send mouse clicks
« Reply #12 on: September 02, 2014, 08:23:05 am »
Im working with your second suggestion but im still unable to make it work.

Will post details tomorrow.
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Check the windows desktop to send mouse clicks
« Reply #13 on: September 02, 2014, 08:26:09 am »
change the coordinates calculation to
Code: [Select]
  Coordinates.X := (aRect.Left+aRect.Right) div 2;
  Coordinates.Y := (aRect.Top+aRect.Bottom) div 2;
or add the left and top values to the existing calculations.

Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Check the windows desktop to send mouse clicks
« Reply #14 on: September 02, 2014, 12:11:11 pm »
Neither works, not at least in the desired way

Code: [Select]
Procedure tform1.hacerclick(xpunto,ypunto: integer);
var
  Coordinates: TSmallPoint;
  aPt: POINT;
  aWnd: HWND;
  aRect: TRect;
begin
  aPt.x := 124; //<--- provide the correct values
  aPt.y := 214;

  aWnd := WindowFromPoint(aPt);
  GetWindowRect(aWnd, aRect);

  Coordinates.X := (aRect.Left+aRect.Right) div 2;
  Coordinates.Y := (aRect.Top+aRect.Bottom) div 2;

  SendMessage(aWnd, WM_LBUTTONDOWN, 0, LParam(Coordinates));
  SendMessage(aWnd, WM_LBUTTONUP, 0, LParam(Coordinates));

end;   

if at selected point there is a banner in a web page, the navigator window brings to font, but nothing else (the link do not open at all) and even sometimes it have an unpredictable result (produces the same effect that a click over the windows desktop, even if there is a window there)

Looks like moving the cursor there is the only way.
« Last Edit: September 02, 2014, 12:25:28 pm by torbente »
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

 

TinyPortal © 2005-2018