Read OpenGLContent mouse position and convert back to coords.

[ranny]

Hello,

I have a couple of programs I use for graphing 3D polynomials where the data and curve shape is displayed in a OpenGLContent object, I got a lot of help with this from the forum in the past.  I am now working on a 3D beam analysis program for which I have all the maths and can draw the model in 3D in my OpenGLContent and it is beginning to look good.  I am having some scaling problems but I think I will get there.

My difficulty now is finding a way of clicking the mouse near a drawn node point on the OpenGLContent object and getting back world coordinates I can then use in other calculations, for example, being able to add another node offset to the one I have selected.  I am having no luck with this at all yet I am sure there is a neat method out there.  I am struggling to understand how, I think, that anything I draw on the OpenGLContent object always has it's centre 0,0,0 in the middle of the object but the mouse position is relative to the top and left of the object.  That's my first problem!

Does anybody have an idea where I can look for a solution?

Thanks in advance.

[Dzandaa]

Hi,

I have the same problem

I didn't find any clue in the OpenGl manual.

Most of the solutions use ray casting:

https://antongerdelan.net/opengl/raycasting.html

Also look a the feedback mode glRenderMode(GL_FEEDBACK), not sure it is related.

B->

[ranny]

Hi,

I have the same problem

I didn't find any clue in the OpenGl manual.

Most of the solutions use ray casting:

https://antongerdelan.net/opengl/raycasting.html

Also look a the feedback mode glRenderMode(GL_FEEDBACK), not sure it is related.

B->

Hi Dzandaa,

So I am plotting lines to represent a beam structure which displays the beams and the numbers at the beam ends and middle of each beam.  The 3D coords from the file of data get scaled and drawn in the OpenGLContent object.  I was hoping I could "reverse" the scaling routine from a cursor position and get at least an X Y pair of coords in world terms that I can check against all of the nodes and find the nearest node to the cursor.  However, as it appears to me, how OpenGL plots lines and polygons etc. is a bit hard to work backwards from.  I also thought that I could store the OpenGLContent position in an array for interrogating later but not sure about how I will get it to work.

I would have thought there was some sort of routine that can do this, there must be OpenGL gaming situations where you place the cursor at a location and have code that responds to it.....



 

[Khrys]

Are you using "modern" OpenGL (shaders, vertex buffer objects/arrays, framebuffers,  glDrawElements)? Or are you using the old fixed-function pipeline (glBegin, glEnd, glVertex3f, glCallList  etc.)? I'm asking because these are very different approaches, with the former being insanely more flexible compared to the latter.

In my (modern) OpenGL project for example I'm rendering the scene again with a shader that outputs (among other things) the position of each fragment (interpreted as RGB it may look something like the attached image). Then I do  glReadPixels at the mouse position to get the position.

The same thing should be possible by sampling the Z-buffer instead (which is pretty much always present), although that isn't ideal (nonlinear, loss of precision for objects far away from the camera, needs more camera information).

Unfortunately this technique isn't implementable in the fixed-function pipeline (glReadPixels  may still be useful however, for example by enumerating objects with specific color values to at least check what is under the mouse pointer). In that case I'm afraid you'll have to use CPU-based ray casting. In any case, the fixed-function pipeline makes anything slightly advanced immediately much, much harder if not outright impossible to accomplish (while also being deprecated since 2008 + shaders have been available for 20 years).

[KodeZwerg]

Isnt that task not just a normalization? Like ClientToScreen/ScreenToClient methods.

[Dzandaa]

Hi,

This could be interesting:
https://stackoverflow.com/questions/29912559/opengl-get-3d-coordinates-of-nearest-world-3d-point-to-the-current-mouse-locatio

B->

[Seenkao]

Привет!
Решения были давно найдены, это:
- glReadPixels
- glRenderMode
- прямые вычисления (не используя OpenGL, для 3D мало подходит, но при желании можно использовать).

Все три способа подойдут для любого созданного контекста.
Видео с примерами. Извиняюсь, но на русском, думаю если это проблема, то нужную информацию вы сможете найти на других языках программирования.

----------------
Google translate:
Hello!
Solutions were found a long time ago, these are:
- glReadPixels
- glRenderMode
- direct calculations (not using OpenGL, not very suitable for 3D, but can be used if desired).

All three methods are suitable for any created context.
Video with examples. I apologize, but the video is in Russian, I think if this is a problem, then you can find the necessary information in other programming languages.

[ranny]

Hello again,

Thanks everybody for the comments, sorry for the late reply - it was a nice sunny weekend and the garden needed attention!

I am surprised that there appears to be little about this issue of mine that has an obvious solution.  I am a novice coder and not very good at graphics (I prefer the maths part) but I would have thought this was a common requirement.  Apologies if that seems rude but this is a make or break for my program.  This from Dzandaa seems interesting right enough, will explore (thanks Dzandaa):-

Hi,

This could be interesting:
https://stackoverflow.com/questions/29912559/opengl-get-3d-coordinates-of-nearest-world-3d-point-to-the-current-mouse-locatio

B->

A little bit more about the program I am working on.  Many years ago (more than 20 I am horrified to report) I got some Fortran code for a 2D beam analysis program and converted it to 3D in Delphi.  I wrote my own "3D graphics" routines that showed the image (which is just lines and text mostly) which could be rotated and scaled to some degree but it was very crude.

To build a 3D model for analysis one specifies nodes and the beam elements join two nodes.  The beam elements have mechanical properties.  Loads can be applied to nodes and when the analysis is run there are a load of matrix multiplications and solving which then provides detailed mechanical info for each node and element.

To build a model it is much easier to start with some nodes placed at their coordinates then one can select a node by using the mouse and then add new nodes by an offset dimension.  Beam elements can be added by selecting two nodes and entering (or selecting already established) properties.

So it is only a worthwhile coding exercise if I can get to read capturing a screen position, getting it back to world coordinates to find the nearest node or beam, then carrying on from there.

Here is a pic of my old program with my crude graphics.

Thanks all.

[Khrys]

If you're only drawing thin lines between a few points, then OpenGL is definitely not the right tool for calculating the closest point to some reference. OpenGL isn't some all-encompassing fully-fledged 3D library; it's for coloring pixels using the GPU - massively parallel tasks involving lots of data, we're talking hundreds of thousands to millions of pixels. In this context, a few hundred nodes is nothing in comparison.

[...] I am a novice coder and not very good at graphics (I prefer the maths part) [...]

Since you're not afraid of maths and matrices and need to locate only a few points, I suggest doing the calculations on the CPU like this:

• Transform the clicked point into normalized device coordinates (NDC; axis range is  [-1, 1], i.e. in NDC the viewport's left edge has  x = -1  and top edge has  y = 1)
• Find the camera's combined projection matrix; OpenGL uses it (*) to transform the points you feed it into NDC space.
• Find each node's NDC location by multiplying its position (with W = 1) by the combined projection matrix and then dividing the result's components by its  W  component).
• Find the closest node by comparing X-Y plane distances in NDC space.

* Typically 3 matrices lie between a source and a destination point in OpenGL:

• The model matrix (per-object offset, scale, rotation; transforms from model space to world space)
• The view matrix (camera position and orientation; transforms from world space to camera space)
• The projection matrix (transforms from world space to clip space)

Going from clip space to NDC space involves doing a perspective divide, which is done by dividing all components by the  W  component (OpenGL uses homogenous coordinates, so for input points it is assumed that  W = 1).
To find the combined matrix you'd need to multiply these matrices together (in the right order of course). I'm not familiar with fixed-function OpenGL, but I think  glGet with  GL_PROJECTION_MATRIX and  GL_MODELVIEW_MATRIX  should give you the matrices you need.

[ranny]

Thanks Khrys.

I found this:-

Code: Pascal  [Select][+][-]

1.  
2. type T3D_Point = array [1..3] of Double;
3.  
4. function GetOGLPos(X, Y: Integer): T3D_Point;
5. var
6.     viewport:   array [1..4]  of Integer;
7.     modelview:  array [1..16] of Double;
8.     projection: array [1..16] of Double;
9.     winZ: Single;
10. begin
11.     glGetDoublev( GL_MODELVIEW_MATRIX, @modelview );
12.     glGetDoublev( GL_PROJECTION_MATRIX, @projection );
13.     glGetIntegerv( GL_VIEWPORT, @viewport );
14.  
15.     // In Delphi A Y Value Of 0 Returns An Unknown Value
16.     // I Discovered This While I Was Testing A Crosshair
17.     if( Y = 0 )then Y := 1;
18.  
19.     glReadPixels(   X, -Y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, @winZ );
20.     gluUnProject(   X, viewport[4]-Y, winZ,
21.             @modelview, @projection, @viewport,
22.             Result[1], Result[2], Result[3]);
23. end;

However, I cannot get it to work with the following errors:-

Incompatible types: got "Double" expected "PGLdouble" which I cannot resolve.

And sometimes
Foward declaration not solved "GetOGLPos(Longint,Longint):Array[1..3]of PGLDouble

I feel this is what I am looking for, but cannot get it to work, don't know what PGLDouble is either.

Actually at one point I did get it to work but it just stated Access violation and stopped.

I understand I may be using OpenGL for something so small but I have the rest of the code working well to display what I want with the ability to rotate the model and zoom in etc. if I can fix this I can go forward...

[wp]

Why is "viewport" an array of Integer when you put it in a call to glGetDoublev? I'd rather declare it as an array of double or GLDouble.

PGLDouble is a pointer to GLDouble: CTRL-click on it, and the IDE opens the unit in which it is declared:

Code: Pascal  [Select][+][-]

1. type
2.   PGLdouble  = ^GLdouble;

At which line is that error reported? I assume in the glUnproject line because the result is returned as a PGLdouble and you assign it to the fields of a T3D_Point which probably is a record of three doubles (hopefully, rather than single!)

Code: Pascal  [Select][+][-]

1. // unit glu;
2. ...
3.   gluUnProject : function(winX:GLdouble; winY:GLdouble; winZ:GLdouble; model:PGLdouble; proj:PGLdouble;
4.     view:PGLint; objX:PGLdouble; objY:PGLdouble; objZ:PGLdouble):GLint;extdecl;

This might work (but not tested):

Code: Pascal  [Select][+][-]

1. var
2.   objx, objY, objZ: GLDouble;
3. ...
4.     gluUnProject(   X, viewport[4]-Y, winZ,
5.             @modelview, @projection, @viewport,
6.             @objx, @objY, @objZ);
7.     Result[1] := objX;
8.     Result[2] := objY;
9.     Result[3] := objZ;  
10.  
11. // or maybe simpler, if the elements of Result are doubles
12.     gluUnProject(   X, viewport[4]-Y, winZ,
13.             @modelview, @projection, @viewport,
14.             @Result[1], @Result[2], @Result[3]);

[ranny]

Thanks WP, always useful, but alas I still get this when compiling...

Error:Forward declaration not solved "GetOGLPos(Longint;Longint):Array[1..3]Of Double

I have declared T3D_Point after uses in the main form, don't know if that makes a difference.

Code: Pascal  [Select][+][-]

1. type
2.   TPaletteItem = record
3.     Value: GLFloat;
4.     Color: TColor;
5.   end;
6.  
7.   TPalette = array of TPaletteItem;
8.  
9.   T3D_Point = array[1..3]of Double;
10.  
11.  
12.   { TForm1 }
13.  
14.   TForm1 = class(TForm)
15.     btnReset: TButton;      
16.  

[Khrys]

I think I forgot to clarify why only drawing thin lines makes OpenGL unsuitable for the task of finding the closest point...
It's perfectly fine and reasonable to use OpenGL for drawing lines, that's (part of) what it was designed for. What doesn't really work in that case is relying on sampled buffer values (glReadPixels); that's because the buffers are only updated where OpenGL actually draws something. Lines are thin and only span a few pixels, and so trying to hit one with the mouse pointer is difficult and frustrating. Look at the attached example of a tilted triangle: on the left it's drawn as a solid face and has a well-defined depth value at each pixel inside it, while on the right it's drawn in wireframe mode and is mostly undefined (technically  glClear's argument).


So to reiterate, I'm afraid you'll have to do the calculation yourself, without using  glReadPixels.

[ranny]

Thanks Khrys  for your comments.

If you look further back in this thread I provided a screenshot from my old program of what I am working with.  What I am trying to read from the screen is nearby nodes and elements, not really in any great detail, just approximately where they are.  So if I can get a screen position that I can convert back to the 3D world of the model, all I have to do is step through the node (or element) coordinates and find the closest one to that 3D world position.  Once selected I can carry out the rest of my code to add new nodes, change position etc.  So I don't need to try and hit any item too accurately.  This is what I did in my early program.

I enjoy doing this sort of stuff, not very good at it but it is rewarding when it works out.....

Cheers

[Khrys]

OpenGL won't be of any help here unless you click directly on the drawn geometry (which, as I said, isn't something to rely on when doing wireframe). I'd still recommend manually projecting everything to 2D and only then looking for the closest element (in 2D). The user only sees the 2D projection, the depth doesn't matter and in is ill-defined in many cases anyway. What if multiple nodes are collinear from your POV, so that they are exactly on top of each other in the 2D projection? You're only trying to find an approximation of the nearest relevant thing's position, you don't need a rigorous 3D inverse for every pixel on the screen.