I originally wanted to post this as a question but while writing it I came up with the solution already which I wanted to share with you. So this is more like a tutorial. I did not go into detail very much with the code, as this post would get too long, but if you have any questions regarding the implementation feel free to ask.

For my OpenGL-based UI I which is inspired by the LCL I had to make heavy use of the glViewport function. For each control I had a canvas. The canvas had to be locked which would set the viewport to the calculated screen rect to prevent overdraw, which is especially useful for text which might be bigger than the control. If you imagine an edit for example where this is often the case. If you have nested controls with parenting and implement scrolling you might also have overdraw. The glViewport or glScissor functions can be used to do the clipping here. This has however one drawback. You can only render one control at a time resulting in a lot of gl function calls and making the drawing comparably slow, even if you only render a few quads. Wanting to draw all shapes in one batch I felt I needed something different (and shader based).

Let's start a basic vertex shader. We use a vec4 uniform for the clipping for now for keeping this simple (So only one viewport).

`#version 120 // choose whichever version you like`

attribute vec4 Position;

attribute vec4 Color;

varying vec4 vPosition;

varying vec4 vColor;

uniform mat4 view; // for scrolling and zooming or rotating

uniform mat4 proj;

uniform vec4 ClipRect; // xy = upper left, zw = lower right

void main(){

vColor = Color;

vPosition = view * Position;

gl_Position = proj * vPosition; //This could be done with one matrix in one operation

//but we are going to need the value of vPosition later on

}

At first I thought this line in the vertex shader would do the trick:

`vPosition.xy = max( min( vPosition.xy, ClipRect.zw ), ClipRect.xy );`

This kind of works for filled rectangles but If we have a line rectangle or any other shape it will still render the clipped lines at the edges instead of removing them. If the lines are angled it gets even worse. Because the endpoints of the lines are clipped the line direction might change (Unfortunately I did not make a screen shot to illustrate that).

Then I came up with the idea of doing the clipping in the fragment shader.

`#version 120`

uniform mat4 proj;

varying vec4 vPosition;

varying vec4 vColor;

uniform vec4 ClipRect; // xy = upper left, zw = lower right

void main(){

// actually the clip variable does the opposite as expected

// if it is 1 no clipping occurs and if it is zero the pixel is clipped, therefore the "1-..." part

float clip = 1 - clamp( 0, 1, dot( vec2( 1 ), step( ClipRect.zw, vPosition.xy ))); //check if clipping of the lower right occurs

clip *= 1 - clamp( 0, 1, dot( vec2( 1 ), 1-step( ClipRect.xy, vPosition.xy )));

gl_FragColor = clip * vColor; // do the final clipping by changing the alpha (and also the color, which is optional)

}

We use a float clip variable which we later multiply by the color value, so it's alpha will be zero if clipping should be done.

We could have used an if, when setting the clip variable as well but branching can slow down the rendering. The driver might do some optimizing here but we cannot rely on that. The dot part is also an optimization for adding the clipping of x and y axis (See here:

https://www.khronos.org/opengl/wiki/GLSL_Optimizations#Dot_products).

The step part works component-wise and will return 0 if a component is lower than the threshhold and otherwise 1 ). We do this both for the upper left corner and for the lower right corner, where we have to "inverse" the result.

If we store all our draw calls and make the ClipRect an attribute we can use different "view ports" for each primitive (Actually each vertex, but that's still better than drawing each primitive separately).

That's not all yet. With this method we can also use rotation. If we rotate the view matrix the rectangles are clipped at the viewport like before.

If we instead rotate the projection matrix the clipping rectangle is rotated (This is not possible with glViewport). With some effort we could even do a perspective transform here. This all depends on which space we do the clipping (view or projection space) or if we rotate prior the clipping or afterwards because the vPosition attribute is assigned before applying the projection matrix. Because of that we can define our clipping rect in pixel space given the right projection matrix.

So our view and projection matrices are calculated like this in our program (not in the shader

) code:

mProj:= mat4orthoRH( TopLeft, BottomRight );// * mat4rotate( vec3_Axis_PZ, sin( SDL_GetTicks / 1000 ) * 0.5 );

//uncomment for rotating the projection matrix

mWorld:= mat4translate( vec3( -TopLeft, 0 ));// * mat4( Zoom ) * mat4rotate( vec3_Axis_PZ, 0.2 );

//uncomment to rotate the view matrix

//we translate to start in the corner of the clipping rect like glViewport.

//Unlike glViewport we start in the _upper_ left corner for drawing our ui.

//Without the translation part this works like glScissor.

Z-Buffer: If we stack our primitives on top of each other like normal draw calls, we don't need a zbuffer. If we have a different order and use a Z-buffer we might have to introduce branching to discard the clipped pixels instead of setting it's alpha to zero.

What do you think of this method? Do you know any optimizations?