Compiled the demo projects at 96ppi and ran them at 144ppi - the inner dimensions scale correctly, fine!. With one exception: the images do not scale or are scaled by the system to become very pixelated - see screenshot.
Looking at the code of the demo_simple I see that you do not scale the user-provided imagelist. I set ImageList.Scaled to true, added two image resolutions for 24x24 and 32x32 images (in the ImageList editor) and replaced the icons by similar ones from the Lazarus icon collection (folder images/general_purpose). This changed the two user-icons on two tabs but left the internal icons for left/right arrows, close-crosses and add-tab unchanged. I guess these are the ButtonImages properties - but assigning some indices of my imagelist did not change anything. So, I am afraid that something's wrong here in image handling.
The other issue is that the user-provided images do not change their size when the projects runs under 144 ppi. You need some changes to take care of the "real" image size under High-DPI:
* In function TExtTabCtrl.GetTabImageWidth use FImages.WidthForPPI[0, Font.PixelsPerInch] to measure the image width; the "0" indicates that the default width of the images (i.a. at 96ppi) should be assumed to be the Width of the ImageList, but the images can be scaled to any other base width (normally there is an extra property "ImagesWidth" next to the "Images" property in controls using an ImageList). You may also have to apply the result of CanvasScaleFactor, but I don't have experience with it since Windows does not use it (macOS does...)
* Likewise, in GetTabImageHeight use FImages.HeightForPPI[...].
* I did not dive into your drawing code which is more complicated than usual because you are caching rotated images. But basically you should always call the XXXXForPPI methods of the imagelist to get the image at the correct size for the current resolution. Example: Draw the image of index AIndex at position AX, AY - AImageWidthAt96PPI is 0 (or FImagesWidth if you'd provide it), ATargetPPI is usually taken from the font settings (Font.PixelsPerInch), ACanvasFactor is the result of GetCanvasScaleFactor:
procedure DrawForPPI(ACanvas: TCanvas; AX, AY, AIndex: Integer;
AImageWidthAt96PPI, ATargetPPI: Integer; ACanvasFactor: Double; AEnabled: Boolean = True);
* In order to extract an image for caching you should call ImageList.ResolutionForPPI[AImageWidth, APPI: Integer; const ACanvasScaleFactor: Double].GetBitmap(Index: Integer; Image: TCustomBitmap)
In the attachment there's a version of demo_simple where everything should be set up for correct scaling.
Two more remarks:
* I am missing an imageindex entry for the close button in the ButtonImages property.
* You should declare all image index properties as TImageIndex rather than integer. Then the LCL can select the corresponding property editor which displays preview images in a combobox.
I have tried to implement all of your suggestions. I hope to have done it right (I have zero experience with high-DPI and don't have any setup to test it). I had a lot of troubles with GetCanvasScaleFactor (on my macOS it was returning 2, and this resulted in the image stretched to twice its size, extending outside the tab and pixelated), hence, I commented this out. Maybe somebody can tell me what was wrong.
Please, wp, can you give it a try?