[SOLVED] - "Tween" two images (Onionskin) & Chroma Key

[TheBlackSheep]

ok fixed up the source to improve the efficiency (it does make it faster on Linux but this is still considerably slower than the Win32 version for some reason even in Linux64)

the function is now;

Code: [Select]

...
  r := trkbrRedChromakey.Position;
  g := trkbrGreenChromakey.Position;
  b := trkbrBlueChromakey.Position;
  t := trkbrTolerance.Position;

  for n := FG.NbPixels-1 downto 0 do begin
    diff := BGRADiff(p2^, BGRA(r,g,b));
    if diff <= t
      then p3^ := p1^
      else p3^ := p2^;
    inc(p1);
    inc(p2);
    inc(p3);
  end;

  Composite.InvalidateBitmap;
  Bitmap := Composite.MakeBitmapCopy(clBlack);
  imgComposite.Picture.Bitmap := Bitmap;
  Bitmap.Free;
  Composite.Free;
...

I've attached the two images showing it working ok in Win32 but not in Linux - if I change the threshold value then the foreground disappears (and just displays the background) or takes over the whole image but at a point when the green value and the threshold are as shown on the screenshots - the effect is displayed and it works well, just not in Linux  .

Note, I'm looking at a the "diff" being less than or equal to the threshold - it seems to work in win32 but I don't know if this is how it should be or if it actually should be within a range - I've tried it with a few greenscreen images and it appears to work ok although the tolerance and green values need adjusting to cope with the varying light/shadow levels for each particular image as would be expected (however it doesn't work anywhere near as well with blue-screen images).

I tried BGRAWordDiff and that doesn't produce any "merged" image at all but that could be because I'm using the BGRADiff and BGRAWordDiff incorrectly.

TheBlackSheep

[circular]

That's right, you just need to compare the diff with some number. With WordDiff, the number is just bigger, so if you use numbers that are too small, you never get the color match with the chroma key.

I suppose the problem in Linux comes when you load the images :

Code: [Select]

  BG := TBGRABitmap.Create(imgBackground.Picture.Bitmap);//background source
  FG := TBGRABitmap.Create(imgForeground.Picture.Bitmap);//foreground source with greenscreen
This should work, but maybe there is a problem of encoding. Can you try to load directly from the files instead ?

Anyway congratulations ! I am very excited by that software. That's so cool. 

[TheBlackSheep]

yep that works ... so there's a difference between the two platforms when copying the bitmap from a TImage to a BGRABitmap.

I'd need to speed up the operation too as there's a significant drop in performance between Win32 and Linux - I was hoping to do this on a video image coming from a webcam but I just don't think it'll be fast enough.

TheBlackSheep

[circular]

What about preloading images ? It seems that you load the file, or import it from TImage each time you draw it, right ?

[TheBlackSheep]

hi circular

switched to using scanline - that appears to be faster in Linux and tidied up the source which I've attached together with the sample foreground and background image that are preloaded in the demo's timage's. 

If you press the shift key and click on the foreground image it will set the trackbar to the color under the mouse cursor.  Similarly on the composite image you can zoom into the image by pressing the shift key there too (again the zoom image shows the area under the mouse) - I just used these bits to try and work out the color differences to see if it was working.   clicking the zoom image shows the color under the mouse there too.  You can load in other images to see if it works on other images (the green chroma key - or other chroma color might need adjusting to suit other images) - a blue chroma key image I got from Google images works well after setting the key via the shift key setting rather than trying to do it manually - it looks like if the background chroma color is consistent then setting the color via the mouse "dropper" works fine if it's varied because of shadow/lighting differences then you need to play around with the threshold and chroma color.       

Pre-loading the images must be the right way to do this (you wouldn't want to do it on each trackbar change but in my defense it was only a test prog to prove the theory) - I'll try and integrate it into the 5dpo webcam code and see if there are problems with performance. 

Thanks for your continued support 

TheBlackSheep

[circular]

You're welcome. I was very interesting and we made BGRABitmap better. 

[circular]

I've tested your program. In fact I realised that maybe BGRADiff is not the best way to do this.

I suggest that you use BGRAToHSLA or BGRAToGSBA to get the a THSLAPixel, and then add the difference of both Hue and Saturation to get a color difference. BGRADiff also returns a difference between dark and light pixels, which is not what you want here I guess. Or maybe you can include the Lightness in the difference but divided by some value. See what I mean ?

[Shebuka]

Hi, i'v tested your app and i have problems with mouse click... in intfgraphics.pas the TLazIntfImage.Create(ARawImage: TRawImage; ADataOwner: Boolean); receive the raw image, but when Create(Desc.Width, Desc.Height, []); is called the coordinates are negatives, instead of 389, 114 they are -388, -113 and then raise FPImageException.Create('Invalid Size'); is called inside TLazIntfImage.InternalSetSize

[TheBlackSheep]

hi circular

I've tried the following based on your suggestion (note the trackbars have max values of 65535 now);

Code: [Select]

...
  h := trkbrHueValue.Position;
  s := trkbrSaturationValue.Position;

  ht := trkbrHueTolerance.Position;


  for y := 0 to FG.Height - 1 do begin

    p1 := BG.ScanLine[y];
    p2 := FG.ScanLine[y];
    p3 := Composite.ScanLine[y];

    for x := 0 to FG.Width-1 do begin

      diff := (BGRAToHSLA(P2^).hue - h) + (BGRAToHSLA(P2^).saturation - s);
      if diff >= ht
        then p3^ := p1^
        else p3^ := p2^;

      inc(p1);
      inc(p2);
      inc(p3);
    end;
  end; 
...

but it doesn't seem to be as effective as the BGRADiff  method previously - unless I've misunderstood what you meant.

I've also tried comparing the HSL on an individual basis, i.e between ranges like this (ht,st being the tolerance +ve & -ve either side of the comparison values (h & s) on the basis that the test pixel from the fg image is between the relevant range of hue and saturation.

Code: [Select]



  h := trkbrHueValue.Position;
  s := trkbrSaturationValue.Position;
 
  ht := trkbrHueTolerance.Position;
  st := trkbrSatTolerance.Position;

  for y := 0 to FG.Height - 1 do begin

    p1 := BG.ScanLine[y];
    p2 := FG.ScanLine[y];
    p3 := Composite.ScanLine[y];

    for x := 0 to FG.Width-1 do begin

    h2 := BGRAToHSLA(P2^);
 
    if  (h2.hue > (h - ht)) and (h2.hue < (h + ht))
      and (h2.saturation > (s - st)) and (h2.saturation < (s + st))
        then p3^ := p1^
        else p3^ := p2^;   

      inc(p1);
      inc(p2);
      inc(p3);
    end;
  end; 
...
 

however it doesn't work particularly well either so I don't think this is correct either.

Shebuka - I haven't seen any issues with the mouse click so not sure how that's happening - I'm using CodeTyphon 2.60 as my platform but with the latest BGRABitmap code from Sourceforge although your issue sounds like it's a problem with the standard TImage and the mouseover - which image is it happening on? the foreground, the composite or the zoom?  Wherever I click on these images it works fine.

TheBlackSheep

[Shebuka]

There is some sort of bug in CarbonCanvas GetPixel, and probably also in CarbonObject RawImage_FromCarbonBitmap

bug reported http://bugs.freepascal.org/view.php?id=22111

[TheBlackSheep]

hi Shebuka,

ah you're on a Mac - that would explain why I'm not seeing that particular issue.

Have you checked the co-ordinates are positive in the example and are just negative when they're called internally by CarbonCanvas - i.e. it's not something to do with the fact that it's expected a word and getting an integer and deciding it's the wrong side of zero?  That would explain a 1 difference in the positive and negative values you're seeing - the Round function returns an int64 so it could be that in which case you might just be able to wrap a typecase to word around the call;

Pixel := imgForeground.Canvas.Pixels[word(Round(DropperX)),word(Round(DropperY))]; 
 
Or you could try kidding the result i.e. deliberately set it to a negative value by multiplying the GetPixel call by -1 as in;

imgZoom.Canvas.Pixels[X * -1, Y * -1];

depending on which mouseclick function is causing the issue (presume it happens on all of the images with this event)

Alternatively find the call where it's breaking in the CarbonCanvas and use abs() as in;

..GetPixel(abs(X), abs(Y));

it looks as if it's just negating the co-ordinates incorrectly (albeit off by one) and this should put them positive again although they may not be the actual pixel you're clicking (it's a bit of an estimate anyway on a stretched image - I couldn't readily find a function to work out the actual underlying pixel coordinates of a stretched image from the XY coords of a mouseclick event).

Iit might be worth checking to see if there are any updates in svn for the Carbon Canvas - I'd have thought GetPixel is a fairly common function to call and it should be causing issue elsewhere if it's not working correctly.

TheBlackSheep
 

[circular]

hi circular

I've tried the following based on your suggestion (note the trackbars have max values of 65535 now);

Code: [Select]

...
  h := trkbrHueValue.Position;
  s := trkbrSaturationValue.Position;

  ht := trkbrHueTolerance.Position;


  for y := 0 to FG.Height - 1 do begin

    p1 := BG.ScanLine[y];
    p2 := FG.ScanLine[y];
    p3 := Composite.ScanLine[y];

    for x := 0 to FG.Width-1 do begin

      diff := (BGRAToHSLA(P2^).hue - h) + (BGRAToHSLA(P2^).saturation - s);
      if diff >= ht
        then p3^ := p1^
        else p3^ := p2^;

      inc(p1);
      inc(p2);
      inc(p3);
    end;
  end; 
...

but it doesn't seem to be as effective as the BGRADiff  method previously - unless I've misunderstood what you meant.

You forgot to put an "abs". Try with :

Code: [Select]

diff := abs(BGRAToHSLA(P2^).hue - h) + abs(BGRAToHSLA(P2^).saturation - s);

[Shebuka]

hi Shebuka,

ah you're on a Mac - that would explain why I'm not seeing that particular issue.

Have you checked the co-ordinates are positive in the example and are just negative when they're called internally by CarbonCanvas - i.e. it's not something to do with the fact that it's expected a word and getting an integer and deciding it's the wrong side of zero?  That would explain a 1 difference in the positive and negative values you're seeing - the Round function returns an int64 so it could be that in which case you might just be able to wrap a typecase to word around the call;

Pixel := imgForeground.Canvas.Pixels[word(Round(DropperX)),word(Round(DropperY))]; 
 
Or you could try kidding the result i.e. deliberately set it to a negative value by multiplying the GetPixel call by -1 as in;

imgZoom.Canvas.Pixels[X * -1, Y * -1];

depending on which mouseclick function is causing the issue (presume it happens on all of the images with this event)

Alternatively find the call where it's breaking in the CarbonCanvas and use abs() as in;

..GetPixel(abs(X), abs(Y));

it looks as if it's just negating the co-ordinates incorrectly (albeit off by one) and this should put them positive again although they may not be the actual pixel you're clicking (it's a bit of an estimate anyway on a stretched image - I couldn't readily find a function to work out the actual underlying pixel coordinates of a stretched image from the XY coords of a mouseclick event).

Iit might be worth checking to see if there are any updates in svn for the Carbon Canvas - I'd have thought GetPixel is a fairly common function to call and it should be causing issue elsewhere if it's not working correctly.

TheBlackSheep

Hi, i'v reported on bugtraker what i'v digget out, the coords are always positive so it must be some internal problems

In TCarbonBitmapContext.GetPixel the is an assignment

R := Classes.Rect(X, Y, 1, 1);

The R is the propagated to TCarbonWidgetSet.RawImage_FromCarbonBitmap and in this piece of code

Width := R.Right - R.Left;
Height := R.Bottom - R.Top;

Width and Height become negative... If i put S.X, S.Y instead of 1, 1 and remove the Right and Bottom it still not fixed because after returning back to GetPixel i then get "Invalid horizontal pixel index error" in

Result := IntfImage.TColors[X, Y];

Seems all ok after setting [X-1, Y-1], but sometimes instead of green color i get blue, brown or clBlack color.

Also sporadically it raise an access violation on the first assignment line (VBytes.Ah := B0;) of TLazIntfImage.GetColor_BPP32_A8R8G8B8_BIO_TTB

[TheBlackSheep]

hi circular

I've retried with abs() but can't seem to get the effect as good as using BGRADiff -  I've attached the amended source of Unit2 so you'd need to copy these over the source of the original if you want to play around with this - tbh I couldn't get my head around how this formula works with the colorspace in relation to the green background but you'd be better placed to understand that ...

Ok Shebuka - sorry, I couldn't help on the Mac side hopefully it'll get fixed shortly.

TheBlackSheep

[circular]

Okay, try with this adjustment :