Lazarus

Programming => Graphics and Multimedia => Graphics => Topic started by: af0815 on February 20, 2020, 02:52:38 pm

Title: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 20, 2020, 02:52:38 pm
I get a raw picture from a camera and i have to convert from SRGGB8 (or 16) to RGB.

A (long) search give me no (native) implementation in pascal. In LazVideoCapture i found YUV to RGB. The camera works with this, but slow 1 FPS. With bayer i get a speed of 19 FPS. Tested with the v4l2 test utility

Is there any implementation known of deBayering ?

 
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: circular on February 21, 2020, 02:36:41 pm
I think it is not too complicated to write debayering. Here you can see the matrix of colors:
https://en.wikipedia.org/wiki/Bayer_filter#/media/File:Bayer_pattern_on_sensor_profile.svg

Basically you need to interpolate the 3 color planes. First put the know values into the corresponding pixels in the final image.

Interpolating between pixels is the mean of the know pixels. There are some edge cases to consider.


Title: Re: deBayering - Bayer to RGB conversion needed
Post by: marcov on February 21, 2020, 03:38:29 pm
debayering is going from bayer to rgb.

I have some code but it is not that fast, but I lack a good test image. Do you have an example of your camera. Preferably with some object in primary colors, and a description of which is which.

There are several different bayer formats, depending on which line is the first line.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 21, 2020, 08:43:00 pm
V4L2 Test Bench says RGGB (8-Bit Bayer RGRG/GBGB) (70434752-0000-0010-8000-00aa003). Possible is also RG16 (16-Bit Bayer RGRG/GBGB). I have cameras and a running fpcVideo4L2.
Cameras:
DFK 33UX183 = 5.472×3.648 (20 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
DFK 33UX226 = 4.000×3.000 (12 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
from TheImageSource.

What is a good Testimage :-) ?! I have made a picture converted to jpg and RawDate of RGGB. But i am not able to share it here in the forum because the 7z it is wit 8MB to large.

https://drive.google.com/file/d/18zoyCfAWu8s2UGoqu9GgOKcMH1z4IDX8/view?usp=sharing

My Testcode to generate the raw data

Code: Pascal  [Select][+][-]
  1. procedure TFormVideoTmp.VideoFrameSynchronized(Sender: TObject;
  2.   Buffer: pointer; Size: integer; Error: boolean);
  3. begin
  4.  
  5.   if (FScanInProgress) then exit;
  6.  
  7.   FScanInProgress:=True;
  8.  try
  9.    // V4L2_PIX_FMT_SRGGB8 for theImageSource possible
  10.     if Video.PixelFormat = V4L2_PIX_FMT_SRGGB8 then
  11.     begin
  12.       aMS.Clear;
  13.       aMS.Position:=0;
  14.       aMS.WriteBuffer(Buffer^,Size);
  15.       aMS.Position:=0;
  16.       aMS.SaveToFile('/tmp/test.raw');
  17.     end;
  18.  
  19.     if Video.PixelFormat = V4L2_PIX_FMT_YUYV then
  20.     begin
  21.       BMP.BeginUpdate;
  22.       YUYV_to_BGRA(PLongWord(Buffer), PLongWord(BMP.RawImage.Data), BMP.Width*BMP.Height);
  23.       BMP.EndUpdate;
  24.     end;
  25.  
  26.     if Video.PixelFormat = V4L2_PIX_FMT_MJPEG then
  27.     begin
  28.       aMS.Clear;
  29.       aMS.Position:=0;
  30.       aMS.WriteBuffer(Buffer^,Size);
  31.       aMS.Position:=0;
  32.       aImage.LoadFromStream(aMS,aImgReader);
  33.       aMS.Clear;
  34.       aMS.Position:=0;
  35.       aImage.SaveToStream(aMS,aImgWriter);
  36.       aMS.Position:=0;
  37.       BMP.BeginUpdate;
  38.       Move(aMS.Memory^,PByte(BMP.RawImage.Data)^,Video.Width*Video.Height*aImgWriter.Bpp);
  39.       BMP.EndUpdate;
  40.     end;
  41.   finally
  42.     FScanInProgress:=False;
  43.   end;
  44. end;
  45.  

 
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: marcov on February 22, 2020, 04:12:06 pm
I get an image that looks similar in about 50ms on a core i7-3770 using Delphi. (Delphi because I used some image routines from work for ease of use, but the procedure itself is compiler agnostic). The code could maybe get a bit faster, but for some real performance one would probably need SSE/AVX.

Since the output image is about 60MB, it is about 1MB/ms, which is still way under move() performance, so it is not yet cache bound yet.

The code originally may have come from OpenCV, considering the CV_ identifiers.

Code: Pascal  [Select][+][-]
  1. {$O+}
  2. type
  3.   cvsize = packed record
  4.     width : cint;
  5.     height: cint;
  6.     end;
  7.   cvstatus=cint;
  8.  
  9.  
  10. function icvBayer2BGR_8u_C1C3R( bayer0:pbbyte;bayer_step:cint;dst0:pbbyte;dst_step:cint;size:cvsize;code:cint):CVSTATUS;
  11.  
  12. var blue : cint;
  13.     start_with_green : cint;
  14.     t0,t1 :cint;
  15.     bayer_end,
  16.     bayer,dst : pbbyte;
  17.  
  18. label 123;
  19. begin
  20.     if (code= CV_BayerBG2BGR ) or (code= CV_BayerGB2BGR) then
  21.       blue:=-1
  22.     else
  23.       blue:=1;
  24.     if (code = CV_BayerGB2BGR) or (code =CV_BayerGR2BGR) then
  25.      start_with_green:=1
  26.     else
  27.      start_with_green:=0;
  28. //    int blue = code == CV_BayerBG2BGR || code == CV_BayerGB2BGR ? -1 : 1;
  29.  
  30.  
  31. //    int start_with_green = code == CV_BayerGB2BGR || code == CV_BayerGR2BGR;
  32.     fillchar(dst0^,size.width*3*sizeof(dst0[0]),#0);
  33.     fillchar( (dst0 + (size.height - 1)*dst_step)^, size.width*3*sizeof(dst0[0]),#0 );
  34.  
  35. //    memset( dst0, 0, size.width*3*sizeof(dst0[0]) );
  36. //    memset( dst0 + (size.height - 1)*dst_step, 0, size.width*3*sizeof(dst0[0]) );
  37.     inc(dst0 ,dst_step + 3 + 1);
  38.     dec(size.height , 2);
  39.     dec(size.width , 2);
  40.     repeat
  41. //    for( ; size.height-- > 0; bayer0 += bayer_step, dst0 += dst_step )
  42.     bayer:=bayer0;
  43.     dst:=dst0;
  44.  
  45.  
  46.        bayer_end:=bayer+size.width;
  47. {        const uchar* bayer = bayer0;
  48.         uchar* dst = dst0;
  49.         const uchar* bayer_end = bayer + size.width;
  50.  }
  51.  
  52.         dst[-4] :=0;
  53.         dst[-3] :=0;
  54.         dst[-2] :=0;
  55.         dst[size.width*3-1] :=0;
  56.         dst[size.width*3]   :=0;
  57.         dst[size.width*3+1] :=0;
  58.  
  59.         if( size.width <= 0 ) then
  60.             goto 123; // continue; // can't use contineu because of decrement operator abuse + for
  61.  
  62.         if( start_with_green )<>0 then
  63.          begin
  64.             t0 := (ord(bayer[1]) + ord(bayer[bayer_step*2+1]) + 1) shr {>>} 1;
  65.             t1 := (ord(bayer[bayer_step]) + ord(bayer[bayer_step+2]) + 1) shr {>>} 1;
  66.             dst[-blue] := t0;
  67.             dst[0] := bayer[bayer_step+1];
  68.             dst[blue] := (t1);
  69.             inc(bayer);
  70.             inc(dst, 3);
  71.          end;
  72.  
  73.         if( blue > 0 ) then
  74.          begin
  75.             repeat
  76.  //           for( ; bayer <= bayer_end - 2; bayer += 2, dst += 6 )
  77.  
  78.                 t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +
  79.                       ord(bayer[bayer_step*2+2]) + 2) shr {>>} 2;
  80.                 t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +
  81.                       ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) shr {>>} 2;
  82.                 dst[-1] := (t0);
  83.                 dst[0] := (t1);
  84.                 dst[1] := bayer[bayer_step+1];
  85.  
  86.                 t0 := (ord(bayer[2]) + ord(bayer[bayer_step*2+2]) + 1) shr {>>} 1;
  87.                 t1 := (ord(bayer[bayer_step+1]) + ord(bayer[bayer_step+3]) + 1) shr {>>} 1;
  88.                 dst[2] := (t0);
  89.                 dst[3] := bayer[bayer_step+2];
  90.                 dst[4] := (t1);
  91.  
  92.               inc(bayer,2);
  93.               inc(dst,6);
  94.             until bayer>(bayer_end-2);
  95.          end
  96.         else
  97.          begin
  98. //            for( ; bayer <= bayer_end - 2; bayer += 2, dst += 6 )            {\
  99.            repeat
  100.                 t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +
  101.                       ord(bayer[bayer_step*2+2]) + 2) {>>} shr 2;
  102.                 t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +
  103.                       ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) {>>} shr 2;
  104.                 dst[1] := (t0);
  105.                 dst[0] := (t1);
  106.                 dst[-1] := bayer[bayer_step+1];
  107.  
  108.                 t0 := (ord(bayer[2]) + ord(bayer[bayer_step*2+2]) + 1) {>>} shr 1;
  109.                 t1 := (ord(bayer[bayer_step+1]) + ord(bayer[bayer_step+3]) + 1) {>>} shr 1;
  110.                 dst[4] := (t0);
  111.                 dst[3] := bayer[bayer_step+2];
  112.                 dst[2] := (t1);
  113.                 inc(bayer,2);
  114.               inc(dst,6);
  115.             until bayer>(bayer_end-2);
  116.          end;
  117.  
  118.         if( bayer < bayer_end ) then
  119.         begin
  120.             t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +
  121.                   ord(bayer[bayer_step*2+2]) + 2) {>>} shr 2;
  122.             t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +
  123.                   ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) shr { >>} 2;
  124.             dst[-blue] := (t0);
  125.             dst[0] := (t1);
  126.             dst[blue] := bayer[bayer_step+1];
  127.             inc(bayer);
  128.             inc(dst,3);
  129.         end;
  130.  
  131.         blue := -blue;
  132.         start_with_green := not start_with_green;
  133. //( ; size.height-- > 0; bayer0 += bayer_step, dst0 += dst_step )
  134.  
  135.   123:
  136.      inc(bayer0,bayer_step);
  137.      inc(dst0,dst_step);
  138.      dec(size.height);
  139.     until (size.height<=0);
  140.  
  141.     result:=CV_OK;
  142. end;
  143.  

called as

Code: Pascal  [Select][+][-]
  1.    icvBayer2BGR_8u_C1C3R((b.getimagepointer(0,0)),b.RowPitch,pbbyte(b2.getimagepointer(0,0)),b2.RowPitch,x,  CV_BayerBG2BGR);
  2.  

where getimagepointer just gets a pointer to the first pixel of a raw image and "rowpitch" is the difference (in bytes) between the first pixels of two rows that follow eachother (@scanline[1]-@scanline[0]).  x is a TSize like record with dimensions


Of course the smart thing to do (if you have at least a dual core) is to capture in one thread, and debayer in another. But it depends on if the video driver can operate zero copy (iow if you can queue your own buffers)
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 23, 2020, 09:41:44 am
Of course the smart thing to do (if you have at least a dual core) is to capture in one thread, and debayer in another. But it depends on if the video driver can operate zero copy (iow if you can queue your own buffers)
Thank you for sharing the code, i have now a lot to do :-) The 2 CPU solution is not so important, because i take one picture out of the stream comming from the camera, convert this picture and store it on a server. After then i have to wait a long time for the next picture.
I must have the camera running because the light is complex and can change. If i make only one shoot with the camera, the automatic gain is not working correct. If the camera is working the full time, the result is nearly perfect for my need.

The camera is excelent, because you can store fixed startup configuration. so you can test it life. After the tests are ok, you can fix nearly all params in the camera and can store it with the original software of the supplieren. This can be done in the next majore release of his software (i have test this with a prerelease). The Camera itselft is not expensive and the support is excellent. Working on WIndows and Linux. Windows have a well documented API (win32 and win64) and in Linux you can work with V4L2 and gstreamer. Both soloutions are working on Rasbian (RPi4, poor on RPi3B+) (Linux/arm) and Linux/64 (tested). Gstreamer can work with the GigE Versions too, is a little limited in Speed because i need a secand Ethetnet Adapter (one for camera with changed MTU)   

I will test it on a RasPi 4 with 4GB :-) because this should be the targetplattform. And for me, its FPC/Lazarus only. 
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: georgje on February 25, 2020, 10:29:44 am
My first post here. I'm looking for something like this too.

Raw data, the sensor image is stored as 12 or 14 bit, Nikon anyway. Can anybody explain to me what SRGGB8 means. If I google it looks like a list of byte representing the colors in a specific sequence. But I'm missing 4 or 6 bits.

How do you get to that raw data?

George

Title: Re: deBayering - Bayer to RGB conversion needed
Post by: marcov on February 25, 2020, 10:39:13 am
Btw, what camera do you use with RPI3/4? I'm thinking about getting one, but it seems yours is quite highres? Is it linescan?
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 25, 2020, 11:19:14 am
Cameras:
DFK 33UX183 = 5.472×3.648 (20 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
DFK 33UX226 = 4.000×3.000 (12 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
from TheImageSource.

And GigE DFK 33GX178e running with a rude gstreamer implementation and with a external EthernetToUSB3 Adapter. Speed of USB3 Cams is better. Conversion from Bayer to jpg is done by gstreamer.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 25, 2020, 11:41:51 am
My first post here. I'm looking for something like this too.

Raw data, the sensor image is stored as 12 or 14 bit, Nikon anyway. Can anybody explain to me what SRGGB8 means. If I google it looks like a list of byte representing the colors in a specific sequence. But I'm missing 4 or 6 bits.

How do you get to that raw data?

George


Normally you have 8 or 16 Bits per Pixel, if not you have to look for the mapping.

I have getting the data shown in the sample near the start of the thread.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: georgje on February 25, 2020, 11:02:46 pm
I think we're meaning different types of raw data.

George
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: marcov on February 25, 2020, 11:05:50 pm
Some sensors support 12-bit "packed", which means 3 bytes for 2 samples.

But aside from that exception af0815 is right, usually pixel sizes are rounded up to whole multiples of 8 bits because of bus architectures.

Anyway, step one is establishing a very precise definition of the input format, then you can start thinking about developing (or finding) a debayer routine for it.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: marcov on February 25, 2020, 11:12:03 pm
TheImageSource.

I know them, but this was for private use, for which that is a bit out of the price range. Mostly I'm interested in black white cameras anyway.

I know the cameras, I tested their 2MPish range, but eventually went with 2mpx AVT manta 235 B/W because of (iirc) SNR. Areascan are freaks for us anyway, we usually do linescan for larger resolutions.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 26, 2020, 06:18:19 am
TheImageSource.
I know them, but this was for private use, for which that is a bit out of the price range. Mostly I'm interested in black white cameras anyway.
A cheaper version can be the Logitech Brio on the other hand, not BW but can be mounted with a screw and is stable working.

The areascan camera was choiced, because the large product is laying on a table without a movement and handled by hand. I need only one picture for documentation. 
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: georgje on February 26, 2020, 07:53:30 am
Nikon,Canon,Sony,Leica by example use 12 or 14 bits, and the choice of compressed or not.
Just a Google on "nikon raw bit depth" etc.
I'm looking for a way to reach the raw data, not for debayering but to make an impression for how the sensor catches the image.
Maybe we're looking at totally different cameras.

George
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 26, 2020, 08:00:10 am
Maybe we're looking at totally different cameras.

Yes, i am looking for my used camera only. Consumer products like Nikon, Canon, Sony, Leica are not for my production needs.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: marcov on February 26, 2020, 11:08:55 am
Maybe we're looking at totally different cameras.

Yes, i am looking for my used camera only. Consumer products like Nikon, Canon, Sony, Leica are not for my production needs.

(Sony afaik is the biggest sensor make in the world, specially CCD. Many if not most upscale industrial cameras have Sony sensors. They also had industrial area cameras afaik, but not very prominent. Nikon also is a pro club but more in lenses I think, most better quality "vision"  lenses that we buy are from Nikon)
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: af0815 on February 26, 2020, 11:52:05 am
Quote
I'm looking for a way to reach the raw data, not for debayering but to make an impression for how the sensor catches the image.
This was the reason for me to different between consumer products - make internal processing of the data and put only processed data - and industrial cameras, which normal deliver different formats for processing purposes.

consumer = get processed data in a common (maybe companyspecific) format (raw, jpeg, ...) and you take this on your vacaition
industrial = get data in a datastream (more or less preprocessed) for processing.

it my definition. Maybe i am wrong.
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: georgje on February 26, 2020, 12:28:30 pm
In what you call consumer camera's raw is raw, the Color Filter Array. It contains the digitized values of the sensor elements. Editing is done during and/or after demosaicing. The problem is to extract that array from the raw-file and eventual decompress it. Once that is done it's becoming easy.

George
Title: Re: deBayering - Bayer to RGB conversion needed
Post by: circular on February 27, 2020, 12:23:36 pm
You can already try displaying a grayscale image with whatever values you find.

If you know the image dimensions, you can divide the size in byte by the number of pixels. If it is not round at all, multiply by 8 to get the bit depth of the pixel.
TinyPortal © 2005-2018