### Bookstore

 Computer Math and Games in Pascal (preview) Lazarus Handbook

### Author Topic: deBayering - Bayer to RGB conversion needed  (Read 968 times)

#### af0815

• Sr. Member
• Posts: 497
##### deBayering - Bayer to RGB conversion needed
« on: February 20, 2020, 02:52:38 pm »
I get a raw picture from a camera and i have to convert from SRGGB8 (or 16) to RGB.

A (long) search give me no (native) implementation in pascal. In LazVideoCapture i found YUV to RGB. The camera works with this, but slow 1 FPS. With bayer i get a speed of 19 FPS. Tested with the v4l2 test utility

Is there any implementation known of deBayering ?

regards
Andreas

#### circular

• Hero Member
• Posts: 3149
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #1 on: February 21, 2020, 02:36:41 pm »
I think it is not too complicated to write debayering. Here you can see the matrix of colors:
https://en.wikipedia.org/wiki/Bayer_filter#/media/File:Bayer_pattern_on_sensor_profile.svg

Basically you need to interpolate the 3 color planes. First put the know values into the corresponding pixels in the final image.

Interpolating between pixels is the mean of the know pixels. There are some edge cases to consider.

Conscience is the debugger of the mind

#### marcov

• Global Moderator
• Hero Member
• Posts: 8035
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #2 on: February 21, 2020, 03:38:29 pm »
debayering is going from bayer to rgb.

I have some code but it is not that fast, but I lack a good test image. Do you have an example of your camera. Preferably with some object in primary colors, and a description of which is which.

There are several different bayer formats, depending on which line is the first line.

#### af0815

• Sr. Member
• Posts: 497
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #3 on: February 21, 2020, 08:43:00 pm »
V4L2 Test Bench says RGGB (8-Bit Bayer RGRG/GBGB) (70434752-0000-0010-8000-00aa003). Possible is also RG16 (16-Bit Bayer RGRG/GBGB). I have cameras and a running fpcVideo4L2.
Cameras:
DFK 33UX183 = 5.472×3.648 (20 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
DFK 33UX226 = 4.000×3.000 (12 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
from TheImageSource.

What is a good Testimage :-) ?! I have made a picture converted to jpg and RawDate of RGGB. But i am not able to share it here in the forum because the 7z it is wit 8MB to large.

My Testcode to generate the raw data

Code: Pascal  [Select]
1. procedure TFormVideoTmp.VideoFrameSynchronized(Sender: TObject;
2.   Buffer: pointer; Size: integer; Error: boolean);
3. begin
4.
5.   if (FScanInProgress) then exit;
6.
7.   FScanInProgress:=True;
8.  try
9.    // V4L2_PIX_FMT_SRGGB8 for theImageSource possible
10.     if Video.PixelFormat = V4L2_PIX_FMT_SRGGB8 then
11.     begin
12.       aMS.Clear;
13.       aMS.Position:=0;
14.       aMS.WriteBuffer(Buffer^,Size);
15.       aMS.Position:=0;
16.       aMS.SaveToFile('/tmp/test.raw');
17.     end;
18.
19.     if Video.PixelFormat = V4L2_PIX_FMT_YUYV then
20.     begin
21.       BMP.BeginUpdate;
22.       YUYV_to_BGRA(PLongWord(Buffer), PLongWord(BMP.RawImage.Data), BMP.Width*BMP.Height);
23.       BMP.EndUpdate;
24.     end;
25.
26.     if Video.PixelFormat = V4L2_PIX_FMT_MJPEG then
27.     begin
28.       aMS.Clear;
29.       aMS.Position:=0;
30.       aMS.WriteBuffer(Buffer^,Size);
31.       aMS.Position:=0;
33.       aMS.Clear;
34.       aMS.Position:=0;
35.       aImage.SaveToStream(aMS,aImgWriter);
36.       aMS.Position:=0;
37.       BMP.BeginUpdate;
38.       Move(aMS.Memory^,PByte(BMP.RawImage.Data)^,Video.Width*Video.Height*aImgWriter.Bpp);
39.       BMP.EndUpdate;
40.     end;
41.   finally
42.     FScanInProgress:=False;
43.   end;
44. end;
45.

« Last Edit: February 21, 2020, 08:47:57 pm by af0815 »
regards
Andreas

#### marcov

• Global Moderator
• Hero Member
• Posts: 8035
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #4 on: February 22, 2020, 04:12:06 pm »
I get an image that looks similar in about 50ms on a core i7-3770 using Delphi. (Delphi because I used some image routines from work for ease of use, but the procedure itself is compiler agnostic). The code could maybe get a bit faster, but for some real performance one would probably need SSE/AVX.

Since the output image is about 60MB, it is about 1MB/ms, which is still way under move() performance, so it is not yet cache bound yet.

The code originally may have come from OpenCV, considering the CV_ identifiers.

Code: Pascal  [Select]
1. {\$O+}
2. type
3.   cvsize = packed record
4.     width : cint;
5.     height: cint;
6.     end;
7.   cvstatus=cint;
8.
9.
10. function icvBayer2BGR_8u_C1C3R( bayer0:pbbyte;bayer_step:cint;dst0:pbbyte;dst_step:cint;size:cvsize;code:cint):CVSTATUS;
11.
12. var blue : cint;
13.     start_with_green : cint;
14.     t0,t1 :cint;
15.     bayer_end,
16.     bayer,dst : pbbyte;
17.
18. label 123;
19. begin
20.     if (code= CV_BayerBG2BGR ) or (code= CV_BayerGB2BGR) then
21.       blue:=-1
22.     else
23.       blue:=1;
24.     if (code = CV_BayerGB2BGR) or (code =CV_BayerGR2BGR) then
25.      start_with_green:=1
26.     else
27.      start_with_green:=0;
28. //    int blue = code == CV_BayerBG2BGR || code == CV_BayerGB2BGR ? -1 : 1;
29.
30.
31. //    int start_with_green = code == CV_BayerGB2BGR || code == CV_BayerGR2BGR;
32.     fillchar(dst0^,size.width*3*sizeof(dst0[0]),#0);
33.     fillchar( (dst0 + (size.height - 1)*dst_step)^, size.width*3*sizeof(dst0[0]),#0 );
34.
35. //    memset( dst0, 0, size.width*3*sizeof(dst0[0]) );
36. //    memset( dst0 + (size.height - 1)*dst_step, 0, size.width*3*sizeof(dst0[0]) );
37.     inc(dst0 ,dst_step + 3 + 1);
38.     dec(size.height , 2);
39.     dec(size.width , 2);
40.     repeat
41. //    for( ; size.height-- > 0; bayer0 += bayer_step, dst0 += dst_step )
42.     bayer:=bayer0;
43.     dst:=dst0;
44.
45.
46.        bayer_end:=bayer+size.width;
47. {        const uchar* bayer = bayer0;
48.         uchar* dst = dst0;
49.         const uchar* bayer_end = bayer + size.width;
50.  }
51.
52.         dst[-4] :=0;
53.         dst[-3] :=0;
54.         dst[-2] :=0;
55.         dst[size.width*3-1] :=0;
56.         dst[size.width*3]   :=0;
57.         dst[size.width*3+1] :=0;
58.
59.         if( size.width <= 0 ) then
60.             goto 123; // continue; // can't use contineu because of decrement operator abuse + for
61.
62.         if( start_with_green )<>0 then
63.          begin
64.             t0 := (ord(bayer[1]) + ord(bayer[bayer_step*2+1]) + 1) shr {>>} 1;
65.             t1 := (ord(bayer[bayer_step]) + ord(bayer[bayer_step+2]) + 1) shr {>>} 1;
66.             dst[-blue] := t0;
67.             dst[0] := bayer[bayer_step+1];
68.             dst[blue] := (t1);
69.             inc(bayer);
70.             inc(dst, 3);
71.          end;
72.
73.         if( blue > 0 ) then
74.          begin
75.             repeat
76.  //           for( ; bayer <= bayer_end - 2; bayer += 2, dst += 6 )
77.
78.                 t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +
79.                       ord(bayer[bayer_step*2+2]) + 2) shr {>>} 2;
80.                 t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +
81.                       ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) shr {>>} 2;
82.                 dst[-1] := (t0);
83.                 dst[0] := (t1);
84.                 dst[1] := bayer[bayer_step+1];
85.
86.                 t0 := (ord(bayer[2]) + ord(bayer[bayer_step*2+2]) + 1) shr {>>} 1;
87.                 t1 := (ord(bayer[bayer_step+1]) + ord(bayer[bayer_step+3]) + 1) shr {>>} 1;
88.                 dst[2] := (t0);
89.                 dst[3] := bayer[bayer_step+2];
90.                 dst[4] := (t1);
91.
92.               inc(bayer,2);
93.               inc(dst,6);
94.             until bayer>(bayer_end-2);
95.          end
96.         else
97.          begin
98. //            for( ; bayer <= bayer_end - 2; bayer += 2, dst += 6 )            {\
99.            repeat
100.                 t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +
101.                       ord(bayer[bayer_step*2+2]) + 2) {>>} shr 2;
102.                 t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +
103.                       ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) {>>} shr 2;
104.                 dst[1] := (t0);
105.                 dst[0] := (t1);
106.                 dst[-1] := bayer[bayer_step+1];
107.
108.                 t0 := (ord(bayer[2]) + ord(bayer[bayer_step*2+2]) + 1) {>>} shr 1;
109.                 t1 := (ord(bayer[bayer_step+1]) + ord(bayer[bayer_step+3]) + 1) {>>} shr 1;
110.                 dst[4] := (t0);
111.                 dst[3] := bayer[bayer_step+2];
112.                 dst[2] := (t1);
113.                 inc(bayer,2);
114.               inc(dst,6);
115.             until bayer>(bayer_end-2);
116.          end;
117.
118.         if( bayer < bayer_end ) then
119.         begin
120.             t0 := (ord(bayer[0]) + ord(bayer[2]) + ord(bayer[bayer_step*2]) +
121.                   ord(bayer[bayer_step*2+2]) + 2) {>>} shr 2;
122.             t1 := (ord(bayer[1]) + ord(bayer[bayer_step]) +
123.                   ord(bayer[bayer_step+2]) + ord(bayer[bayer_step*2+1])+2) shr { >>} 2;
124.             dst[-blue] := (t0);
125.             dst[0] := (t1);
126.             dst[blue] := bayer[bayer_step+1];
127.             inc(bayer);
128.             inc(dst,3);
129.         end;
130.
131.         blue := -blue;
132.         start_with_green := not start_with_green;
133. //( ; size.height-- > 0; bayer0 += bayer_step, dst0 += dst_step )
134.
135.   123:
136.      inc(bayer0,bayer_step);
137.      inc(dst0,dst_step);
138.      dec(size.height);
139.     until (size.height<=0);
140.
141.     result:=CV_OK;
142. end;
143.

called as

Code: Pascal  [Select]
1.    icvBayer2BGR_8u_C1C3R((b.getimagepointer(0,0)),b.RowPitch,pbbyte(b2.getimagepointer(0,0)),b2.RowPitch,x,  CV_BayerBG2BGR);
2.

where getimagepointer just gets a pointer to the first pixel of a raw image and "rowpitch" is the difference (in bytes) between the first pixels of two rows that follow eachother (@scanline[1]-@scanline[0]).  x is a TSize like record with dimensions

Of course the smart thing to do (if you have at least a dual core) is to capture in one thread, and debayer in another. But it depends on if the video driver can operate zero copy (iow if you can queue your own buffers)
« Last Edit: February 22, 2020, 04:20:37 pm by marcov »

#### af0815

• Sr. Member
• Posts: 497
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #5 on: February 23, 2020, 09:41:44 am »
Of course the smart thing to do (if you have at least a dual core) is to capture in one thread, and debayer in another. But it depends on if the video driver can operate zero copy (iow if you can queue your own buffers)
Thank you for sharing the code, i have now a lot to do :-) The 2 CPU solution is not so important, because i take one picture out of the stream comming from the camera, convert this picture and store it on a server. After then i have to wait a long time for the next picture.
I must have the camera running because the light is complex and can change. If i make only one shoot with the camera, the automatic gain is not working correct. If the camera is working the full time, the result is nearly perfect for my need.

The camera is excelent, because you can store fixed startup configuration. so you can test it life. After the tests are ok, you can fix nearly all params in the camera and can store it with the original software of the supplieren. This can be done in the next majore release of his software (i have test this with a prerelease). The Camera itselft is not expensive and the support is excellent. Working on WIndows and Linux. Windows have a well documented API (win32 and win64) and in Linux you can work with V4L2 and gstreamer. Both soloutions are working on Rasbian (RPi4, poor on RPi3B+) (Linux/arm) and Linux/64 (tested). Gstreamer can work with the GigE Versions too, is a little limited in Speed because i need a secand Ethetnet Adapter (one for camera with changed MTU)

I will test it on a RasPi 4 with 4GB :-) because this should be the targetplattform. And for me, its FPC/Lazarus only.
regards
Andreas

#### georgje

• Newbie
• Posts: 4
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #6 on: February 25, 2020, 10:29:44 am »
My first post here. I'm looking for something like this too.

Raw data, the sensor image is stored as 12 or 14 bit, Nikon anyway. Can anybody explain to me what SRGGB8 means. If I google it looks like a list of byte representing the colors in a specific sequence. But I'm missing 4 or 6 bits.

How do you get to that raw data?

George

#### marcov

• Global Moderator
• Hero Member
• Posts: 8035
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #7 on: February 25, 2020, 10:39:13 am »
Btw, what camera do you use with RPI3/4? I'm thinking about getting one, but it seems yours is quite highres? Is it linescan?

#### af0815

• Sr. Member
• Posts: 497
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #8 on: February 25, 2020, 11:19:14 am »
Cameras:
DFK 33UX183 = 5.472×3.648 (20 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
DFK 33UX226 = 4.000×3.000 (12 MP) with USB3.0 8-Bit Monochrome, 8-Bit Bayer (RG), 12-Bit Bayer Packed (RG), 16-Bit Bayer (RG), RGB24, YUV 4:2:2, YUV 4:1:1
from TheImageSource.

And GigE DFK 33GX178e running with a rude gstreamer implementation and with a external EthernetToUSB3 Adapter. Speed of USB3 Cams is better. Conversion from Bayer to jpg is done by gstreamer.
« Last Edit: February 25, 2020, 11:36:31 am by af0815 »
regards
Andreas

#### af0815

• Sr. Member
• Posts: 497
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #9 on: February 25, 2020, 11:41:51 am »
My first post here. I'm looking for something like this too.

Raw data, the sensor image is stored as 12 or 14 bit, Nikon anyway. Can anybody explain to me what SRGGB8 means. If I google it looks like a list of byte representing the colors in a specific sequence. But I'm missing 4 or 6 bits.

How do you get to that raw data?

George

Normally you have 8 or 16 Bits per Pixel, if not you have to look for the mapping.

I have getting the data shown in the sample near the start of the thread.
regards
Andreas

#### georgje

• Newbie
• Posts: 4
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #10 on: February 25, 2020, 11:02:46 pm »
I think we're meaning different types of raw data.

George

#### marcov

• Global Moderator
• Hero Member
• Posts: 8035
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #11 on: February 25, 2020, 11:05:50 pm »
Some sensors support 12-bit "packed", which means 3 bytes for 2 samples.

But aside from that exception af0815 is right, usually pixel sizes are rounded up to whole multiples of 8 bits because of bus architectures.

Anyway, step one is establishing a very precise definition of the input format, then you can start thinking about developing (or finding) a debayer routine for it.

#### marcov

• Global Moderator
• Hero Member
• Posts: 8035
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #12 on: February 25, 2020, 11:12:03 pm »
TheImageSource.

I know them, but this was for private use, for which that is a bit out of the price range. Mostly I'm interested in black white cameras anyway.

I know the cameras, I tested their 2MPish range, but eventually went with 2mpx AVT manta 235 B/W because of (iirc) SNR. Areascan are freaks for us anyway, we usually do linescan for larger resolutions.

#### af0815

• Sr. Member
• Posts: 497
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #13 on: February 26, 2020, 06:18:19 am »
TheImageSource.
I know them, but this was for private use, for which that is a bit out of the price range. Mostly I'm interested in black white cameras anyway.
A cheaper version can be the Logitech Brio on the other hand, not BW but can be mounted with a screw and is stable working.

The areascan camera was choiced, because the large product is laying on a table without a movement and handled by hand. I need only one picture for documentation.
« Last Edit: February 26, 2020, 06:20:22 am by af0815 »
regards
Andreas

#### georgje

• Newbie
• Posts: 4
##### Re: deBayering - Bayer to RGB conversion needed
« Reply #14 on: February 26, 2020, 07:53:30 am »
Nikon,Canon,Sony,Leica by example use 12 or 14 bits, and the choice of compressed or not.
Just a Google on "nikon raw bit depth" etc.
I'm looking for a way to reach the raw data, not for debayering but to make an impression for how the sensor catches the image.
Maybe we're looking at totally different cameras.

George