Help with FAST read from screen, compression, and save to file.

[cpicanco]

Hi everyone,

I am writing a class to load a bitmap from a window and save it as a timestamped compreessed image in disk. So far my current approach is slow, because I am saving individual files. Here is a quick benchmark:

Load (Avg)   Save (Avg)   Sum
0,019   0,049   0,068   

This approach gives ~14fps. I would like to have at least 30 fps, but I am really not sure if it is feasible. Any suggestions would be great.
My current approach is using a TBitmap and TJPEGImage in a custom TFPCustomImage descendent:

TBMPtoJPEGConversor = class(TBitmap)
  protected
    class function GetWriterClass: TFPCustomImageWriterClass; override;
    procedure InitializeWriter(AImage: TLazIntfImage; AWriter: TFPCustomImageWriter); override;
  end;

implementation


{ TBMPtoJPEGConversor }

class function TBMPtoJPEGConversor.GetWriterClass: TFPCustomImageWriterClass;
begin
  Result := TFPWriterJPEG;
end;

procedure TBMPtoJPEGConversor.InitializeWriter(AImage: TLazIntfImage;
  AWriter: TFPCustomImageWriter);
var
  LFPWriterJPEG : TFPWriterJPEG;
begin
  inherited InitializeWriter(AImage, AWriter);
  LFPWriterJPEG := TFPWriterJPEG(AWriter);
  LFPWriterJPEG.CompressionQuality := 90;
  LFPWriterJPEG.GrayScale := False;
  LFPWriterJPEG.ProgressiveEncoding := False;
end;   

After an initial setup, I am calling it from inside a thread:

Code: Pascal  [Select][+][-]

1. procedure TVideoWriter.Execute;
2. var
3.   LConversor : TBMPtoJPEGConversor;
4.  
5.   procedure SaveToFile;
6.   var
7.     LStart  : Int64;
8.     LEnd    : Int64;
9.     //hFile: THandle;
10.     //dwBytesWritten: DWORD = 0;
11.  
12.     function Elapsed : string;
13.     begin
14.       Result := FloatToStrF(
15.         FTimestamp - StartTimestamp, ffFixed, 0, 9)+'.jpeg';
16.     end;
17.   begin
18.     with FFrame do begin
19.       // load from DC
20.       LStart := SDL_GetTicks64;
21.       LConversor.LoadFromDevice(WindowDeviceContextHandle);
22.       LEnd := SDL_GetTicks64;
23.  
24.       FTimestamp := GetTimestamp;
25.       Write('LoadFromDevice' + '|' + ((LEnd - LStart)*1e-3).ToStringF);
26.  
27.       // JPEG conversion then save to file
28.       LStart := SDL_GetTicks64;
29.       LConversor.SaveToFile(Elapsed);
30.       LEnd := SDL_GetTicks64;
31.  
32.       WriteLn('|SaveToFile' + '|' + ((LEnd - LStart)*1e-3).ToStringF);
33.     end;
34.   end;
35.  
36. begin
37.   NameThreadForDebugging(ClassName);
38.   LConversor := TBMPtoJPEGConversor.Create;
39.   try
40.     while not Terminated do begin
41.       SaveToFile;
42.       SDL_Delay(1000); // will adjust later
43.     end;
44.   finally
45.     LConversor.Free;
46.   end;
47.   Log(ClassName+': Terminated ...');
48. end;
49.  
50.  

PS.:

Hi everyone,

Here is some more background information about what I am doing. This time I wrote some direct questions.

TLDR
- What is the fastest way you know to make a copy of some part of the screen (like a window) for video recording?
- Would the libpng implementation of TCastleImage be faster for reading from screen and writing to file?
- Do you known any up-to-date av (ffmpeg) bindings for Free Pascal?
- Castle Engine have a wrapper around ffmpeg for calling it with `TProcess` and save some short effects. Do you know if this same approach can be used to save a continuous stream of frames?
- I am sharing my PyAv blueprints for a video recorder in Free Pascal.
- I am sharing some ideas for an image conversor/compressor based on TBitmap from Graphics (LCL).
___

I am trying to write a fast screen recorder for https://github.com/cpicanco/stimulus-control-sdl2. This program is an experiment builder and I am dealing with eye movements tracking. For my urgent needs, I need to record timestamped frames with something around 30fps. I did some tests with Windows APIs (mainly based on reading a device context (DC), then `BitBlt` and `GetDIBits`). This approach is fast (~30ms) for copying from memory and save to file, but 4mb bitmap files are not feasible, I am expecting 40min-60min recordings. So, I need some fast compression before saving a frame.

I also did some tests with `TBitmap` from `Graphics` unit (LCL) and its dependencies (`FPImage` writers and readers). It was really easy to create a "Conversor" class, for example:

Code: Pascal  [Select][+][-]

1. unit GraphicsBMPtoPNG;
2.  
3. {$mode objfpc}{$H+}
4.  
5. interface
6.  
7. uses
8.   Classes, SysUtils, Graphics, IntfGraphics, FPImage, FPWritePNG, FPReadBMP;
9.  
10. type
11.  
12.   { TBMPtoPNGConversor }
13.  
14.   TBMPtoPNGConversor = class(TBitmap)
15.   protected
16.     class function GetWriterClass: TFPCustomImageWriterClass; override;
17.     procedure InitializeWriter(AImage: TLazIntfImage; AWriter: TFPCustomImageWriter); override;
18.   end;
19.  
20. implementation
21.  
22. uses ZStream;
23.  
24. { TBMPtoPNGConversor }
25.  
26. class function TBMPtoPNGConversor.GetWriterClass: TFPCustomImageWriterClass;
27. begin
28.   Result := TFPWriterPNG;
29. end;
30.  
31. procedure TBMPtoPNGConversor.InitializeWriter(AImage: TLazIntfImage;
32.   AWriter: TFPCustomImageWriter);
33. var
34.   LFPWriterPNG : TFPWriterPNG;
35. begin
36.   inherited InitializeWriter(AImage, AWriter);
37.   LFPWriterPNG := AWriter as TFPWriterPNG;
38.   LFPWriterPNG.CompressionLevel := clmax;
39.   LFPWriterPNG.GrayScale := False;
40.   LFPWriterPNG.UseAlpha := False;
41. end;
42.  
43. end.

So, I thought I could write a reader/writer based on libpng to test if it will improve the speed of my frame by frame recorder.

The frame-by-frame recorder uses a TThread. The TThread.Execute method creates a timestamp, loads the image from screen and save it to a file:

Code: Pascal  [Select][+][-]

1. // TVideoWriter = class (TThread)
2. procedure TVideoWriter.Execute;
3. var
4.   LConversor : TBMPtoPNGConversor;
5.   LStart  : Int64;
6.   LEnd    : Int64;
7.  
8.   function Elapsed : string;
9.   begin
10.     Result := 'recording'+DirectorySeparator+FloatToStrF(
11.       FFrame.Timestamp - StartTimestamp, ffFixed, 0, 9)+'.png';
12.   end;
13.  
14. begin
15.   NameThreadForDebugging(ClassName);
16.   LConversor := TBMPtoPNGConversor.Create;
17.   try
18.     while not Terminated do begin
19.       FFrame.Timestamp := ET.Elapsed;
20.       LConversor.LoadFromDevice(FFrame.WindowDeviceContext);
21.       LConversor.SaveToFile(Elapsed);
22.     end;
23.   finally
24.     LConversor.Free;
25.   end;
26.   Log(ClassName+': Terminated ...');
27. end;

After making sure that my frame-by-frame recorder is doing a good work, it would be great to have a proper video recorder writter in Free Pascal. Right now, I have a prototype in PyAv:

Code: Python  [Select][+][-]

1. import os
2. import av
3. import glob
4. from PIL import Image
5. from fractions import Fraction
6.  
7. input_dir = os.path.dirname(os.path.abspath(__file__))
8. image_files = glob.glob(f"{input_dir}/*.png")
9. image_files_timestamps = [float(os.path.basename(i).replace(',', '.').replace('.png', '')) for i in image_files]
10. image_files_timestamps_dts = [t - image_files_timestamps[0] for t in image_files_timestamps]
11. image_files = [(x, t, d) for t, x, d in sorted(zip(image_files_timestamps, image_files, image_files_timestamps_dts))]
12. video_duration = image_files[-1][1]-image_files[0][1]
13. fps_p = len(image_files)/video_duration
14. fps = round(fps_p)
15.  
16. container = av.open("output.mp4", 'w')
17. video_stream = container.add_stream('libx265', rate=fps)
18. image = Image.open(os.path.join(input_dir, image_files[0][0]))
19. video_stream.width = image.width
20. video_stream.height = image.height
21. video_stream.codec_context.time_base = Fraction(1, 1000000000)
22.  
23. first_timestamp = None
24. for (image_file, timestamp, dts) in image_files:
25.     image_path = os.path.join(input_dir, image_file)
26.  
27.     frame_pts = timestamp
28.     frame_pts *= 1 + (fps_p - fps) / fps_p
29.  
30.     frame = av.VideoFrame.from_image(Image.open(image_file))
31.     frame.time_base = video_stream.codec_context.time_base
32.     frame.pts = int(round(frame_pts / video_stream.codec_context.time_base))
33.     frame.dts = int(round(dts / video_stream.codec_context.time_base))
34.  
35.     packet = video_stream.encode(frame)
36.     container.mux(packet)
37. container.mux(video_stream.encode())
38. container.close()

However, I am not sure if there are any up-to-date av (ffmpeg) bindings for free pascal (compatible with this prototype).

[dsiders]

So is there a question hidden in there somewhere?

[TRon]

So is there a question hidden in there somewhere?

Me thinks so ?

This approach gives ~14fps. I would like to have at least 30 fps, but I am really not sure if it is feasible. Any suggestions would be great.

Don't convert to jpeg before saving ? Use faster storage device ? hard to tell because it is not clear where the bottleneck is located.

[cpicanco]

Some more data. I am pretty sure that there is room for improvements, specially in "SaveToFile". I am looking for ideas on how to improve speed in software part. Of course a better gear would improve these results though.

in miliseconds
Load (Avg)   Save (Avg)   Sum
0,013   0,050   0,063

[cpicanco]

Here is some profiling data. It seems that people know that JPEG_FDCT_ISLOW 

JPEG_FDCT_ISLOW (15%, 307 samples)
TLAZINTFIMAGE.GETCOLOR_BPP32_B8G8R8A8_BIO_TTB (14%, 284 samples)
TWIN32WIDGETSET.RAWIMAGE_FROMDEVICE* (11%, 225 samples)
TLAZINTFIMAGE.GETINTERNALCOLOR (9%, 185 samples)
FORWARD_DCT (7%, 147 samples)
DESCALE (7%, 144 samples)
RGB_YCC_CONVERT (7%, 133 samples)
TFPWRITERJPEG.WRITEPIXELS (5%, 102 samples)
MOVE (5%, 99 samples)
TFPCUSTOMIMAGE.GETCOLOR (4%, 70 samples)
ENCODE_ONE_BLOCK (2%, 48 samples)
TFPCUSTOMIMAGE.CHECKINDEX (2%, 45 samples)
MULTIPLY (2%, 44 samples)
FILECREATE* (2%, 38 samples)
H2V2_DOWNSAMPLE (1%, 24 samples)
EMIT_BITS (1%, 19 samples)
CFREEMEM* (1%, 10 samples)
TWIN32WIDGETSET.DELETEOBJECT* (1%, 10 samples)
COMPRESS_DATA (0%, 9 samples)
FILEWRITE* (0%, 9 samples)
ENCODE_MCU_HUFF (0%, 8 samples)
CREALLOCMEM* (0%, 8 samples)
DO_WRITE* (0%, 8 samples)
GETBITMAPORDER* (0%, 5 samples)
FILECLOSE* (0%, 4 samples)
DOBITMAP* (0%, 3 samples)
TWIN32WIDGETSET.GETDEVICESIZE* (0%, 2 samples)
TRAWIMAGELINESTARTS.INIT (0%, 1 samples)
_fpc_local_unwind* (0%, 1 samples)
TSTRINGLIST.INTERNALCLEAR (0%, 1 samples)
PRE_PROCESS_DATA (0%, 1 samples)
TRASTERIMAGE.SAVESTREAMNEEDED (0%, 1 samples)
CGETMEM* (0%, 1 samples)
FMEMCOPY (0%, 1 samples)
SYSRELOCATETHREADVAR* (0%, 1 samples)
SEP_DOWNSAMPLE (0%, 1 samples)

[TRon]

I am pretty sure that there is room for improvements, specially in "SaveToFile". I am looking for ideas on how to improve speed in software part.

- Don't store into bmp (raw data is always faster)
- don't convert/save to jpeg (raw data is always faster)
- stupid idea but have you considered threads ? (f.e. even/odd frames should be able to improve your framerate by almost double)

btw your profiling is interesting. Seems that the file saving itself is not so much a problem but the time for file creation seem a bit steep in relation to the other file operations. At least it shows it is not a bottleneck (yet  ).

[domasz]

Do the same benchmark but don't save to file, save to TMemoryStream. I think you should get a lot more FPS.
If so then the solution might be to first save frames to TMemoryStream and then to one big file, or several big files.
Depends how much data you have. If you want to save 10 minutes of video then 1 file should be fine. But if you want to save hours of video stream then you should every X minutes create a new file or append to the previous file.
JPEG should be fast enough to give you 50 fps if the resolution is not super ultra HD. But I don't know what kind of computer you have.

[cpicanco]

- Don't store into bmp (raw data is always faster)

How to copy a raw image from the device context?

- don't convert/save to jpeg (raw data is always faster)

I did some tests using only Windows APIs and bitmaps will take 4mb for 1366 x 768. Saving them as such is faster, but I expect 1h recordings, so not feasible. I really need some sort of fast compression.

- stupid idea but have you considered threads ? (f.e. even/odd frames should be able to improve your framerate by almost double)

I am already running from a separated thread.

But if you want to save hours of video stream then you should every X minutes create a new file or append to the previous file.

I see. I am facing some difficulties, it is hard to create a video from scratch. I could not find av bindings (like PyAv) for free pascal. I know it is part of ffmpeg, but do you know where to find ffmpeg bindings for free pascal?

__

Using the FPWriterPNG gives much better compression (~6kb for png,  ~18kb for jpeg) but a lot slower too (6fps).

[cpicanco]

Hi everyone,

Here is some more background information about what I am doing. This time I wrote some direct questions.

TLDR
- What is the fastest way you know to make a copy of some part of the screen (like a window) for video recording?
- Would the libpng implementation of TCastleImage be faster for reading from screen and writing to file?
- Do you known any up-to-date av (ffmpeg) bindings for Free Pascal?
- Castle Engine have a wrapper around ffmpeg for calling it with `TProcess` and save some short effects. Do you know if this same approach can be used to save a continuous stream of frames?
- I am sharing my PyAv blueprints for a video recorder in Free Pascal.
- I am sharing some ideas for an image conversor/compressor based on TBitmap from Graphics (LCL).
___

I am trying to write a fast screen recorder for https://github.com/cpicanco/stimulus-control-sdl2. This program is an experiment builder and I am dealing with eye movements tracking. For my urgent needs to record timestamped frames with something around 30fps. I did some tests with Windows APIs (mainly based on reading a device context (DC), then `BitBlt` and `GetDIBits`). This approach is fast (~30ms) for copying from memory and save to file, but 4mb bitmap files are not feasible, I am expecting 40min-60min recordings. So, I need some fast compression before saving a frame.

I also did some tests with `TBitmap` from `Graphics` unit (LCL) and its dependencies (`FPImage` writers and readers). It was really easy to create a "Conversor" class, for example:

Code: Pascal  [Select][+][-]

1. unit GraphicsBMPtoPNG;
2.  
3. {$mode objfpc}{$H+}
4.  
5. interface
6.  
7. uses
8.   Classes, SysUtils, Graphics, IntfGraphics, FPImage, FPWritePNG, FPReadBMP;
9.  
10. type
11.  
12.   { TBMPtoPNGConversor }
13.  
14.   TBMPtoPNGConversor = class(TBitmap)
15.   protected
16.     class function GetWriterClass: TFPCustomImageWriterClass; override;
17.     procedure InitializeWriter(AImage: TLazIntfImage; AWriter: TFPCustomImageWriter); override;
18.   end;
19.  
20. implementation
21.  
22. uses ZStream;
23.  
24. { TBMPtoPNGConversor }
25.  
26. class function TBMPtoPNGConversor.GetWriterClass: TFPCustomImageWriterClass;
27. begin
28.   Result := TFPWriterPNG;
29. end;
30.  
31. procedure TBMPtoPNGConversor.InitializeWriter(AImage: TLazIntfImage;
32.   AWriter: TFPCustomImageWriter);
33. var
34.   LFPWriterPNG : TFPWriterPNG;
35. begin
36.   inherited InitializeWriter(AImage, AWriter);
37.   LFPWriterPNG := AWriter as TFPWriterPNG;
38.   LFPWriterPNG.CompressionLevel := clmax;
39.   LFPWriterPNG.GrayScale := False;
40.   LFPWriterPNG.UseAlpha := False;
41. end;
42.  
43. end.

So, I thought I could write a reader/writer based on libpng to test if it will improve the speed of my frame by frame recorder.

The frame-by-frame recorder uses a TThread. The TThread.Execute method creates a timestamp, loads the image from screen and save it to a file:

Code: Pascal  [Select][+][-]

1. // TVideoWriter = class (TThread)
2. procedure TVideoWriter.Execute;
3. var
4.   LConversor : TBMPtoPNGConversor;
5.   LStart  : Int64;
6.   LEnd    : Int64;
7.  
8.   function Elapsed : string;
9.   begin
10.     Result := 'recording'+DirectorySeparator+FloatToStrF(
11.       FFrame.Timestamp - StartTimestamp, ffFixed, 0, 9)+'.png';
12.   end;
13.  
14. begin
15.   NameThreadForDebugging(ClassName);
16.   LConversor := TBMPtoPNGConversor.Create;
17.   try
18.     while not Terminated do begin
19.       FFrame.Timestamp := ET.Elapsed;
20.       LConversor.LoadFromDevice(FFrame.WindowDeviceContext);
21.       LConversor.SaveToFile(Elapsed);
22.     end;
23.   finally
24.     LConversor.Free;
25.   end;
26.   Log(ClassName+': Terminated ...');
27. end;

After making sure that my frame-by-frame recorder is doing a good work, it would be great to have a proper video recorder writter in Free Pascal. Right now, I have a prototype in PyAv:

Code: Python  [Select][+][-]

1. import os
2. import av
3. import glob
4. from PIL import Image
5. from fractions import Fraction
6.  
7. input_dir = os.path.dirname(os.path.abspath(__file__))
8. image_files = glob.glob(f"{input_dir}/*.png")
9. image_files_timestamps = [float(os.path.basename(i).replace(',', '.').replace('.png', '')) for i in image_files]
10. image_files_timestamps_dts = [t - image_files_timestamps[0] for t in image_files_timestamps]
11. image_files = [(x, t, d) for t, x, d in sorted(zip(image_files_timestamps, image_files, image_files_timestamps_dts))]
12. video_duration = image_files[-1][1]-image_files[0][1]
13. fps_p = len(image_files)/video_duration
14. fps = round(fps_p)
15.  
16. container = av.open("output.mp4", 'w')
17. video_stream = container.add_stream('libx265', rate=fps)
18. image = Image.open(os.path.join(input_dir, image_files[0][0]))
19. video_stream.width = image.width
20. video_stream.height = image.height
21. video_stream.codec_context.time_base = Fraction(1, 1000000000)
22.  
23. first_timestamp = None
24. for (image_file, timestamp, dts) in image_files:
25.     image_path = os.path.join(input_dir, image_file)
26.  
27.     frame_pts = timestamp
28.     frame_pts *= 1 + (fps_p - fps) / fps_p
29.  
30.     frame = av.VideoFrame.from_image(Image.open(image_file))
31.     frame.time_base = video_stream.codec_context.time_base
32.     frame.pts = int(round(frame_pts / video_stream.codec_context.time_base))
33.     frame.dts = int(round(dts / video_stream.codec_context.time_base))
34.  
35.     packet = video_stream.encode(frame)
36.     container.mux(packet)
37. container.mux(video_stream.encode())
38. container.close()
39.  

However, I am not sure if there are any up-to-date av (ffmpeg) bindings for free pascal (compatible with this prototype).

[jamie]

There is a reason why most if not all video systems in PC's have basic GPU with direct grabs of the image from hardware memory.

 You might want to to look into using DX on windows for example.

[cpicanco]

Some new results with SDL2, SDL2_image (with zlib). Getting better. Next step is to use a stream to save (I am not sure how to do this right now).

SDL_RenderReadPixels(AVG)
0,007

IMG_SavePNG(AVG)
0,020

SUM
0,027

[TRon]

How to copy a raw image from the device context?

I assume you are aware that depending on OS device context not the most fastest ways of processing graphics. Usually such things are done using your video chipset.

I did some tests using only Windows APIs and bitmaps will take 4mb for 1366 x 768. Saving them as such is faster, but I expect 1h recordings, so not feasible. I really need some sort of fast compression.

It is the compression and conversions that takes up the time. Resolution has nothing to do with it. Why would it not be feasible for longer recordings ? You mean to say that you don't have as much storage ? That is a hardware limitation, not a software one. So imho there would be no need for fast compression.

I see. I am facing some difficulties, it is hard to create a video from scratch. I could not find av bindings (like PyAv) for free pascal. I know it is part of ffmpeg, but do you know where to find ffmpeg bindings for free pascal?

There are bindings for ffmpeg available, posted on these forums if not mistaken.

Using the FPWriterPNG gives much better compression (~6kb for png,  ~18kb for jpeg) but a lot slower too (6fps).

Yes I know. The ever most fastest compression of them .... is none et all. Did you noticed yourself that that is what you actually wrote/asked there ?

There is no shame in processing the captures after storing them if you insist on and/or can't live without using a 'regular' storage format.

[KodeZwerg]

Your endless long listings of numbers.... instead of doing it like that, upload a text file for those that interested and just type average numbers in here, I've closed this topic again and again because of your technical spam 

My suggestion:

loop: canvas -> direct to unique bitmap -> direct into a unique thread that compress and save the bitmap stream in whatever way you like, when finished thread -> free bitmap

that you can do again and again until your RAM is full, so take a look on RAM
depending on how much you actually want to store (really 30FPS???.....) you need to find a good thread managing way.

[lainz]

I agree with KodeZwerg, RAM is the solution, hard disk or ssd is slow...

[Dzandaa]

Hi,

I agree with cpicanco, a video recorder completely written in Pascal and portable to all platforms would be fine.

Even AVI format.

Like
https://github.com/rolinh/libgwavi/tree/master
https://learn.microsoft.com/en-us/windows/win32/directshow/avi-riff-file-reference

Maybe a programmer younger than me will be interested in converting to Pascal


B->