Delphi Firemonkey and Multimedia, specially render/display video, should I switch to another language?

  

I am trying to create a multimedia player, to be able to compile the project for different platforms, it looked like a good choise to use FMX. I am quite new to multimedia, also to firemonkey.

However, first, I have tried TMediaplayer; which needs codecs to be able to play Audio/Video formats which are not supported by default by the OS, for windows it’s not a big problem, but for example for android it looks like it’s not so easy to find and install together with the created apk.
A bigger problem for me, TMediaplayerControl doesn’t allow/display any controls above itself while playing.

Second try was, SDL2 + FFmpeg libraries. In this case, when I use VCL mode, I can design the form on any way I like, but this is only for windows.
SDL+FMX+FFmpeg creates a blank (video)form because SDL needs the handle of a component to create a part of the form into videosurface (Tpanel).

Currently, using FFmpeg libraries for decoding video/audio and using Timage for displaying the pictures(video). Placeing a TImage component on the form, draw every decoded picture into Timage.Bitmap.
This works, it’s fast enough (tried only on windows currently), I can put other controls like progressbar above TImage. It’s great ! the progress is moving, video is going smooth… Until I grab the trackbar and move to seek the video or set the volume. In this case, video starts stuttering until I release the trackbar or stop moving the TTrackbar.
I was planning to display some kind of a list over the videosurface (TImage in this case), think of a Channel list on a TV. If only one trackbar can slow down the refreshing of pictures, I cannot imagine what will happen when scrolling a list (FMX UI or Alicone components are suitable, standard FMX listview is not even fast enough for standard usage)?

procedure TPlayer2.Video_Display(rem_time: PDouble);
var af:PFrame;
fpict:pAVFrame;
vdelay,
adelay:Double;
data:TBitmapData;
fbmp:TBitmap;
ret:Integer;
srect, drect:TRectF;
tw,th:Single;
txt:string;
fs:TFontStyles;
begin
if (vState <> vsPlaying) and (not paused)
then vState := vsPlaying;
if quit
then begin
rem_time^ := -1;
Exit;
end;

adelay := 0;
vdelay := 0;
af := frame_queue_get(pictq);
if af = nil
then begin
if pictq.eof
then rem_time^ := AVERROR_EOF;
exit;
end;

vdelay := af.pts – last_vid_pts;
if last_aud_pts > 0
then adelay := af.pts – last_aud_pts
else adelay := vdelay;

vid_pts := af.pts;
last_vid_pts := vid_pts;

rem_time^ := FFMIN_d(vdelay, adelay);
if (rem_time^ <= 0) or (rem_time^ >= 0.1)
then rem_time^ := 0.01;

fpict := av_frame_alloc;
try
fpict.format := integer(AV_PIX_FMT_BGRA);
fpict.height := af.pframe.height;
fpict.width := af.pframe.width;

sws_ctx := sws_getCachedContext(sws_ctx, af.pframe.width, af.pframe.height,
AVPixelFormat( af.pframe.format), fpict.width, fpict.height,
AVPixelFormat( fpict.format), SWS_BICUBIC, nil, nil,nil);

try
fbmp := TBitmap.Create(fpict.width, fpict.height);

//fbmp.SetSize();
fbmp.Canvas.Lock;
//size := av_image_get_buffer_size(AVPixelFormat( pict.format), pict.width, pict.height, 1);

fbmp.Map(TMapAccess.Write, data);
ret := av_image_fill_arrays(@fpict.data, @fpict.linesize, data.Data, AVPixelFormat( fpict.format), fpict.width, fpict.height, 1);
//avpicture_fill(pavpicture(pict), data.Data, AVPixelFormat( pict.format) , af.pframe.width, af.pframe.height);
sws_scale(sws_ctx,
@af.pframe.data,
@af.pframe.linesize,
0,
af.pframe.height,
@fpict.data,
@fpict.linesize
);

fbmp.Unmap(data);

if (sub_index >= 0) and (sub_dec.codec_ctx.codec.id = AV_CODEC_ID_SUBRIP)
then begin
if subframe <> nil
then begin
if ((subframe.pts + subframe.duration) < af.pts)
then begin
av_freep(subframe);
subframe := nil;
end;
end;

if (subpicq.size > 0) and (subframe = nil)
then subframe := frame_queue_get(subpicq);

if (subframe <> nil) and
(af.pts >= subframe.pts) and
(af.pts <= subframe.pts + subframe.duration) and
(subframe.subtext <> ”)
then begin
srect := fbmp.BoundsF;
//bmp.Canvas.Lock;

fbmp.Canvas.Font.Size := round(18 * (srect.Width / srect.Height));

fs := [];
txt := UnformatText(subframe.subtext, fs);

tw := Round( fbmp.Canvas.TextWidth(txt));
th := Round( fbmp.Canvas.TextHeight(txt));
while (tw > fbmp.Width) or (th > (fbmp.Height div 100) * 20)
do begin
fbmp.Canvas.Font.Size := fbmp.Canvas.Font.Size – 1;
tw := Round( fbmp.Canvas.TextWidth(txt));
th := Round( fbmp.Canvas.TextHeight(txt));
end;

srect.Top := srect.Height – (th + 20) ;
srect.Height := th;
fbmp.Canvas.Font.Style := fs;

fbmp.Canvas.BeginScene();
fbmp.Canvas.FillText(srect, txt, False, 1, [], TTextAlign.Center);
fbmp.Canvas.EndScene;

//bmp.Canvas.Unlock;
end;
end;
fbmp.Canvas.Unlock;
imgc.Assign(fbmp);
finally
fbmp.Free;
end;
finally
av_frame_free(fpict);
end;
end;

Above Code is responsible for refreshing the TImage.bitmap image. This procedure is being called from a tthread. while there is no user interaction, like execute opendialog or move trackbar, it seems like it is more than fast enough(windows).
Any suggestions ?

I was also thinking about give a try to create a Direct3D surface for windows to start and later Platform dependent HW video surface, but if I’m right, FMX does already make usage of HW rendering for (all ?) controls. This looks like the reason why TMediaPlayer is rendering on top ?

Is it worth to try create Direct3D surface, I will need to dive into a new not well known area, because I am afraid of getting the same effect as the TMediaplayer component. I need to be able to put controls above the video surface ? If it’s worth on Windows, is it worth in case of Android ?

Or should I just give up Delphi (FMX) for Multimedia(multiplatform) programming and switch for example to C/C++, because there are enough examples and (opensource) libraries. Another problem with Delphi (FMX) is that there are a lot of multimedia related questions unanswered, most answers include of needing of usage of external libraries which are made using C/C++.

Swithing to another language will be a big dive, but I am starting to think that Delphi is not suitable for multimedia programming. I need suggestions of Delphi (FMX) experts.

Comments are closed.