ofVideoGrabber - slow framerate, needs direct to texture option (via OMX)

@arturoc

RPI has hardware acceleration for H264 videos.

There is a standalone video player called omxplayer on each RPI.

There is also som demo code here https://github.com/raspberrypi/firmware/tree/master/opt/vc/src/hello_pi/hello_video

For openFrameworks, this might best be implemented as a GST plugin / element, that would be auto-selected, by the current GST player, meaning we won't need to change any core oF code. This should happen once the element is in the core raspbian distro. More on progress here:

http://cgit.freedesktop.org/gstreamer/gst-omx/log/?h=raspberry
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=38&t=6852&sid=b679037b35691215df6e1c65ff95fa7b&start=100

It should support the paid codecs offered by RPI (i.e. VC1 and MPEG2 http://www.raspberrypi.com/)

From @arturoc: "openmax is actually a open standard from the kronos group, is also supported on latest versions of android so it would be good to have an ofOMXVideoPlayer."

btw, this will actually only return textures if i'm correct so anything that needs pixels will be really slow since you need to download the pixels from the graphics card, i've used ffmpeg on android in the past and the performance was ok for small videos, in a 2 year old phone so might be useful to try the gstreamer player as it is now, for getting pixels it might be faster than openmax

I thought we could pull the pixels from the hardware demuxer?

On Wed, Nov 21, 2012 at 3:01 AM, arturo notifications@github.com wrote:

btw, this will actually only return textures if i'm correct so anything
that needs pixels will be really slow since you need to download the pixels
from the graphics card, i've used ffmpeg on android in the past and the
performance was ok for small videos, in a 2 year old phone so might be
useful to try the gstreamer player as it is now, for getting pixels it
might be faster than openmax

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-10588469.

"I believe in science. Unlike mathematical theorems, scientific results
can't be proved. They can only be tested again and again, until only a fool
would not believe them.

I cannot prove that electrons exist, but I believe fervently in their
existence. And if you don't believe in them, I have a high voltage cattle
prod I'm willing to apply as an argument on their behalf. Electrons speak
for themselves."

-- Seth Lloyd: Quantum Mechanical Engineer, MIT

/.

yes you should be able to get the pixels but downloading things from the graphics card is usually super slow

that depends on a couple things. the broadcom chip has a separate
hardware demuxer for encoded video that is different from the GPU (or at
least that is what I understand, please correct me if I'm wrong)/ so we
should be able to pull the video frameBuffer. I will have to do some
research.

On Wed, Nov 21, 2012 at 3:52 PM, arturo notifications@github.com wrote:

yes you should be able to get the pixels but downloading things from the
graphics card is usually super slow

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-10613824.

"I believe in science. Unlike mathematical theorems, scientific results
can't be proved. They can only be tested again and again, until only a fool
would not believe them.

I cannot prove that electrons exist, but I believe fervently in their
existence. And if you don't believe in them, I have a high voltage cattle
prod I'm willing to apply as an argument on their behalf. Electrons speak
for themselves."

-- Seth Lloyd: Quantum Mechanical Engineer, MIT

/.

I started an attempt of converting hello_video and have all the components initializing. While doing some troubleshooting I ran across this which looks like it could be ported to OF - going to try this next

https://github.com/esrlabs/AndroidTransporterPlayer/blob/master/RPiPlayer.cpp

I got the hello_pi/hello_video sample functionality in an OF app - I am curious if it works for others as I am having issues trying to link ilclient lib through the project.make system

I haven't gotten to the point to where I manipulating pixels yet but you should be able to compile and play video

http://jvcref.com/files/PI/tests/jvc_video.1.zip

@arturoc

@danthemellowman @arturoc I think this may be a clue on how to get from the card (been trying to hijack the input without success, going to try this tomorrow) https://github.com/raspberrypi/firmware/blob/master/opt/vc/src/hello_pi/hello_encode/encode.c

in ubuntu omap4 (will only work with pandaboard and similar) there's a gstreamer plugin that decodes and encodes in hardware and returns pixels, i'm still having crashes with it but it works in totem... it's actually based on openmax so gst-omx should work too on the rpi

i have the videoplayer working under gstreamer with hw acceleration but it's kind of slow, i think it's related with the colorspace conversion + uploading to the graphics card again but it's weird since i've done this in software in android and the framerate was ok

@arturoc

@arturoc do you have your videoplayer published anywhere yet? curious to compare

it's the usual videoplayer in linux, the gstreamer one, you just need to use ofVideoPlayer, the videoPlayerExample should work. I fixed it for omap but it should work ok with the raspberry, all i did was adding a colorspace conversion that the pipeline wasn't doing automatically

I am currently getting this - I've stripped everything out but loadMovie, play, update and draw

[verbose] ofGstUtils: gstreamer inited
[verbose] loading file:///home/pi/openFrameworks/apps/devApps/videoPlayerExample/bin/data/fingers.mov
[verbose] attaching callbacks
[verbose] Gstreamer: speed change to 1.000000
[warning] received a preroll without allocation

here is the test code - anything I should try?

#include "testApp.h"

bool hasStarted = false;
//--------------------------------------------------------------
void testApp::setup(){
    ofSetLogLevel(OF_LOG_VERBOSE);
}

//--------------------------------------------------------------
void testApp::update(){
    int delay = 4000;
    unsigned long long currentTime = ofGetElapsedTimeMillis();
    if (currentTime > delay) 
    {
        if (!hasStarted) 
        {
            hasStarted = true;
            videoPlayer.loadMovie(ofToDataPath("fingers.mov", true));
            videoPlayer.play();
        }else {
            videoPlayer.update();
        }

    }else 
    {
        ofLogVerbose() << "waiting " << delay - currentTime << "ms";
    }
}

//--------------------------------------------------------------
void testApp::draw(){
    if (!hasStarted) {
        return;
    }
    videoPlayer.draw(0, 0, 320, 240);
}

it's weird that you are not getting any other messages but seems like
the pipeline is not able to get the size of the movie and when you get
the fist frame the buffers are not allocated yet.

you can try to deebug gstreamer to see if there's any more errors.
before executing the program, run:

export GST_DEBUG=:2

then run the example and you should get a long debug trace from the same
gstreamer, if you want to post it use also

export GST_DEBUG_NO_COLOR=1

so it doesn't generate weird characters for the color output

On 11/30/2012 09:19 AM, Jason Van Cleave wrote:

I am currently getting this - I've stripped everything out but
loadMovie, play, update and draw

|[verbose] ofGstUtils: gstreamer inited
[verbose] loading file:///home/pi/openFrameworks/apps/devApps/videoPlayerExample/bin/data/fingers.mov
[verbose] attaching callbacks
[verbose] Gstreamer: speed change to 1.000000
[warning] received a preroll without allocation
|

here is the test code - anything I should try?

|#include "testApp.h"

bool hasStarted = false;
//--------------------------------------------------------------
void testApp::setup(){
ofSetLogLevel(OF_LOG_VERBOSE);
}

//--------------------------------------------------------------
void testApp::update(){
int delay = 4000;
unsigned long long currentTime = ofGetElapsedTimeMillis();
if (currentTime > delay)
{
if (!hasStarted)
{
hasStarted = true;
videoPlayer.loadMovie(ofToDataPath("fingers.mov", true));
videoPlayer.play();
}else {
videoPlayer.update();
}
 }else
 {
     ofLogVerbose() << "waiting " << delay - currentTime << "ms";
 }
}

//--------------------------------------------------------------
void testApp::draw(){
if (!hasStarted) {
return;
}
videoPlayer.draw(0, 0, 320, 240);
}
|

—
Reply to this email directly or view it on GitHub
#21 (comment).

Here is the output from that (did you mean GST_DEBUG=2 ? I tried both...)

[verbose] loading file:///home/pi/openFrameworks/apps/devApps/videoPlayerExample/bin/data/fingers.mov
0:00:00.858212852 30166  0x17f4890 WARN                 qtdemux qtdemux_types.c:191:qtdemux_type_get: unknown QuickTime node type avc1
0:00:00.859994845 30166  0x17f4890 WARN                 qtdemux qtdemux_types.c:191:qtdemux_type_get: unknown QuickTime node type avcC
0:00:00.861679839 30166  0x17f4890 WARN                 qtdemux qtdemux.c:4613:qtdemux_parse_container:<qtdemux0> length too long (720896 > 15)
0:00:00.863300833 30166  0x17f4890 WARN                 qtdemux qtdemux_types.c:191:qtdemux_type_get: unknown QuickTime node type ?aut
0:00:00.865078827 30166  0x17f4890 WARN                 qtdemux qtdemux.c:4613:qtdemux_parse_container:<qtdemux0> length too long (720896 > 15)
0:00:00.866533821 30166  0x17f4890 WARN                 qtdemux qtdemux.c:4613:qtdemux_parse_container:<qtdemux0> length too long (720896 > 15)
0:00:00.868093816 30166  0x17f4890 WARN                 qtdemux qtdemux.c:8854:qtdemux_parse_tree:<qtdemux0> Can't handle datetimes before 1970 yet, please file a bug at http://bugzilla.gnome.org
[verbose] attaching callbacks
0:00:01.636221999 30166  0x17f4ac0 WARN                  ffmpeg gstffmpegdec.c:2299:gst_ffmpegdec_frame:<ffdec_h2640> ffdec_h264: decoding error (len: -1, have_data: 0)
[verbose] Gstreamer: speed change to 1.000000
[warning] received a preroll without allocation

is difficult to say what's going on but it seems like it couldn't find the size of the video, i would try with some other codec, or even with a different h264 video, i was also receiving some errors from the qt demuxer in the pandaboard but it was working for me. i'll check in the rpi as soon as i get mine ;)

it was super buggy it only worked some times

interesting - if you still have the version you got to work I would be interested in seeing it - mostly to see what I was doing different.

this guy had some numbers around using shaders using Qt
http://www.raspberrypi.org/phpBB3/viewtopic.php?p=192425#p192425

I followed the thread above, found and was able to install gst-plugins-gl:
http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/log/

It says ES2 support so maybe it is useful? I couldn't get it to display anything (but I couldn't get gst-omx to either)

if you decide to clone it and try it you will have to do this fix in the master branch
http://bugzilla-attachments.gnome.org/attachment.cgi?id=216088

you need libv4l-0 and v4l-utils for the pi

@jvcleave

ping? what's the state of the art on this @jvcleave @danthemellowman @arturoc ?

@arturoc

I think it all leads back to what @arturoc was talking about - have a specialized OMX Player

Most of the stuff I researched around gst-omx (and what I think arturo experienced) is that it didn't work well yet or work on raspbian.

This seems the most promising lead I have found but it is currently using Wheezy/Qt based and pretty heavy
http://thebugfreeblog.blogspot.it/2012/12/decoding-and-rendering-to-texture-h264.html

@arturoc

have you guys looked at https://github.com/huceke/omxplayer it is the test
code the XBMC guys wrote for the pi.

On Thu, Jan 3, 2013 at 10:49 PM, Jason Van Cleave
notifications@github.comwrote:

I think it all leads back to what @arturoc https://github.com/arturocwas talking about - have a specialized OMX Player

Most of the stuff I researched around gst-omx (and what I think arturo
experienced) is that it didn't work well yet or work on raspbian.

This seems the most promising lead I have found but it is currently using
Wheezy/Qt based and pretty heavy

http://thebugfreeblog.blogspot.it/2012/12/decoding-and-rendering-to-texture-h264.html

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-11874016.

"I believe in science. Unlike mathematical theorems, scientific results
can't be proved. They can only be tested again and again, until only a fool
would not believe them.

I cannot prove that electrons exist, but I believe fervently in their
existence. And if you don't believe in them, I have a high voltage cattle
prod I'm willing to apply as an argument on their behalf. Electrons speak
for themselves."

-- Seth Lloyd: Quantum Mechanical Engineer, MIT

/.

@obviousjim

i think we should just wait for gstreamer to get fixed, it can take some months but seems like the most universal solution. there's ongoing work for accelerated video playing with gst on desktop, omap and rpi. also no matter which solution we implement direct to texture needs some kind of hardware colorspace conversion (gst also will implement that at some point, there's something like this in the desktop through vaapi) or it'll be useless. implementing it ourselves will mean adding support for lots of pixel formats, like every yuv in ofPixels, what we can do at some point +using shaders for colorspace conversion from the videoplayer/grabber what can be done but would need some radical architectural changes in the core

at some point i talked with @obviousjim about having a usePixels methods in videoPlayers and grabbers so you can disable pixels to get only textures if the video can be hardware accelerated but i think we should do this later for all platforms after merging this into the core

by now i want to try a faster colorspace conversion in gstreamer which is already there, not hardware but still faster than the super slow ffmpegcolorspace element, if that works i think that's enough by now but if someone wants to try to implement omx meanwhile we can have it as an addon

As for going the OMX addon route - I've been digging through the code from the last link I sent. It isn't as heavy as I previously thought as much of the code is based off the hello_video example that I had mostly working in OF anyway.

Most of the magic seems to center around this function where you pass in a EGLImage for the video to write to

OMX_UseEGLImage
http://maemo.org/api_refs/5.0/beta/libomxil-bellagio/group__buf.html#ga9d093a08220dd91e4a95f7adbaa30dc

i've just updated both develop in the main repo and develop-raspberrypi here to be compatible with gstreamer-1.0 which incliudes faster colorspace conversion. it's still not present in raspbian but as soon as it is it should work, i still need to change the install scripts to detect if 1.0 can be installed

@arturoc

@arturoc @kalwalt do you think this issue applies to the Pandaboard? I'm currently classifying some of these issues to get a better sense of what should be prioritized.

yes there's also hw accelerated decoding in the pandaboard, and you can use it through gstreamer but there's no hw accelerated colorspace conversion yet, but as i said before i would just wait till it fixes itself :) i mean at some point gstreamer gl elements will work on gles properly and we'll get hw colorspace conversion. for the rpi we could have the omx video player as an addon, since doing it in the core will need lots of refactoring and it won't work in any other platform

@bakercp

@bakercp @arturoc @jvcleave can we close this?

I think we can close it as it is a broad issue and works as well as it can without gst-omx (ofVideoPlayer-wise). ofVideoGrabber is going to be limited by the USB processing. The only OMX solutions for camera are for specific chipsets that have embedded cameras (like an Android phone's camera)

Navigation Menu

Repository navigation

ofVideoGrabber - slow framerate, needs direct to texture option (via OMX) #21

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ofVideoGrabber - slow framerate, needs direct to texture option (via OMX) #21

Description

Activity

arturoc commented on Nov 21, 2012

danzeeeman commented on Nov 21, 2012

arturoc commented on Nov 21, 2012

danzeeeman commented on Nov 21, 2012

jvcleave commented on Nov 22, 2012

jvcleave commented on Nov 23, 2012

jvcleave commented on Nov 24, 2012

arturoc commented on Nov 24, 2012

arturoc commented on Nov 24, 2012

jvcleave commented on Nov 30, 2012

arturoc commented on Nov 30, 2012

jvcleave commented on Nov 30, 2012

arturoc commented on Nov 30, 2012

jvcleave commented on Nov 30, 2012

arturoc commented on Nov 30, 2012

17 remaining items

arturoc commented on Dec 14, 2012

jvcleave commented on Dec 14, 2012

jvcleave commented on Dec 14, 2012

danzeeeman commented on Dec 29, 2012

bakercp commented on Jan 3, 2013

jvcleave commented on Jan 4, 2013

danzeeeman commented on Jan 4, 2013

arturoc commented on Jan 4, 2013

jvcleave commented on Jan 4, 2013

arturoc commented on Jan 5, 2013

bakercp commented on Jan 9, 2013

arturoc commented on Jan 10, 2013

danzeeeman commented on Apr 6, 2013

jvcleave commented on Apr 7, 2013

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions