extracting frames from movies on iPhone
The iPhone SDK provides a high level movie player, but so far there is no way to decode movies or extract frames. In this post I will show you how to do that using FFmpeg.
First, thanks to the FFmpeg project, Martin Böhme for his tutorial on using libavformat and libavcodec and David Conrad for the gas-preprocessor project that allows us to compile FFmpeg for the iPhone.
Before you link the FFmpeg libraries to your project, make sure that you understand FFmpeg’s licensing. My project uses the LGPL version of FFmpeg, which means that I have to make FFmpeg’s source code available (included in the project).
Before you create a commercial project that decodes H264 video, make sure that you get a license from MPEG LA.
The source code is posted on github: git://github.com/lajos/iFrameExtractor.git.
To compile and run the project on your iPhone:
- open Terminal
- clone the repository: git clone git://github.com/lajos/iFrameExtractor.git
- go to the ffmpeg folder in the project: cd iFrameExtractor/ffmpeg
- build the ffmpeg libraries: ./build_universal
- open the xcode project and run it on your iPhone device
The build_universal shell script creates libraries that work on both armv6 and armv7 processors. If you need libraries for pre-3Gs devices (iPhone 2g, iPhone3G, iPod Touch), use the build_armv6 script. For iPhone 3Gs only libraries, use build_armv7.
iFrameExtractorAppDelegate creates an instance of VideoFrameExtractor (the class that grabs images from the video). When you press the play button, iFrameExtractorAppDelegate sets up a loop and displays all the images from the video until it reaches the end of the movie.
The VideoFrameExtractor class is based on Martin Böhme’s tutorial, upgraded to use swscale. Initialize this class with the full path of the movie:
VideoFrameExtractor *video = [[VideoFrameExtractor alloc]
initWithVideo:@"/full/path/to/movie.mov"];
By default, the output size will match the movie’s size. You can set outputWidth and outputHeight to change the dimensions:
video.outputWidth = 426; video.outputHeight = 320;
Use stepFrame to grab the next frame. This method returns false if there are no more frames (reached the end of video):
BOOL result = [video stepFrame];
The processed frame can be accessed as a UIImage on the currentImage property:
UIImage *myImage = video.currentImage;
You can seek to a specific time (in seconds) using the seekTime: method:
[video seekTime:1.0];
The movie’s length can be queried by checking the duration property:
NSLog(@"video duration: %f",video.duration);
This should get you started with extracting frames from iPhone movies 
If you find any problems, please submit a patch.
Known issues:
- Images are always decoded as landscape.
Hi,
first, this is a very nice example how to use the FFMPEG together with Cocoa, very nice and clean design, thank you for the source code.
But do you have also a Cocoa example for the opposite way? For example to encode a bunch of UIImages to a video with the FFMPEG?
Regards,
GM
Hi,
another question, is it possible to use the iFrameExtractor with the iPhone Simulator? As far as I can see, the FFmpeg builds only for armv6 and armv7, is a version for the iPhone Simulator possible as well?
Regards,
GM
@George: I’m working on a project for video encoding. You can compile ffmpeg for i386 to run on the simulator, I’ll probably take a look at that when the encoding project is done.
Thanks for example, i want to ask if you have a sample code or tutorial for using ffmpeg with stream audio, and if its possible using libmms (for mms;//).
Regards,
YD
Hi,
Can u help me in converting array of frames into video in iPhone using FFmpeg library. I’m greatly suffering from this and its urgent. Any link of code snippet will be very helpful.
Please its urgent
Thanks in Advance.
Thanks for this tutorial - I think this ffmpeg stuff is so interesting. I also would love to see your encoding project!
I too would like to see the encoding project.
Also, how would i make it so that the user can open the camera role a choose the video that they would extract the frame from?
Hi,
Thanks for this great tutorial.
Is there any update to build for iPhone simulator? Thanks
Thanks for this tutorial,
Unfortunately, i have the following error during compilation…
I have exactly the same compilation error when using instructions from gas-preprocessor site…
Any idea regarding this error (gcc-4.2: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags)???
Creating config.mak and config.h…
/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc -DHAVE_AV_CONFIG_H -I. -I”/Users/fvisticot/Documents/Dev_iPhone/iFrameExtractor/ffmpeg” –sysroot=/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.0.sdk -D_ISOC99_SOURCE -D_POSIX_C_SOURCE=200112 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DPIC -O2 -arch armv6 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.1.2.sdk -arch armv7 -mcpu=cortex-a8 -std=c99 -fomit-frame-pointer -fPIC -marm -g -Wdeclaration-after-statement -Wall -Wno-switch -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wno-pointer-sign -Wcast-qual -Wwrite-strings -Wundef -O3 -fno-math-errno -fno-tree-vectorize -MMD -MF libavdevice/alldevices.d -MT libavdevice/alldevices.o -c -o libavdevice/alldevices.o libavdevice/alldevices.c
gcc-4.2: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags
@fvisticot: try compiling without -E, -S, -save-temps and -M?
Thanks for this helpfull remark !!!
Those parameters are not part of the main makefile. I don’t know how to disable those options.
I do not understand why i need to do something specific to compile…
Hi
I wonder if Apple would accept an app using this library. Can anyone confirm this?
@Cyttorak: pretty much any app that encodes or decodes video uses ffmpeg, so it doesn’t seem like apple would have any problems with it. However you definitely want to get a license from MPEG LA.
Hi lajos,
Thanks for the code, it’s really nice.
I am trying to compile the code for 2 weeks to use on iPhone simulator but not success yet.
Pls give some suggestion for compiling.
thanks lajos.
i run it on my iphone 3gs success.
but i have a quesion,how to convert mov to mp4,use ffmpeg.
Thanks a lot, this was just what I needed.
I finally got it to work on the simulator (after about 4 hours of messing around) using this configure line:
./configure –enable-cross-compile –arch=i386 –cc=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.2.1 –sysroot=/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.0.sdk –extra-cflags=”-mdynamic-no-pic” –target-os=darwin –disable-neon –disable-pic –enable-postproc –disable-debug –disable-stripping –enable-avfilter –enable-avfilter-lavf –enable-gpl –disable-bzlib
and of course modifying the appropriate lines in the build_universal script:
lipo -create -arch armv7 armv7/libavcodec.a -arch i386 i386/libavcodec.a -output universal/libavcodec.a
etc.
Hope this helps,
Shahar
Hi,
When I running the build-universal script I getting the error: “Unrecognized input filetype at /bin/sh line 23″ I see that its from the gas-preprocessor.pl script.
Pls help me.
Thanks,
Eyal.
@Eyal: Make sure that you are running the build script in terminal from the ffmpeg folder.
[...] http://www.codza.com/extracting-frames-from-movies-on-iphone [...]
Hi,Thanks for example,it’s wonderful.
But there is no sound when play movie.
How to slove it ?
@eidolon: this project only supports image extraction. You can extract the audio stream using ffmpeg and send the audio data to the iphone hardware, but I have example code for that.
Hi,iFrameExtractor doesn’t work on iOS4(3.1.3 upgrade)…
@sukiyaki: Thanks for the note. I think this is because 3.1.2 gcc might not be part of sdk 4.0. I’ll take a look.
@lajos: i was able to compile i386 and universal but when i run the project on xcode i’m getting this error:
“_BZ2_bzDecompressInit”, referenced from: _matroska_decode_buffer in libavformat.a(matroskadec.o)
“_BZ2_bzDecompress”, referenced from:
_matroska_decode_buffer in libavformat.a(matroskadec.o)
“_BZ2_bzDecompressEnd”, referenced from: _matroska_decode_buffer in libavformat.a(matroskadec.o)
ld: symbol(s) not found
any ideas how to fix this?
thanks in advance.
@lajos: i got it working… i edited line 225 of build_i386 from:
./configure –disable-mmx –disable-doc –disable-ffmpeg –disable-ffplay –disable-ffserver –cc=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.2.1 –as=’gas-preprocessor/gas-preprocessor.pl /Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.2.1′ –sysroot=/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.0.sdk –extra-cflags=’-fmessage-length=0 -pipe -Wno-trigraphs -fpascal-strings -O0 -fasm-blocks -Wreturn-type -Wunused-variable -D__IPHONE_OS_VERSION_MIN_REQUIRED=30000 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.0.sdk -fvisibility=hidden -mmacosx-version-min=10.5 -gdwarf-2′
to
./configure –disable-bzlib –disable-mmx –disable-doc –disable-ffmpeg –disable-ffplay –disable-ffserver –cc=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.2.1 –as=’gas-preprocessor/gas-preprocessor.pl /Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.2.1′ –sysroot=/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.0.sdk –extra-cflags=’-fmessage-length=0 -pipe -Wno-trigraphs -fpascal-strings -O0 -fasm-blocks -Wreturn-type -Wunused-variable -D__IPHONE_OS_VERSION_MIN_REQUIRED=30000 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.0.sdk -fvisibility=hidden -mmacosx-version-min=10.5 -gdwarf-2′
basically just added the –disable-bzlib at the beginning of the config line to disable the bzlib… recompiled it again and eurika!!
i guess you can do the same for the universal builds.
hope this helps.
@junjet: thanks! I’ll make that fix in the repo.