Technical Blog

Use OpenCV with the AR.Drone SDK

OpenCV (Open Source Computer Vision Library) is a powerful image processing library. I will detail in this post how to use it with the AR.Drone’s C SDK.

Compiling the AR.Drone SDK with OpenCV

The first step is to install OpenCV. If you’re using Ubuntu, you may refer to this page.

Once the library is installed, we need to modify the Makefile to add the correct flags. We will edit sdk_demo/Build/Makefile.

To add the correct cflags, find the line:

GENERIC_INCLUDES:=$(addprefix -I,$(GENERIC_INCLUDES))

and add underneath:

GENERIC_INCLUDES += `pkg-config --cflags opencv` 

To add the correct libraries, change the following line:

GENERIC_LIBS=-lpc_ardrone -lgtk-x11-2.0 -lrt

to

GENERIC_LIBS=-lpc_ardrone -lgtk-x11-2.0 -lrt `pkg-config --libs opencv`

Creating an OpenCV image from the drone’s image

We need to update the output_gtk_stage_transform in Video/video_stage.c to transform the data received from the drone to an IplImage, the OpenCV image structure. First, we need to add some includes:

#include "cv.h"
#include "highgui.h" // if you want to display images with OpenCV functions

We will use a method close to what we did to create a GdkPixbuf:

IplImage *ipl_image_from_data(uint8_t* data)
{
  IplImage *currframe;
  IplImage *dst;

  currframe = cvCreateImage(cvSize(320,240), IPL_DEPTH_8U, 3);
  dst = cvCreateImage(cvSize(320,240), IPL_DEPTH_8U, 3);

  currframe->imageData = data;
  cvCvtColor(currframe, dst, CV_BGR2RGB);
  cvReleaseImage(&currframe);
  return dst;
}

We call it from output_gtk_stage_transform in Video/video_stage.c:

IplImage *img = ipl_image_from_data((uint8_t*)in->buffers[0], 1);

Vertical camera handling

As detailed in a previous article, the images captured with the vertical camera has a lower size than the horizontal camera. The data transmitted has the same size in both cases, but with empty pixels. I updated the ipl_image_from_data:

IplImage *ipl_image_from_data(uint8_t* data, int reduced_image)
{
  IplImage *currframe;
  IplImage *dst;

  if (!reduced_image)
  {
    currframe = cvCreateImage(cvSize(320,240), IPL_DEPTH_8U, 3);
    dst = cvCreateImage(cvSize(320,240), IPL_DEPTH_8U, 3);
  }
  else
  {
    currframe = cvCreateImage(cvSize(176, 144), IPL_DEPTH_8U, 3);
    dst = cvCreateImage(cvSize(176,144), IPL_DEPTH_8U, 3);
    currframe->widthStep = 320*3;
  }

  currframe->imageData = data;
  cvCvtColor(currframe, dst, CV_BGR2RGB);
  cvReleaseImage(&currframe);
  return dst;
}

The trick is the same as detailed in the previous article. We set that each new line starts every 320*3 bytes, but we only use 176*3 byes per line.

Converting OpenCV images to GdkPixbuf

If you’re using a GTK interface as detailed in previous articles, you may want to display the OpenCV image inside your GTK Window. To do this, I use the following function to create a GdkPixbuf structure that can be displayed by GTK:

GdkPixbuf* pixbuf_from_opencv(IplImage *img, int resize)
{
  IplImage* converted = cvCreateImage(cvSize(img->width, img->height), IPL_DEPTH_8U, 3);
  cvCvtColor(img, converted, CV_BGR2RGB);

  GdkPixbuf* res = gdk_pixbuf_new_from_data(converted->imageData,
                                           GDK_COLORSPACE_RGB,
                                           FALSE,
                                           8,
                                           converted->width,
                                           converted->height,
                                           converted->widthStep,
                                           NULL,
                                           NULL);
  if (resize)
    res = gdk_pixbuf_scale_simple(res, 320, 240, GDK_INTERP_BILINEAR);

  return res;
}
  • http://gauth.fr/2011/09/create-a-video-from-the-ar-drone/ Create a video from the AR.Drone : : Gauth

    [...] The AR.Drone is an efficient source of images: it can fly, being remotely controlled, etc. We will see in this article how to create avi files from its cameras. We will use OpenCV to create it, so you’ll probably need first to take a look to Use OpenCV with the AR.Drone SDK. [...]

  • Guest

    Thanks for the tutorial. In our project, the commande “currframe->widthStep = 320*3;” for the vertical camera added pink strips and resize the camera image. We had to change it to “currframe->widthStep = 320*3*2;”
    Cheers

  • Anonymous

    I think you’re using a newer version of the Ar.Drone, which has a better resolution. It would explain that ;-)

  • Holymoo

    I had an issue using ‘pkg-config –cflags opencv’ in the makefile; however, I was able to resolve it by simply putting in the directory location.

    I’d like to make whatever changes necessary to get it to work with original command to make the codebase a little more portable. Any suggestions.

    I believe OpenCV is indeed installed correctly, since I can compile the sample executables.

  • http://overblogged.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/ Parrot ARDrone 2.0 Video Streaming through OpenCV in Linux | Physical Sciences & Programming

    [...] big help for tackling this problem comes from this site, which is tailored for ARDrone 1.x series: http://gauth.fr/2011/09/use-opencv-with-the-ar-drone-sdk. Moving [...]

  • http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/ Kout Petr

    I also got this to work for ARDrone 2.0, which uses a very different video encoding and TCP instead of UDP. See the steps here:

    http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/

  • http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/ Guest2

    Hi. How do you actually use the function pix_from_opencv() to display it using the GTK interface? I tried calling:

    vp_os_memcpy(cfg->frameBuffer, pixbuf_from_opencv(openCVImg, 0), cfg->fbSize);

    in my code to fill the buffer with the content of what is returned by your function, but that doesn’t work. Nothing is displayed. Sorry if this is obvious. I am new to this topic. Thank you!

  • Jose Mendez

    HI Holymoo

    Iam having the same problem, can you explain me how to proceed

    thanks