Thursday, 9 February 2017

Tracking with OpenCV and Python (Anton)


Because the timeframe for this proof of concept was pretty short I was tempted to use RoboRealm to get the quick and dirty results so I could focus on the part I really enjoy (C / C++ coding).

Sadly the RoboRealm is Windows and I'm not willing to setup my dev environment. On my Debian I have everything ready, customized keyboard layout, shortcuts and tools ready with all settings. Vanilla environment would be pretty unproductive for me. Even for internet browsing I miss my own customised Chromium browser. Running Windows and Linux at the same time is possible, but I wanted the machine learning process to be non-intrusive and not requiring a lot to get it running.

So I decided to give Python with OpenCV a shot. 

Training failure

Prepared my set of positives and negatives images. And I wanted to do haar training or cascade training. I tried half dozen examples and tutorials, but the tutorials required extra tools, which I had problem running. Tried to compile them in Jessie and Sid environment as well. I have feeling the tutorials were old and abandoned and required bit old tools, I had to downgrade few libraries just to solve few errors, just to be get different new errors. Gave up on the compilation process after trial / error, googling and 2 days of little to no progress. There was python version of the tool but either python3 nor python2 was able to run it. And contained syntax errors for both versions. Tried to fix few and made small steps to get stopped again by something else. Had good bit read on forums where some people described the tools as less buggy, but still buggy. That didn't gave me much confidence.

After 3 days wasted I was ready to give up and write fake serial sender and call it "a feature". My reasons is that for testing of the mbed and behaviours could be beneficial to have easy and consistent inputs and stimulus. And I don't want to be waving objects in front of the camera every time I want to test something.

Last attempt

But in last effort I gave it the very last attempt and found example which didn't require training nor any 3rd party tools and worked out of the box, 

Cloned it, made few modifications and published it here

Added features

  • Now it can send the tracked object over serial port
  • Normalises the center of the tracked rectangle into 0.0 to 1.0 range
  • Now works even when num-lock is on (this annoyed me more than I'm willing to acknowledge)

Using the tracker

Example usage:

python -l -d 0

 Will open first video device (a webcam probably)

python -l -v track.avi

 Will open video file to track, should work with AVI files, but works with MP4 and WEBM as well.

The controls are bit glitchy and annoying, but the tracking works good enough for something with no training. Problem is that it requires manual initialisation of the tracked area and often loses the track if it moves too fast, it needs to be close to previous position and if the object goes out of the frame it will mess up the tracking a lot. Often shutting it down and starting fresh was only solution.

Another problem is that probably this wasn't never designed to be fully automatic tracker (which we will need) and even doing small modifications took me longer than they should because I didn't understand what I was doing. This tracker doesn't recognize object, only tracks movement of the pixels from selected rectangle. In some videos where I select red ball after it goes out the frame it will not get recognized again after the red ball gets back into the frame. The OpenCV could be lovely if I had full whole year of free time to mess around and learn about all the innerworkings in detail. Without that knowledge I have feeling RoboRealm is better option even when it doesn't have as many features.


Here are the better examples (many object were completely failure to track, it hated screwdrivers with passion and refused to track them no matter what size or colour). First video is tracking from videos, second is from a webcam.

The above videos as external links:

No comments:

Post a Comment