Motion Detection with Multiple RTSP Cameras

Well, here we go again.  One of our cameras went on the fritz, so I started looking.  There were two decent cameras that had the same resolution and the same outdoor ratings (e.g. windproof, water resistance, and dust resistance), so I picked them up.  Unfortunately, they do not expose an jpeg stream, so I had to use a different Linux motion tool, available from :

Now for the REAL down-side.  I compiled it, and loaded it up with two camera streams.  It worked great, for a day or two, and then the process would vaporize. I found an obscure message in the log file that consisted of :

May  7 00:36:00 ch abrt[29014]: Saved core dump of pid 26188 (/usr/local/bin/motion) to /var/spool/abrt/ccpp-2016-05-07-00:35:55-26188 (548212736 bytes)

So, I loaded the application up, then attached to it via the debugger (gdb motion 123456) and got the following stack trace after a few days :

[New Thread 0x7fffecdfa700 (LWP 23598)]
[New Thread 0x7fffd3fff700 (LWP 23599)]
[New Thread 0x7fffd21fc700 (LWP 23600)]
Detaching after fork from child process 23603.
Detaching after fork from child process 23604.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff72a8700 (LWP 23594)]
0x00000030b5032625 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000030b5032625 in raise () from /lib64/libc.so.6
#1  0x00000030b5033e05 in abort () from /lib64/libc.so.6
#2  0x00000030b5070537 in __libc_message () from /lib64/libc.so.6
#3  0x00000030b5075e66 in malloc_printerr () from /lib64/libc.so.6
#4  0x00000030b50789b3 in _int_free () from /lib64/libc.so.6
#5  0x0000000000416b20 in netcam_cleanup (netcam=0x7fffa0000030, 
    init_retry_flag=0) at netcam.c:2581
#6  0x0000000000412d28 in vid_close (cnt=0x678a10) at video_common.c:555
#7  0x0000000000406f19 in motion_cleanup (cnt=0x678a10) at motion.c:991
#8  0x00000000004097b8 in motion_loop (arg=)
    at motion.c:2231
#9  0x00000030b58079d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000030b50e88fd in clone () from /lib64/libc.so.6
(gdb)

So, I then checked in the log, found the above stack trace was not related, and that that run may have had other issues going on.  My real stack trace is :

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/local/bin/motion...done.

warning: core file may not match specified executable file.
[New Thread 4667]
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `/usr/bin/keyutil --command genreq --ascii --renew /etc/pki/tls/certs/cottonwood'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000030b5047e2c in ?? ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
(gdb) bt
#0  0x00000030b5047e2c in ?? ()
#1  0x0000000000000023 in ?? ()
#2  0x00007fff1c7768c0 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb)

Bah!  Next, in the logs, I found :

/var/log/messages:May 22 20:29:00 ch motion: [1] [ERR] [EVT] event_image_snapshot: Could not create symbolic link [01-20160522202900-snapshot.jpg]: No such file or directory
/var/log/messages:May 22 20:29:30 ch motion: [1] [ERR] [ALL] create_path: Problem creating directory /home/media/security: Input/output error
/var/log/messages:May 22 20:29:30 ch motion: [1] [ERR] [ALL] put_picture: Can't write picture to file /home/media/security/cameras/camera-backyard//01-20160522202930-snapshot.jpg: Input/output error

I completely missed the "Input/output error" and checked the filesystem location, only to find my entire disk had failed, so it could not write videos.  Sometimes, debugging can be such a pain!  Here comes a nice recovery fee and a nice fat check going out for a NAS with mirroring!

Status of the Home Brew Security System

I decided to upgrade from an Arduino and a large media server to an Udoo quad core (http://shop.udoo.org/usa/product/udoo-quad.html), because I didn't want to have to have the event server on the same network as everyone else.  I'm sort of paranoid.  What I found is :

• The Udoo uses GPIO inputs that ARE available through the Linux image.
• The Udoo failed to work with my relays.  This was because the relays were 5v relays (perfect for the Arduino Mega2560 I was using), but the Udoo uses a 3v logic level.  Simply put, the relays would never work.
• I could put my entire event server package onto the Udoo, and have a self-contained alert system.  This was my goal.

Here's what happened.

I found a great tutorial from Adafruit at http://www.udoo.org/ProjectsAndTutorials/linux-gpio-manipulation/.  I slapped together some listener programs to watch the GPIO states, and I was suddenly in business again.  I no longer needed to upload any code to the Atmel chip on board - I could do it all in native C code, standard-Linux stuff.  Those tools were checked into the event server repository (subversion), and have been thoroughly tested.

Next, I needed to be able to work the garage door.  I toyed with building a separate, stand-alone device, but I'd already done that, and I was starting to get lazy.  I found a device on ebay, http://www.ebay.com/itm/111248757109?ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1497.l2649, that was $50, and had two relays and Ethernet.  The downside is this came hard coded to an IP address of 192.168.1.199.  I had to use a crossover cable to connect it up, and reconfigure it to use DHCP.  It has a simple web-based interface.  So, being the nerd I am, I watched network traffic to see what requests were being sent.  When I'd click one of the relay buttons on the web page, it would send a request :

http://192.168.1.199/relays.cgi?relay=1

When I turned it back off, it sent an identical request.  There was no "off" or "on" designation in the requests.  Apparently, the device simply took "toggle" commands".

I did find if I left off the ?relay=1, it simply gave me the following page:

<div>
<font size="5"> one                             <span style="color:red">0 </span> </font>
<br />
<br />
<br />
<font size="5"> two                             <span style="color:red">0 </span> </font>
</div>

<div>
<p hidden>
Status: 0 0 
Relay1: one                          
Relay2: two                          

Now I was getting somewhere.  I toggled it, ran the previous, and found that the "Status: 0 0" line went to "Status: 1 0".  It's a two relay board, so now I know how to get the current state.  Next, I needed to identify what had to be done to ensure it was always turning the switch off when it was done acting like a momentary push-button :

1. Ensure the switch was in the off position.
2. Toggle the switch to the "on" position.
3. Wait for 1 second.
4. Toggle the switch back to the "off" position.

So, I used the request without the "?relay=1" to get the "Status: " line for the relay I needed.  If it had a 1 for that one, I had to turn it off first.  In my case, it should ALWAYS be in the off position already.  Next, I had to toggle it to "on" position, and a simple request to the following would work perfectly :

http://192.168.1.199/relays.cgi?relay=1

Then, I slept for a second, and then made that toggle request a second time to turn it off.  I hooked it up, and ran it.  Woohooo!  I'm back online again!  Here's the code I used :

#!/usr/bin/perl

use LWP;

use HTTP::Request;

my $relay_host = '192.168.1.199';

my $relay_index = 0;

my $ua = LWP::UserAgent->new('alarmsystem');

$ua->credentials($relay_host.':80',"Protected", 'admin', 'admin');

# FIRST - Ensure this thing is OFF

my $res = $ua->get('http://'.$relay_host.'/relays.cgi');

print $res->content;

exit;

if ($res->content =~ /status: ([\d\s]+)/i) {

  my (@rows) = split(/[\s\t]+/,$1);

  if ($rows[$relay_index] == 1) {

    my $res = $ua->get('http://'.$relay_host.'/relays.cgi?relay='.($relay_index+1));

  }

};

# SECOND - Turn it ON

my $res = $ua->get('http://'.$relay_host.'/relays.cgi?relay='.($relay_index+1));

# sleep for a second

sleep(1);

# LAST - Turn it back to OFF

my $res = $ua->get('http://'.$relay_host.'/relays.cgi?relay='.($relay_index+1));

Introducing Project Homebrew Security System

The Arduino home security system has led into a new project.  A year ago, I had acquired some outdoor cameras from Harbor Freight for relatively cheap costs ($40 for a color camera).  At the time, I had a PVR device (home built DVR system to record on two channels), and enabled the "motion" application on Linux to work with them.  For a simple and cheap "security" system, it sufficed, but it was isolated to a single room, and it wasn't very "secure", and was fairly limited (the requirements list, points one and two).

After purchasing a home, I realized that I needed something a little better.  I then purchased a 4 channel recorder from a maker.  That device filled the need (almost).  The device kept freezing every few months, without so much of a notification (point 3).  It did show when someone broke into the car, but didn't provide enough clarity (point 4 in the requirements).  The device primarily was for a video stream, and the search mechanism was pretty poor when looking for video files (leading to requirement 5).  Though the device handled input for "GPIO's" (general input pins), there were no instructions on how to use them.

I picked up an Arduino (my first Arduino project), and thought about using it as a "sensor" for doors being opened.  I realized that there were some added requirements.  Those requirements culminated into the following list :

The things learned so far (e.g. the requirements of the system) :

1. An external alert mechanism
2. Handle more than two video streams
3. Stability
4. Clarity on video streams, both night and day
5. Good interface
6. Able to handle external notifications (e.g. a door, gate, motion, heat, or seismic sensor)
7. Logs for events, from cameras to sensors
8. Able to send notifications to an off-site location
9. Nothing "wireless" - wireless is easily spoofed and jammed, and is easily eavesdropped on
10. Expandable (e.g. connecting in to phone systems via Asterisk)
11. System monitoring

The above list of requirements gave me solid ground to start working from.  First, since I wanted a system that was stable, I needed a piece of hardware that would not have to worry about doing the device capturing - so I knew I needed IP cameras (or "network cameras").  Because I didn't want to run power lines for each camera, that meant these IP cameras needed to have PoE (power-over-ethernet), and I wanted an effective 40 feet range at night (to cover automobiles, fences, etc).  I finally did find the cameras I wanted : TRENDnet TV-IP322P cameras.  I already have lines run for the smaller cameras, so I will use those to "fish" the CAT cables through the walls.

I was able to design a cheap piece of hardware to cover point 6 in the above.  That was the Interrupt-Driven Arduino Alarm.  I've since expanded that project into the above requirements list, and will document the design and implementation of the system through a few different blog posts.  Additionally, due to time constraints and a bug found in that interrupt-driven code for the Arduino, it is no longer interrupt-driven, but a polling mechanism.  Reasons will be identified in the design post.  However, the parts list so far :

1. TRENDnet TV-IP322P cameras, IP-based, PoE, night vision to 20m, good resolution (1280x1024), cost : $420 each
2. TRENDnet TPE-S44 8-port, 100Mbps, PoE switch [enough ports for all cameras, arduino, LAN link, and local device to handle video capture] ($50.00)
3. Arduino Mega 2560 ($48.00)
4. Arduino Ethernet Shield W5100 ($40.00)
5. Arduino Ethernet Shield PoE module Ag9120-S ($21.00 from Amazon) (I moved the Arduino, and found I didn't need to use the switch for the Arduino, so save yourself $20 and buy yourself a new movie and put the Arduino in place with a power supply).
6. SD card to hold the Arduino configuration ($4.00) (I ended up adding code to the Arduino that, if the SD card fails, will fall back to a coded configuration - and you can actually accomplish all of this without the SD card).
7. Case for Arduino ($15-30, depending on what you want)
8. Linux Server (this can range - I had one that had been decomissioned a while back)
9. Ethernet cable, connectors, and jacks

We'll document the project in the following order :

1. Arduino - Design and Code - should act as the interrupt-driven sensor interface with PoE so that it will report in when events happen in real-time
2. Arduino Assembly and Testing - putting the active sensor together
3. !!Bugs!! - found some gotchas, might help you to see these.
4. Installing the Arduino and doing preliminary testing of at least two doors. This is pretty self explanatory - find a spot and a box, install the arduino to this, and run cables with the door sensors - examples of the installation can be found here).
5. Configure the TrendNet TV-IP322P camera in a motion detection package. After failing with first ZoneMinder and then Linux Motion due to unacceptable frame rate recordings, I found I could use a  flood light connected to the alarm server documented in Arduino Assemly and Testing, but didn't want flood lights by every camera.  After testing and retesting, I found that the highest quality of uploaded video, the fastest to get set up, the easiest to tweak and reduce false positives was the camera's own motion detection and upload functionality.  Incidentally, it turns out that when the arduino was down, this still provided a layer of security as I could configure the CGI events to still alert me.
6. Changing from a prototyping perf board to an actual PCB for a cleaner installation. The board looked great, connections and components fit (the relays were a little snug, but not too bad).  I did run into some issues with the early board (I deleted the old board and uploaded a new one to fix those problems), but things are looking up.  If you need a board professionally done and an initial small count for testing, BatchPCB is the way to go.  Testing the board was successful.
7. Setting up a portal that includes current camera views, event logs (from sensors and camera recordings), and the ability to view live camera feeds.
8. Full testing and results

Questions and comments are welcome.