Motion Detection with Multiple RTSP Cameras

Well, here we go again.  One of our cameras went on the fritz, so I started looking.  There were two decent cameras that had the same resolution and the same outdoor ratings (e.g. windproof, water resistance, and dust resistance), so I picked them up.  Unfortunately, they do not expose an jpeg stream, so I had to use a different Linux motion tool, available from :

Now for the REAL down-side.  I compiled it, and loaded it up with two camera streams.  It worked great, for a day or two, and then the process would vaporize. I found an obscure message in the log file that consisted of :

May  7 00:36:00 ch abrt[29014]: Saved core dump of pid 26188 (/usr/local/bin/motion) to /var/spool/abrt/ccpp-2016-05-07-00:35:55-26188 (548212736 bytes)

So, I loaded the application up, then attached to it via the debugger (gdb motion 123456) and got the following stack trace after a few days :

[New Thread 0x7fffecdfa700 (LWP 23598)]
[New Thread 0x7fffd3fff700 (LWP 23599)]
[New Thread 0x7fffd21fc700 (LWP 23600)]
Detaching after fork from child process 23603.
Detaching after fork from child process 23604.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff72a8700 (LWP 23594)]
0x00000030b5032625 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000030b5032625 in raise () from /lib64/libc.so.6
#1  0x00000030b5033e05 in abort () from /lib64/libc.so.6
#2  0x00000030b5070537 in __libc_message () from /lib64/libc.so.6
#3  0x00000030b5075e66 in malloc_printerr () from /lib64/libc.so.6
#4  0x00000030b50789b3 in _int_free () from /lib64/libc.so.6
#5  0x0000000000416b20 in netcam_cleanup (netcam=0x7fffa0000030, 
    init_retry_flag=0) at netcam.c:2581
#6  0x0000000000412d28 in vid_close (cnt=0x678a10) at video_common.c:555
#7  0x0000000000406f19 in motion_cleanup (cnt=0x678a10) at motion.c:991
#8  0x00000000004097b8 in motion_loop (arg=)
    at motion.c:2231
#9  0x00000030b58079d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000030b50e88fd in clone () from /lib64/libc.so.6
(gdb)

So, I then checked in the log, found the above stack trace was not related, and that that run may have had other issues going on.  My real stack trace is :

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/local/bin/motion...done.

warning: core file may not match specified executable file.
[New Thread 4667]
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `/usr/bin/keyutil --command genreq --ascii --renew /etc/pki/tls/certs/cottonwood'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000030b5047e2c in ?? ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
(gdb) bt
#0  0x00000030b5047e2c in ?? ()
#1  0x0000000000000023 in ?? ()
#2  0x00007fff1c7768c0 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb)

Bah!  Next, in the logs, I found :

/var/log/messages:May 22 20:29:00 ch motion: [1] [ERR] [EVT] event_image_snapshot: Could not create symbolic link [01-20160522202900-snapshot.jpg]: No such file or directory
/var/log/messages:May 22 20:29:30 ch motion: [1] [ERR] [ALL] create_path: Problem creating directory /home/media/security: Input/output error
/var/log/messages:May 22 20:29:30 ch motion: [1] [ERR] [ALL] put_picture: Can't write picture to file /home/media/security/cameras/camera-backyard//01-20160522202930-snapshot.jpg: Input/output error

I completely missed the "Input/output error" and checked the filesystem location, only to find my entire disk had failed, so it could not write videos.  Sometimes, debugging can be such a pain!  Here comes a nice recovery fee and a nice fat check going out for a NAS with mirroring!