I'm slowly migrating off of a virtual server at a service provider. Most things have been moving to docker. In this case, it's going onto the docker server, but not going to run in docker itself.
It has to be on the same network as the Weatherflow device to be able to pick up data, and me keeping it on the unrouteable network meant this docker platform needed to be on two separate networks. So, I fired off the command to create a new connection on the same interface.
nmcli connection add con-name unrouteable ifname enp1s0 type ethernet ipv4.method manual ipv4.addresses 192.168.2.10/24 ipv4.never-default yes
Unfortunately, this resulted in the system not being able to boot onto the network. So, it was backed out, and I moved the weather hub off of the unrouteable to the regular network. The code below, I simply could NOT get to arrive in any docker image, or running directly on the docker server (outside of a container). However, after a lot of troubleshooting to verify that the broadcast packets were getting there using :
sudo tcpdump -i enp1s0 udp port 50222 -XvvvI finally realized that RedHat had firewall-cmd running by default. Now, I'm trying to make all this work with defaults, which means that instead of disabling it, I'm going to open the port :
firewall-cmd --permanent --add-port=50222/udp
firewall-cmd --reload
firewall-cmd --list-allYou should see the ports: line now containing "50222/udp". And, if you run the server, it should show it's receiving packets. Okay, back to building this "listener". I went down this whole path of making it run an exec() to curl, which made it really easy to send the received JSON to a perl script and customize whatever you needed from there. Yes, at the very bottom of the code I have commented out a "sendJSON" function, and instead I'm using a handleJSON function that will do the parsing, so if you need to flip that switch because you are using a non-docker version of HTTPd and Perl, you can install the package and make it happen.
Since the Perl instances in the Apache/HTTPd 2.4 instances ("httpd 2.4") do NOT include either the Perl JSON module, or the DBD::MySQL module, it ultimately means you can't send this to Apache and deal with it nicely there if you are also using native docker. That leaves one choice - write a UDP "listener" (duh, UDP packages don't "listen" on UDP ports, because there is no connection/handshake).
So, I wrote the dumb thing in perl. I'd already had the UDP side, but since it might as well do the local database insertion, whelp, that's what I did.
Fancy thing is that I can specify which datagram types to accept from the Tempest system. I really only care about lightning strikes and obs_st (standard metrics). But, I slapped SQL for all, and did a fancy ".disabled" extension to hide the ones I wanted. If I don't have a disabled, when it sees a packet, it will dump the hash_ref so that I can easily write one if I need it. I just wrote what I saw, then disabled most of them.
Here's the code for the listener :
#!/usr/bin/perl -w
# written by Joe Lewis <joe@joe-lewis.com> on November 29, 2022
# A UDP listener for the tempest weather system. The API for weatherflow (maker
# of the tempest system) can be found at :
# https://weatherflow.github.io/Tempest/api/udp.html
# if on RHEL, install :
# perl-libwww-perl
# perl-Sys-Syslog.x86_64
# for RHEL, this uses the template tool kit (available in the EPEL repository)
# sudo subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms
# sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
# sudo yum install perl-Template-Toolkit.x86_64
use strict;
use warnings;
use Data::Dumper;
use DBI;
use Getopt::Long;
use HTTP::Request;
use HTTP::Response;
use IO::Socket;
use JSON;
use LWP::UserAgent;
use POSIX;
use Sys::Syslog qw(:DEFAULT :standard :macros);
use Template;
my $template_directory = 'sql';
my $mysql_hostname = '127.0.0.1'; # if you use "localhost", it will try the unix socket
my $mysql_username = 'user_only_having_insert_or_update';
my $mysql_password = 'users_password';
my $mysql_database = 'weather';
my $verbose = undef;
my $daemonize = undef;
GetOptions(
'sql-directory=s' => \$template_directory,
'daemonize' => \$daemonize,
'verbose' => \$verbose,
);
my($sock, $newmsg, $MAXLEN, $PORTNO);
my $json = JSON->new->allow_nonref;
my $userAgent = LWP::UserAgent->new(timeout => 10);
my $postURL = 'https://protected.silverhawk.net/cgi-bin/weatherflow.pl';
my $postContentType = 'application/json';
my $name = 'WeatherFlow-UDP-conv';
my $nagios_command_file = '/var/lib/nagios3/rw/nagios.cmd';
my $error_log_qfn = '/var/log/'.$name.'.log';
my $pid_file_qfn = '/var/run/'.$name.'.pid';
$MAXLEN = 1024;
$PORTNO = 50222;
sub sighup_handler {
exit;
};
sub sigusr1_handler {
exit;
};
sub logMessage {
my $msg = shift;
$msg .= "\n" unless $msg =~ /\n/;
syslog('info',$msg);
};
sub handleUDPPacket {
my $weatherstationname = shift;
my $weatherstationclient = shift;
my $data = shift;
my $href = $json->decode($data);
my $weatherstationip = inet_ntoa($weatherstationclient);
# just for my sanity, insert the hostname
$href->{'client_hostname'} = $weatherstationname;
$href->{'client_ip'} = $weatherstationip;
my $sqlTemplate = $template_directory.'/'.$href->{type}.'.sql';
if (-f $sqlTemplate) {
my $tt = Template->new({INCLUDE_PATH=>$template_directory,INTERPOLATE => 1}) or die "Error launching the template toolkit : $Template::ERROR\n";
# establish a MySQL connection
my $dbh = DBI->connect('dbi:mysql:host='.$mysql_hostname.':database='.$mysql_database.':mysql_ssl=1',$mysql_username,$mysql_password);
$href->{dbh} = $dbh;
my $SQL;
$tt->process($href->{type}.'.sql',$href,\$SQL) or warn 'Template Processing Failure: '.$tt->error()."\n";
if ($SQL) {
$dbh->do($SQL) or warn "Error running SQL on type $href->{type}: $DBI::errstr, SQL :\n\t$SQL\n";
}
$dbh->disconnect();
} else {
# if we have a disabled SQL file, do nothing because we're intentionally swallowing it.
if (!-f $sqlTemplate.'.disabled') {
# no template defined, warn it
warn "non-existant SQL template $sqlTemplate containing:\n".Dumper($href);
}
};
};
openlog("",'ndelay','daemon');
if (defined($daemonize)) {
my $pid = fork();
exit if ($pid > 0);
setsid();
$pid = fork();
exit if ($pid > 0);
};
$SIG{HUP} = \&sighup_handler;
$SIG{USR1} = \&sigusr1_handler;
$sock = IO::Socket::INET->new(
LocalPort => $PORTNO,
Proto => 'udp') or die "socket: $@";
logMessage("Awaiting UDP messages on port $PORTNO");
while ($sock->recv($newmsg, $MAXLEN)) {
my($clientport, $clientaddr) = sockaddr_in($sock->peername);
my $clienthost = gethostbyaddr($clientaddr, AF_INET);
handleUDPPacket($clienthost,$clientaddr,$newmsg);
}
my $status = $!;
logMessage("Loop completed, exiting with $status");
die "recv: $status"; Now, that's relatively simple. I created the separate directory for the SQL's specific to the Tempest functionality, and then piled a bunch of files in there :
*** sql/device_status.sql.disabled
INSERT INTO device_status (timestamp,serial_number,client_ip,client_hostname,hub_sn,voltage,rssi,hub_rssi,sensor_status,firmware_revision) VALUES (from_unixtime([% timestamp %]),[% dbh.quote(serial_number) %],[% dbh.quote(client_ip) %],[% dbh.quote(client_hostname) %],[% dbh.quote(hub_sn) %],[% voltage %],[% rssi %],[% hub_rssi %],[% sensor_status %],[% firmware_revision %])
*** sql/evt_precip.sql.disabled
INSERT INTO evt_precip (timestamp,station_id,hub_sn) VALUES (from_unixtime([% evt.0 %]),[% dbh.quote(serial_number) %],[% dbh.quote(hub_sn) %)
*** sql/evt_strike.sql
INSERT INTO lightning_events (epoch,station_id,energy,distance_km,distance_miles) VALUES (from_unixtime([% evt.0 %]),[% dbh.quote(serial_number) %],[% evt.2 %],[% evt.1 %],[% evt.1/1.6093445 %])
*** sql/hub_status.sql.disabled
INSERT INTO hub_status (timestamp,serial_number,frequency,rssi,client_ip,uptime,firmware_revision,hw_version,client_hostname,radio_version,reboot_count,i2c_bus_error_count,radio_status,radio_network_id) VALUES (from_unixtime([% timestamp %]),[% dbh.quote(serial_number) %],[% freq %],[% rssi %],[% dbh.quote(client_ip) %],[% uptime %],[% dbh.quote(firmare_revision) %],[% hw_version %],[% dbh.quote(client_hostname) %],[% radio_stats.0 %],[% radio_stats.1 %],[% radio_stats.2 %],[% radio_stats.3 %],[% radio_stats.4 %])
*** sql/obs_air.sql.disabled
INSERT INTO obs_air (timestamp,serial_number,firmware_revision,hub_sn,station_pressure_mb,station_pressure_inhg,temperature_celcius,temperature_farenheit,rel_humidity,lightning_strike_count,lightning_strike_avg_distance_km,lightning_strike_avg_distance_miles,battery_voltage,report_interval) VALUES (from_unixtime([% obs.0.0 %]),[% dbh.quote(serial_number) %],[% firmware_revision %],[% dbh.quote(hub_sn) %],[% obs.0.1 %],([% obs.0.1 %]*0.029530)+4.7634625,[% obs.0.2 %],([% obs.0.2 %]*1.8)+32,[% obs.0.3 %],[% obs.0.4 %],[% obs.0.5 %],[% obs.0.5 %]/0.621371,[% obs.0.6],[% obs.0.7 %])
*** sql/obs_sky.sql.disabled
INSERT INTO obs_sky (timestamp,serial_number,firmware_revision,hub_sn,illuminance_lux,uv_index,rain_previous_minute_mm,rain_previous_minute_in,wind_lull_mps,wind_lull_mph,wind_avg_mps,wind_avg_mph,wind_gust_mps,wind_gust_mph,wind_direction,battery_voltage,report_interval,solar_radiation,day_rain_accumulation_mm,precipitation_type,wind_sample_interval) VALUES (from_unixtime([% obs.0.0 %]),[% dbh.quote(serial_number) %],[% firmware_revision %],[% dbh.quote(hub_sn) %],[% obs.0.1 %],[% obs.0.2 %],[% obs.0.3 %],[% obs.0.3 %]/25.4,[% obs.0.4 %],[% obs.0.4 %]*2.23694,[% obs.0.5 %],[% obs.0.5 %]*2.23694,[% obs.0.6 %],[% obs.0.6 %]*2.23694,[% obs.0.7 %],[% obs.0.8 %],[% obs.0.9 %],[% obs.0.10 %],[% obs.0.11 %],[% obs.0.12 %],[% obs.0.13 %])
*** sql/obs_st.sql
INSERT INTO twenty_four_hour (epoch,station_id,wind_lull_metric,wind_lull_standard,wind_avg_metric,wind_avg_standard,wind_gust_metric,wind_gust_standard,wind_direction,wind_sample_interval,barometric_pressure_milibars,barometric_pressure_inHg,temperature_celcius,temperature_farenheit,relative_humidity,illuminance,ultraviolet_index,solar_radiation,rain_amount_metric,rain_amount_standard,precipitation_type,lightning_avg_distance_metric,lightning_avg_distance_standard,lightning_strike_count,battery_voltage,report_interval) VALUES (from_unixtime([% obs.0.0 %]),[% dbh.quote(serial_number) %],[% obs.0.1 %],[% obs.0.1 %]*2.23694,[% obs.0.2 %],[% obs.0.2 %]*2.23694,[% obs.0.3 %],[% obs.0.3 %]*2.23694,[% obs.0.4 %],[% obs.0.5 %],[% obs.0.6 %],([% obs.0.6 %]*0.029530)+4.7634625,[% obs.0.7 %],([% obs.0.7 %]*1.8)+32,[% obs.0.8 %],[% obs.0.9 %],[% obs.0.10 %],[% obs.0.11 %],[% obs.0.12 %],[% obs.0.12 %]/25.4,[% obs.0.13 %],[% obs.0.14 %],[% obs.0.14 %]/0.621371,[% obs.0.15 %],[% obs.0.16 %],[% obs.0.17 %])
*** sql/obs_st.sql.original
INSERT INTO obs_st (timestamp,serial_number,firmware_revision,hub_sn,wind_lull_mps,wind_lull_mph,wind_avg_mps,wind_avg_mph,wind_gust_mps,wind_gust_mph,wind_direction,wind_sample_interval,station_pressure_mb,station_pressure_inhg,temperature_c,temperature_f,relative_humidity,illuminance_lux,uv,solar_radiation,rain_previous_minute_mm,rain_previous_minute_in,precip_type,lightning_strike_avg_distance_km,lightning_avg_distance_miles,lightning_strike_count,battery,report_interval) VALUES (from_unixtime([% obs.0.0 %]),[% dbh.quote(serial_number) %],[% firmware_revision %],[% dbh.quote(hub_sn) %],[% obs.0.1 %],[% obs.0.1 %]*2.23694,[% obs.0.2 %],[% obs.0.2 %]*2.23694,[% obs.0.3 %],[% obs.0.3 %]*2.23694,[% obs.0.4 %],[% obs.0.5 %],[% obs.0.6 %],([% obs.0.6 %]*0.029530)+4.7634625,[% obs.0.7 %],([% obs.0.7 %]*1.8)+32,[% obs.0.8 %],[% obs.0.9 %],[% obs.0.10 %],[% obs.0.11 %],[% obs.0.12 %],[% obs.0.12 %]/25.4,[% obs.0.13 %],[% obs.0.14 %],[% obs.0.14 %]/0.621371,[% obs.0.15 %],[% obs.0.16 %],[% obs.0.17 %])
*** sql/rapid_wind.sql.disabled
INSERT INTO rapid_wind (epoch,station_id,speed_mps,speed_mph,direction) VALUES (from_unixtime([% ob.0 %]),[% dbh.quote(serial_number) %],[% ob.1 %],[% ob.1 %]*2.236936,[% ob.2 %])And to ensure it starts on boot (this is outside of docker - meh, I was so frustrated that I'm fine with it), I created an /etc/systemd/system/weatherflow_udp.service file containing :
[Unit]
Description=UDP Listener for Tempest Weatherflow
After=default.target
[Service]
Type=oneshot
ExecStart=/where/ever/you/stored/it/tempest/perl/udp_listener.pl --daemonize --sql-directory /where/ever/you/stored/it/tempest/perl/sql/
[Install]
WantedBy=default.targetAnd then enabled the service :
systemctl daemon-reload
systemctl enable weatherflow_udpViola (or Cello, I don't care), I have data!
One step closer!
No comments:
Post a Comment