Trying Checkmk under Docker

NetworkChuck rolled a video about a "new" monitoring package called "Checkmk".  It was actually fairly intriguing, as it uses a web-based configuration front-end (don't need to change monitoring using the command-line with a restart).  The video : 

https://www.youtube.com/watch?v=Bjifz55TTPI

Now, some of the added stuff is that it looks like a much nicer integration with metrics, not just the CRITICAL/WARNING/OK/UNKNOWN status.  I have started using docker a LOT more lately, just because it might allow things to be portable if I have to move to new hardware, or if I want to update.  So, we'll give this one a shot.  Let's see if we can do this.

It took me a few commands to find out how the image starts and what ports to forward, because nothing about it was intuitive.  So, after starting the image without it, I found a nice little command that dumps the TCP ports of the container itself.  We can use that to figure out the port to forward using trial and error, frankly.

[username@dockerhost ~]$ docker container inspect checkmk --format '{{json .Config.ExposedPorts}}'
{"5000/tcp":{},"6557/tcp":{},"80/tcp":{}}
[username@dockerhost ~]$

This told me it listens on ports 5000, 6557, and 80 (Apache's HTTPd, if I've read the documentation properly).  Unfortunately, port 80 didn't actually work like I anticipated when I forwarded that.  So, I had to delete the container, and then re-create it.  The  command to start it for me :

[username@dockerhost ~]$ docker run --name checkmk -d -p 0.0.0.0:5000:5000 checkmk/check-mk-community
1b54e26451cc125f2a26b1298a9cfdf9a3b3843ecd3f142dc968c17ffd54acb9
[username@dockerhost ~]$

It dumps the container ID, and the -d tells it to run it in the background. I recommend you then grab the logs, because the first run dumps some information we're going to need such as the default username and password (this password is NOT consistent between images - you really need to obtain that).

[username@dockerhost ~]$ docker logs checkmk -f
### CREATING SITE 'cmk'
Adding /opt/omd/sites/cmk/tmp to /etc/fstab.
Going to set TMPFS to off.
Preparing tmp directory /omd/sites/cmk/tmp...Generating configuration for core (type nagios)...
Precompiling host checks...OK
Executing post-create script "01_create-sample-config.py"...OK
Executing post-create script "02_message-broker-certs"...OK
Updating core configuration...
Adding /opt/omd/sites/cmk/tmp to /etc/fstab.
Going to set TMPFS to off.
Skipping Apache restart.
Created new site cmk with version 2.5.0p15.community.

  The site can be started with omd start cmk.
  The default web UI is available at http://1b54e26451cc/cmk/

  The admin user for the web applications is cmkadmin with password: Bo06iHX7qugF
  For command line administration of the site, log in with 'omd su cmk'.
  After logging in, you can change the password for cmkadmin with 'cmk-passwd cmkadmin'.

### STARTING XINETD
 * Starting internet superserver xinetd
   ...done.
### STARTING SITE
Preparing tmp directory /omd/sites/cmk/tmp...Starting agent-receiver...OK
Starting mkeventd...OK
Starting rrdcached...OK
Starting redis...OK
Starting npcd...OK
Starting automation-helper...OK
Starting ui-job-scheduler...OK
Starting nagios...OK
Starting apache...OK
Starting crontab...OK
### STARTING CRON
### CONTAINER STARTED
^C[username@dockerhost ~]$

What you want is the line that starts with "The admin user for the web applications" (I highlighted it in the example I'm working through).  It tells you the username, and the password you are going to need. Get this logged into your password manager.

Now, since I added the port forwarding of 5000 to 5000, I point to my docker host on port 5000, and was then presented with the login.  Entering the credentials from above gives me the wizard to create my first host (I could create a folder - I skipped that).

Monitoring our first host 

The first host I created was my NAS, because I do want to add docker stuff, and I want to port my Nagios into this thing as well. This should let me see how the metrics differ between the two.  I saw the "Problems with 1 data source detected", and immediately panicked.

Breathe, ya big nerd. It means that the agent connection failed, and I had ensured that was NOT connected (it wasn't checked on the form).  No worries.

Well, you can go back and check the box for that checkmk agent thing, and then select the "No API integrations, no checkmk agent" option, save, and re-scan.

I scanned down the list below that big scary error message, and found that it looks like it did pick up everything via SNMP, except for the temperature.  I could add the "undecided" in that list by clicking the "Monitor undecided services".  But, I opted to keep it fairly clean, and only added CPU, fans, memory, RAID status, and the volume that runs on the RAID.  Just click the green "plus" right next to  the undecided monitor you want.

Once you have them selected, you should see a "Changes" on the right side with a yellow number.  It simply means you have to click in there when you are done and activate the pending changes.

We are now monitoring our first "host".

Getting everything into checkmk

Now, as often as I searched the Googles for "migrating from Nagios to Checkmk", they all simply talked about manually putting in the hosts - e.g., there are no real tools built for migration to make it one fell swoop.  So, since the Docker instance of Nagios has all hosts in one single file in my configuration, I found that I could simply grab the unique hostnames from custom.cfg .

grep host_name /opt/docker/nagios/etc/objects/custom.cfg |awk '{print $2}'|sort|uniq

This gave me a quick list of hostnames to add. What makes this easy is that you can just add the agent, then point to the new host, and then have it look.  We'll do that with docker, just for the sanity.  On the left, click "Setup", followed by "Linux" under "Agents".  Then, right click on the RPM (I'm doing this with RHEL), and copy the link.

Then, on your docker host, run "wget " and paste the URL, e.g. :

[username@dockerhost nagios]$ wget http://dockerhost:5000/cmk/check_mk/agents/check-mk-agent-2.5.0p15-1.noarch.rpm
--2026-10-14 17:23:53--  http://docker:5000/cmk/check_mk/agents/check-mk-agent-2.5.0p15-1.noarch.rpm
Resolving docker (docker)... 192.168.1.219
Connecting to docker (docker)|192.168.1.219|:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5877930 (5.6M) [application/x-redhat-package-manager]
Saving to: ‘check-mk-agent-2.5.0p15-1.noarch.rpm’

check-mk-agent-2.5.0p15-1.noarch.rpm 100%[======================================================================>]   5.61M  --.-KB/s    in 0.02s   

2026-10-14 17:23:53 (348 MB/s) - ‘check-mk-agent-2.5.0p15-1.noarch.rpm’ saved [5877930/5877930]

[username@dockerhost nagios]$

Now install it using "sudo rpm -ivh check-mk-agent-2.5.0p15-1.noarch.rpm".  Mine gave me a bright-red error about "Failed to reload xinetd.service: Unit xinetd.service not found.", but it had no impact (utilizes systemd now, and RHEL9 has NOTHING to support xinetd, anyway).

On that setup->agents side bar again, select "Other operating systems", and find the mk_docker.py under "Plug-ins".  RIght click on it, and copy the link.  Then, wget that again on your docker host to copy the plug-in down.  Install it :

sudo install -m 0755 mk_docker.py /usr/lib/check_mk_agent/plugins/

Also make sure the Python module is installed :

sudo yum install python3-docker.noarch

Download the example configuration file (the same way we downloaded the agent - from the browser, get to "Setup" -> "Agents" -> "Linux", and search for "docker.cfg".  Right click on that, copy the link, then in a shell on the docker host, run "wget URL" (paste the URL in place of the URL, right?).  Copy this into the /etc/check_mk directory on your docker host.

Now, re-scan or perform a service discovery. 

In Checkmk, create a new folder for your docker containers ("Setup"->"Hosts"->"Add folder").  I named my folder "Docker Containers", because I have a simple mind.  If you don't have the "IP address family" checkbox showing, expand the "more settings" on the "Network address", and check the box for "IP address family", and then set the dropdown to "No IP".  Under "Monitoring agents", check the override for the "Check mkagent / API integrations", and then select "no API integrations, no checkmk agent".  If you don't see the "piggy back" option, hit the more on the right again.  Under piggy back, select it, then choose the "Always use and expect piggy back data", and hit save.

Let's get a list of container names now.  In a shell on the docker host, run a command to show them all.

[username@dockerhost ~]$ docker ps --format '{{.Names}}'
checkmk
httpd
nagios
grafana
cloudflare
mqtt
mysql-container
[username@dockerhost ~]$

Copy the names of your hosts.

The next step is to click back into "Setup"->"Hosts", and then into the "Docker Containers" folder.  From there, click the "Hosts" at the top, followed by the "Import hosts via CSV file".

Now, click "Setup" and then "Hosts", followed by the "Add Host" button.  We're going to add our host.

Enter the host in the hostname field, select the "Checkmx agent" under "Monitoring agents", and then click the test-connection to the right.  It should say it was successful.  If so, click the "Save & run service discovery".

Paste the list of hosts into your contents box, click the "Perform automatic service discovery" box, then click "Upload".

Next, select the column for the "host name", then click on the "Import" button that shows up.  Note, this screen shot has an ugly error message because I missed the hostname assignment.

Click the "Start" button, then wait for it to finish, and then do the usual "Changes"->"Activate pending changes" step.

Give it time, and it should load all the information on your docker containers.

The Lazy Way - Scanning The Network

By this point, my second-grader attention span has waned, and I'm just "done".  So, I went into the "hosts" from under "setup", and added a folder, and enabled a network scan for my non-Internet-routeable setup.  I did specify the IP range for that network, and then I went all "Frozen" on it and "just let it go".  Eventually pulled everything up.

Setting This Up To Use A Proxied Authentication

Now for my big frustration with Nagios in docker - I would have had to really hack to get REMOTE_USER in as the user you are in for it.  It continually asked for authentication via Basic-Auth, even after performing the Cloudflare authentication.  I REALLY wanted to keep it tied to who you actually authenticated with (auditing).  Here's how.

Log in to your Checkmk instance.  Go into the Users, and add a user in the format your proxy provides (e.g. e-mail address).

Then, under "Setup", under "General", should be a "global settings". Expand "User interface", and locate the "Authenticate users by incoming HTTP requests".

Next, check the box for "Activate HTTP header authentication", and in the "HTTP request header variable" field that pops up, enter the header name used by your access manager/proxy.

If you have Cloudflare in front of it, you would use "Cf-Access-Authenticated-User-Email" for the header.  Save that.

Now you can do into the Cloudflare portal and create a new hostname/mapping, and you have access through your portal.