Skip to content

The importance of good examples in coding and configuration files

My employer has chosen to use DataDog for some of it’s monitoring, and I have been having a really hard time getting simple process monitoring to work reliably. Turns out that the process.yaml file syntax used by DataDog agents is very dependent on Python language psutil calls, and there is quite a difference between single quotes (used in Datadog’s examples) and double quotes (needed for searching for running processes where the unique string is in the middle of a very long line).

Datadog’s Process check is documented pretty well at Process check and the simple checks are easy and work right away. Checking for a running httpd process or nginx process is trivial using the example, and the PID check works, though I am not sure how useful it is as pretty much no one uses static PID assignment. What the examples need to include are an effective fuzzy search to pull the existence of a specific instance of a Node.js or Java Servlet out of many possible running processes. The simple name search for ‘java’ is not very helpful – as I have as many as a dozen separate Java servers running on a host. Likewise a simple name search for ‘node’ is useless as I have as many as thirty node.js servers running at a time… I spent far too many hours trying to get the exact name match to work until I discovered that the switch to double quotes and the use of the exact_match: False boolean operator make this fairly reliable… given that running node and java are so common, why doesn’t DataDog include examples of that?

Here are mine, /etc/dd-agent/conf.d/process.yaml contents:

name: cassandra
search_string: ["java -ea -javaagent:/usr/share/dse/cassandra/lib/jamm-0.2.5.jar"]
exact_match: False
ignore_denied_access: True

search_string: ["node /full/path/to/nodejs/bin/mu/fuzzyblink.js"]
exact_match: False
ignore_denied_access: True

Run service datadog-agent restart ; sleep 8 ; service datadog-agent info to reset you datadog agent and verify the syntax of your process.yaml file.

Now you can set up a process monitor alert through your DataDog cloud account and look for process:cassandra and metrics coming in from the agent. The double quotes are the key.

Tagged ,

3rd generation Ford Focus owner/driver tips

Ford Focus, DCT transmission anomalies explained, and some useful information for 3rd generation Ford Focus owners/drivers.

We’ve been reading loads of useful information and tips at:

In particular – this posting makes a a LOT of sense:

Ford Powershift DCT Transmission info use guide new owners look here

Trump, the GOP, and The Fall

This is absolutely brilliantly written and worded; and expresses exactly my feelings on the current election and self-destruction of the Republican party… I confess I never know if I should cheer on that self-destruction, or lament the loss of a reasonable and rational counterpoint to the Democratic party, which does need a healthy check against it often.

Update: I should also point out the full, original text by Scalzi himself is at, I just think John Gruber did such an amazing job excerpting the key portions and commenting that it was worth linking to his review of Scalzi’s text.

“Trumpster Fire”

“Trumpster fire” – John Gruber knocks it out of the park again with my favorite phrase of the week when he compares the Republican National Convention to a “Trumpster fire” – awesomely apt description.

Dispicable and predatory practices by corporations (Microsoft)

This is nauseating…

I was annoyed by Apple’s pushy iOS 9.3.2 upgrade pressure; but this is way beyond anything Apple has done so far…

Tagged , , ,

Wow – visualizing the massive scale of global shipping is interesting

We live in amazing times indeed where planetary shipping can be visualized like this…

Adding a MediaWiki to BananaPi

In previous post I covered how to stand up CentOS 7 on BananaPi:

This is how to add Apache, PHP, MariaDB and MediaWiki to a BananaPi.

Install a bunch of software:
yum -y install httpd php php-mysql php-gd php-xml mariadb-server mariadb

First, set up the database:
systemctl start mariadb

disable anonymous users
enforce only local root login
remove test database
reload priv tables

mysql -u root -p
CREATE DATABASE wikidatabase;
GRANT ALL PRIVILEGES ON wikidatabase.* TO 'wiki'@'localhost';
SHOW GRANTS FOR 'wiki'@'localhost';
systemctl enable mariadb

I like to store the passwords in a password vault like KeePassX, 1Password, or LastPass.

Set up apache:
systemctl enable httpd
vi /etc/httpd/conf/httpd.conf

Change DocumentRoot “/var/www/html”
to DocumentRoot “/var/www”

Change change ” DirectoryIndex index.html”
to ” DirectoryIndex index.html index.html.var index.php”

echo "It Works" >> /var/www/index.html
systemctl start httpd.service

Configure firewall to allow WebService
yum -y install system-config-firewall-tui
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload

Test by pointing a browser at the IP_OF_BANANAPI

Now install MediaWiki:

cd /root
curl -O
gpg --verify mediawiki-1.26.2.tar.gz.sig mediawiki-1.26.2.tar.gz
cd /var/www
tar -zxf /root/mediawiki-1.26.2.tar.gz
ln -s mediawiki-1.26.2/ mediawiki
chown -R apache:apache /var/www/mediawiki
systemctl restart httpd.service

Finish setting up your MediaWiki Server by pointing a web browser at http://IP_OF_BANANAPI/mediawiki and finish setting up the Wiki.

Enjoy your own Wiki server.

Caching BIND name server on BananaPi

I previously covered how to install CentOS 7 Linux and NTPD on BananaPi here:

This post is how to install BIND name server for caching DNS on BananaPi.

Install and enable the BIND software:
yum -y install bind bind-chroot
systemctl enable named.service

Go get a root hints file:
yum -y install wget
wget --user=ftp --password=ftp -O /var/named/named.root
cp /usr/share/doc/bind-*/sample/etc/named.rfc1912.zones /var/named/chroot/etc

Update your named.conf file:
vi /etc/named.conf

Change the string listen-on port 53 {; };
to listen-on port 53 {; IP_OF_BANANAPI; };

Change allow-query { localhost; };
to allow-query { localhost;; }; (only use whatever subnet you have)

Add to options block:
forward first;
forwarders {;;;;

Use IPs for the public caches that are fastest from your location; I use NameBench on my Mac to determine the fastest local servers.

That creates a caching name server; you can also (optionally) add local zones too if you like.

Configure firewall to allow DNS:
yum -y install system-config-firewall-tui
firewall-cmd --permanent --add-port=53/tcp
firewall-cmd --permanent --add-port=53/udp
firewall-cmd --permanent --add-service=dns
firewall-cmd --reload
chgrp named -R /var/named
chown -v root:named /etc/named.conf
restorecon -rv /var/named
restorecon /etc/named.conf
named-checkconf /etc/named.conf
cd ; systemctl start named
cd ; systemctl restart named

Install dig/nslookup tools and verify your BIND/DNS server:
yum -y install bind-utils

Test from another computer:

Congratulations – you have a caching DNS server.

BananaPi server running CentOS 7 Linux

I enjoy tinkering with small, energy-efficient servers. My latest toy is a LeMaker BananaPi (RaspberryPi clone, but with eSATA and gigabit ethernet).

Grab a CentOS 7 for ARM image from, mine happens to be CentOS-Userland-7-armv7hl-Minimal-1511-BananaPi.img

Stuff that onto an SD card (mine is 16GB, but this should work even smaller) using dd commands. Insert the SD card into the BananaPi, plug the server into a MicroUSB power supply and it should boot and get an initial IP address from your DHCP server if you have one. Find that IP by checking your router; or connect keyboard, mouse, and monitor to see the console of the server (optional).

Find the device name for the SD card by TYPE, NAME, SIZE
df -h or sudo diskutil list

On my Mac the device name is /dev/disk6s1
sudo diskutil umount /dev/disk6s1

For the DD command, switch to the raw device name and the whole disk (/dev/rdisk6) to go much faster.
sudo dd if=/Users/snolan/Downloads/CentOS-Userland-7-armv7hl-Minimal-1511-BananaPi.img of=/dev/rdisk6 bs=4m ; tput bel
Control-T to check progress of DD command – mine took about 20 minutes…

Pop the SD card into the BananaPi and power it on.
Check your router or DHCP server to see the new device’s IP address on the network.

SSH into the root@IP_of_server or use the console to login as root – either way initial password is “centos” which needs to be changed ASAP! Put your new root password into your password vault (I use KeePassX).

Let’s configure this little Server to use a static IP address now…

Find the device name of the active interface (eg: eth0):
nmcli dev status

cd /etc/sysconfig/network-scripts/
ls -l ifcfg*
vi ifcfg-
eth0 (switch to your interface/device name)

The ifcfg-XXX file should look like:

Update the network file:
vi /etc/sysconfig/network

The network file should look like:

Restart networking:
systemctl restart network.service

You will need stable system clock, I like to use GMT so arbitrary daylight saving stupidity does not impact my databases:
yum -y install ntp

Verify you can reach some public time servers:
grep "^server" /etc/ntp.conf

ntpdate -q
You want to see stratum values in low single digits (stratum 2 or stratum 3)…

ntpq -p
Make sure the servers you are looking at are stratum 2 or 3 (the “st” field in the tabular output)…

vi /etc/ntp.conf

Add the following line:
restrict 192.168.subnet.ip netmask nomodify notrap

systemctl stop ntpd
systemctl enable ntpd
systemctl start ntpd

Test locally:
ntpdate -q
That should show a low single digit stratum number (3 or 4)…
That should show current UTC/GMT time…

Configure firewall to allow NTP
yum -y install system-config-firewall-tui
firewall-cmd --permanent --add-service=ntp
firewall-cmd --permanent --add-port=123/udp
firewall-cmd --reload

Test from another computer:
ntpdate -q 192.168.subnet.ip_of_server
That should show a low single digit stratum number (3 or 4)…

See which clients are using my NTP server
ntpdc -c monlist
yum install tcpdump
tcpdump udp port 123 -i any

DNS (Caching name server) and Apache/MariaDB/MediaWiki in another post…

Splunk Forwarder on BananaPi

I have been tinkering with a nifty little RaspberryPi clone made by Lemaker and called BananaPi (basically a RaspberryPi model B with 1 gigabyte memory, eSATA connector, and gigabit ethernet). It’s the size of a deck of playing cards in it’s clear acrylic case, and runs CentOS 7 Linux server operating system.

I have it running ntpd, named (DNS BIND), httpd, MariaDB, PHP, and MediaWiki. I’ll probably put other things on it soon. Like any server, it should be monitored; so I was delighted to find an ARM based Splunk Forwarder at

Now my logs and events are getting shipped to my Splunk indexer and I can monitor and graph anything that logs on the BananaPi.