Logentries (logentries.com) Under FreeBSD

Logentries is a SaaS service which lets you stream logs (and lots of other things) from multiple locations and servers for review and processing on their platform.

They provide out-of-the-box installers and packages for a number of Linux and UNIX operating systems but don’t directly support FreeBSD (yet). However their agent consists of a few Python scripts and is fairly easy to get running under FreeBSD using the steps below. I’m also including an rc.d script for easy and automatic startup/shutdown.

To continue you’ll need the following:

• Root on the server you want to watch logs from. All of these commands should be run as root and the Logentries daemon will run as root.
• A Logentries.com account, and the username/password to it.
• HTTPS and GIT (port 9418) TCP outbound open on your firewall.

Side note: We found too many issues with trying to have it run as a low privileged user, though it is likely possible with a lot more hacking — as long as your log files have appropriate permissions.

Preparation

We’ll need to install the git command line tool and Python 2.7 (Python 3.3 will not work for their agent):

pkg install python27 git

And then we’ll make a few directories, one for the scripts to live (/usr/local/le), one where the configuration files are stored (/usr/local/etc/le) and finally one where PID information is written to (/var/run/le)

mkdir -p /usr/local/le /usr/local/etc/le /var/run/le/

Download

Grab the agent scripts from Logentries’ git hub and grab the files we actually want (inside le/src):

cd /usr/local/le &&
git clone git://github.com/logentries/le.git &&
mv le/src/* . && rm -rf le/

Adjustments

Some tinkering is needed to make things work under FreeBSD (though these were reported to upstream and may be included in future releases):

vi le.py metrics.py utils.py

On all files, adjust interpreter line and config dirs:

#!/usr/local/bin/python2.7

CONFIG_DIR_SYSTEM = '/usr/local/etc/le'

LOCAL_CONFIG_DIR_SYSTEM = '/usr/local/etc/le

In le.py adjust the CA root search path to add the FreeBSD lines at the bottom:

authority_certificate_files = [  # Debian 5.x, 6.x, 7.x, Ubuntu 9.10, 10.4, 13.0
                                 "/etc/ssl/certs/ca-certificates.crt",
                                 # Fedora 12, Fedora 13, CentOS 5
                                 "/usr/share/purple/ca-certs/GeoTrust_Global_CA.pem",
                                 # Amazon AMI
                                 "/etc/pki/tls/certs/ca-bundle.crt",
                                 # FreeBSD
                                 "/etc/ssl/cert.pem",
]

Set the PID file location:

PID_FILE = '/var/run/le/logentries.pid'

Lastly there is a weird issue where Logentries only uses the first part of a hostname to identify the host to their service. This isn’t a FreeBSD bug but we ran into an issue with it because a lot of our clients have web1.location1 and web1.location2. When registering web1.location2 we found that web1.location1’s ID was being used. To fix this In le.py line 3399, remove the split() method and replace with:

            self.name = self.hostname_required()

Register The Server

You’ll be prompted for the user/pass on your account, this is recommended as manually specifying the API keys gets confusing easily.

/usr/local/le/le.py register

Adding A Log

To watch a log file:

/usr/local/le/le.py follow /var/log/mylog --name=mylog

Where /var/log/mylog is the log file name and mylog is the name you will see inside Logentries for it.

Setup rc.d Script

You’ll want the agent to come up on startup and be manageable via service logentries start/stop/restart commands. To make this possible we’re including out BSD-licensed rc.d script:

logentries.sh

Save this file as /usr/local/etc/rc.d/logentries (Note: no .sh at the end) and chmod 755 it. Here are the easy commands for this:

wget -O /usr/local/etc/rc.d/logentries "https://www.ateamsystems.com/downloads/logentries.sh" &&
chmod 755 /usr/local/etc/rc.d/logentries

Add to rc.conf so it comes up automatically on startup:

echo 'logentries_enable="YES"' >> /etc/rc.conf

Finally, bring it up:

service logentries start

Boom, you’re in business!