Skip links

A tutorial on remote logging with rsyslog

16 August 2011 13:49 - by Freek Lijten - 5 comments

Tags: , ,

Why are you logging data? Two reasons come to my mind specifically: statistics and debug information. In the first case not being able to access your data for a period of time is not that big of a deal, statistics are only significant if you can collect them over a long timeframe. But if a specific server has an all out breakdown, and one service after another crashes, you want to determine what is happening right now. But then you'd have to have access to your logs over ssh. And this service has just crashed too...

Once again we can count on our OS for offering a solution to this problem. Starting in 2004 Rainer Gerhards started writing rsyslog, a logging daemon which offers remote logging and strong filtering capacities. This article will cover setting up the system for remote logging and show some examples of possible day to day use. I will include the standard syslogs as well as apache's access and error logs

Disclaimer

Be careful if you start changing your logging setup. Backup relevant data and check if your new setup still functions properly afterwards. This article is just an introduction, not a faultless reference. If you don't know what something means or does, look it up please. Backups of configuration files might come in handy too. If I made a mistake and you found out, please inform me as well.

Logging on linux: a small and very incomplete history

Back in the days, when the empire strikes back was still seen as a masterpiece of special effects, Eric Allman created a logging standard for sendmail. This standard soon was adopted by other programs and became the de facto logging standard for unix systems. A small 20 years later syslog-ng, an open source implementation of syslog, emerged. This brought huge improvements in the fields of filtering and configuration. Finally almost in present times (2004) Aforementioned Rainer Gerhards started writing rsyslog as a competitor for syslog-ng because he thought a competitor was simply necessary.

Getting started

Now this is easy, we're on linux after all. aptitude install rsyslog should be enough. What you could do is check if there are other logging daemons running on your system (or maybe you already have rsyslog running). You might run into sysklogd and others. You'll not be needing them as we're going to start remote logging. If you check de rsyslog.conf file in /etc you'll see that is set up for local logging at the moment. For now remove every rule from the file and add only one line:

*.* @1.2.3.4:514

If you now restart rsyslog every priority of every facility will be send to a server with ip 1.2.3.4 over UDP. By adding a second @ in front of the first and changing your port you can send using TCP but I don't mind a log getting lost every now and then so UDP will do just fine. The *.* may be a bit much. If you know that all you are going to do with specific logs is drop them on the receiving server you might as well drop them on sending servers and spare the bandwidth. Read onwards to see how.

That is all for syslog purposes, but we still need to get Apache to also use syslog instead of listening to its own commands. For the error log this is quite simple, we can tell it to use syslog and be finished with it. For the access logs things lie a little different. I disabled other logging rules in our apache setup and put the following rules in /etc/apache2/conf.d/logging.conf (The filename is free to choose, the location isn't):

CustomLog "|/usr/bin/logger -t apache -i -p local6.notice" procurios-syslog
error log syslog

As you can see error log isn't that big of a deal, but for the access logs we need to have CustomLog do something peculiar. Every access log is piped to /usr/bin/logger which results in the log getting received by rsyslog. As you can see the facility (local6) and priority (notice) are also passed along. Finally a specific log format is chosen (procurios-syslog: in this case defined somewhere else in the same file).

So every server is sending syslogs, apache error logs and apache access logs to 1.2.3.4,  the only problem is: at 1.2.3.4 no one is listening....

Setting up the host

To get 1.2.3.4 to listen we need to change its rsyslog.conf file as well. Below is what is needed to listen for UDP on port 514 (there should be a bunch of other stuff in your file, if you installed rsyslog via aptitude it should be there and you only have to uncomment the UDP part).

$ModLoad imudp
$UDPServerAddress 1.2.3.4
$UDPServerRun 514

The configuration above results in exactly three things:

  1. A module is loaded, making our rsyslog set up and capable of listening for UDP packages.
  2. An ip address is defined, if this is left out or a * is used all ips this server knows are listened to. In general you probably don't want this, in our case the machine only listens to its local ip, meaning there can be no outside flooding.
  3. The port where rsyslog needs to be listening is defined.

If you restart rsyslog you can then check if your configuration worked. Using the command netstat -nlp you should get a result which looks like this:

udp        0      0 1.2.3.4:514           0.0.0.0:*          16637/rsyslogd

Storing the incoming logs

So every log from every server is now received at 1.2.3.4. If you want them al in one file all you'll need to do is add a single rule to your rsyslog.conf and restart it:

*.* /var/log/oneGiantHeapOfLogs.log

As this is probably exactly what you don't want we'll need some filters. But before we do that I'll need to introduce you to another concept called templates.

Templates

Since a lot of servers are sending logs to one machine it won't do to simply filter out local6.notice to /var/log/apache-access.log. You'll want the access logs per server at least! The same goes for other stuff so we'll need a way to dynamically put logs of the same facility into different files. For this purpose templates are used. Below are some examples of what we use:

$template syslog,"/var/log/external/%fromhost%/syslog.log"
$template apacheError,"/var/log/external/%fromhost%/apache/error.log"
$template apacheAccess,"/var/log/external/%fromhost%/apache/%msg:R,ERE,1,ZERO:imp:([a-zA-Z0-9\-]+)\.--end%-access.log"
$template mailError, "/var/log/external/%fromhost%/mail/error.log"

There are two things happening here. First of all you'll notice %fromhost%. This is a placeholder which is dynamically replaced with the DNS-resolved hostname of the machine the current log came from. Other options to use are found here.

The second placeholder (%msg:....) a bit more obscure but in the end it is nothing more than a regular expression. Since our servers host multiple implementations it is very convenient to have access logs per implementation. For this we put some information in the LogFormat on the sending machines which is parsed out here. To see the syntax of regular expressions in templates please read this again, but scroll below the property replacers.

Actual filtering

Since we have templates resulting in dynamic filenames now we can start the actual filtering. First we filter out apaches logs:

local7.* ?apacheError
& ~

local6.notice ?apacheAccess
& ~

I'll explain what these lines accomplish. Apache uses local7 to send error logs and we told apache to use local6.notice for access logs, all we do now is put them in their dynamic files. The question mark is necessary to have rsyslog know a template is following. If an error log is coming from v004 it will be put into /var/log/external/v004/apache/error.log if it comes from v027 it will be stored in /var/log/external/v027/apache/error.log. On the next line (which seems to be necessary in this case) there are an ampersand and a tilde. The tilde tells rsyslog to drop all logs that were filtered out by the preceding command, the ampersand is merely used to connect the two lines.

Since our mailservers are logging remotely too, it would be nice if we get mail related errors in a specific file as well. But I'm only interested in errors from actual mailservers, I don't need specific logs for a postfix on a random virtual machine. This proved to be a little more tricky and I don't know if it is the ideal solution but it is working for me:

if $syslogfacility-text == 'mail' 
and $syslogseverity-text == 'info' 
and $fromhost startswith 'mail' 
then ?mailInfo
& ~

The if-and-then construction can use the same property replacers introduced earlier and can also work with a number of predefined compare operations (isequal, startswith and more). If all conditions are met the log is put into another dynamic file and it is dropped afterwards. Please note that everything up to & ~ must be on a single line. The breaks are there for reading purposes only.

Now our apache access and error logs are stored in seperate files as well as the error logs from our mailservers. All we want now is the rest of our logs in the syslog file:

*.* ?syslog

This is the last filter in the file so all that wasn't catched by earlier filters ends up in the syslog file.

Wrap up

The configuration lines above are snippets from our actual configuration, not all is present there. If you want to setup remote logging yourself, take care to keep thinking  and take your own situation into account. Having said that I hope this article will be of use when you decide to start logging remote!

Share this post!

Comments

  1. gauravb gauravb wrote on 14 January 2013 17:58

    Hi, I am able to forward the messages from the rsyslog server to central syslog server with the debug mode and once the debug mode is disabled the rsyslog doesn't send the messages to the central syslog server and it also doesn't storage the incoming messages anywhere withthin the server.

    I am using spoofing , to make sure the source IP is not changed while forwarding the message to the central server.

    Any idea why this happens ?

    Regards

  2. Erik Erik wrote on 31 May 2013 15:20

    What does it mean: Apache "uses" local7 to send error logs? what is "local7" and where can I find that Apache uses it?

  3. Freek Lijten Freek Lijten wrote on 14 June 2013 10:11

    @gauravb, I can't really say to be honest. It can be all kind of things in your setup :(

    @Erik local7 (and others) are part of the syslog environment, see the part on facility levels here: https://en.wikipedia.org/wiki/Syslog

    Apache uses local7 by default as specified here: https://httpd.apache.org/docs/2.4/mod/core.html#errorlog (search for local7 in that paragraphs)

  4. Prabowo Prabowo wrote on 02 July 2013 11:18

    I trying to get my webserver log send to prtg server, but when i add the configuration it show me the following error.

    [root@colibri ~]# /etc/init.d/httpd restart
    Stopping httpd:                                            [FAILED]
    Starting httpd: Syntax error on line 2 of /etc/httpd/conf.d/logging.conf:
    Invalid command 'error', perhaps misspelled or defined by a module not included in the server configuration


    FYI, i am running

    CentOS 6.2
    httpd-tools-2.2.15-15.el6.centos.1.x86_64
    rsyslog-5.8.10-6.el6.x86_64


    it seem that i neet to enable some kind module in httpd/apache.

    any suggestion will be appreciated

  5. Aneesh Aneesh wrote on 22 November 2013 20:44

    Hi,

    This is sles11 OS and it is not listening syslog server IP 10.250.1.230 and udp port 514. Could you please advice why it is listening.

    slestest:~ # grep -i udp /etc/rsyslog.conf $ModLoad imudp $UDPServerAddress 10.250.1.230 $UDPServerRun 514 slestest:~ # tail -2 /etc/rsyslog.conf *.* @10.250.1.230:514

    #

    slestest:~ # netstat -nlp | grep -i syslog udp
    0      0 0.0.0.0:37420           0.0.0.0:*                           19276/rsyslogd


    slestest:~ # lsof -i :37420 COMMAND
    PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
    rsyslogd 19276 root 21u IPv4 23473806
    0t0  UDP *:37420


    Thanks Aneesh

Leave a comment!

Italic and bold

*This is italic*, and _so is this_.
**This is bold**, and __so is this__.

Links

This is a link to [Procurios](http://www.procurios.nl).

Lists

A bulleted list can be made with:
- Minus-signs,
+ Add-signs,
* Or an asterisk.

A numbered list can be made with:
1. List item number 1.
2. List item number 2.

Quote

The text below creates a quote:
> This is the first line.
> This is the second line.

Code

A text block with code can be created. Prefix a line with four spaces and a code-block will be made.