When you have many zope instances distributed on many servers and point all to the same storage and you have many other servers at the frontend with many software for caching, load balancing, mailing, ..etc, you probably know the hell of maintaining such system. I was asked to found out a solution to centralize the logging part of this system.

Let's take just the zope part:

Zope comes with 5 types of logging handlers: logfile, syslog, win32-eventlog, http-handler and email-notifier. For the problem I want to solve the syslog handler is the best choice.

Why syslog ?



The http-handler sends the log with a http request to an URL. It will be slow and will use more system resources and some coding.
The email-notifier is not for centralizing a logging system it is just for notification.
The win32-eventlog is for windows :-)

The syslogd come pre-installed with all *nix systems and can forward logging requests to other syslogd on other server using the UDP protocol.

Centralizing:



To have all zope log files in the same place, we have to make zope instances log to syslog and then make syslogd on all servers forward requests to a central server. Let call it my.logging.server.com.

From that we can do with the log files what we want to do: use a log viewer, log analyzer, ...etc

To make a zope instance log to syslog is very easy.
We edit the zope.conf file and instead of the standard logfile handler:


<eventlog>

  level info

  <logfile>

    path $INSTANCE/log/event.log

    level info

  </logfile>

</eventlog>

we use the syslog handler:


<eventlog>

  level info

  <syslog>

    address localhost:514

    facility local1

    format myzope1.myserver1: %(message)s 

    level info

  </syslog>

</eventlog>

I put the string "myzope1.myserver1" in the log format so I can grep for log message from this instance (it will be used also as log file name at my.logging.server.com for multi-file log).
For the facility attribute look in your syslogd.conf file and pick one that is not used. in my case I choose local1.

NOTE: You can let zope log directly to the remote server by replacing localhost:514 with my.log.server.com:514. In this case you will not have any trace of zope logging on the local machine and you haven't to reconfigure syslogd locally.

Now we have to configure syslogd on the local machine.
In your syslog.conf file add this line:


local1.*            @my.loggin.server.com

This will tell syslogd to forward all LOCAL1 logging facility to my.logging.server.com
The syslogd daemon must be started with remote logging enabled. This can be done with -r option on debian. If you have other system, type:


$ man syslogd

and see what options you have.
we start syslogd:


$ /sbin/syslogd -r

check that syslogd is listening to UDP port 514 (syslog port):


$ netstat -a --udp

you should see something like this:


udp      0      0  *.syslog         *.*           LISTEN

Now that we have configured all zope instances to use syslog as event logger and configured all servers to forward logging requests to my.logging.server.com, we have to configure the syslogd at my.logging.server.com.

Logging server:



We will just configure the syslogd like we did with other servers with one exception. The syslogd at my.logging.server.com will not forward logging requests but it will write them to a local folder.
Put this line in syslog.conf file:


local1.*            /var/log/zope.log

This will tell syslogd to log all requests with a logging facility 'local1' to the file /var/log/zope.log.
This means that all your zope instances will log to the same file. If you want one log file per instance, you can make syslogd write to a named pipe like this:


local1.*            |/var/log/zopes/syslog_zope_pipe

you create the named pipe:


$ mkfifo /var/log/zopes/syslog_zope_pipe

and here is a little script that reads from the named pipe and writes to the appropriate log file. You can set it to run at startup.


#!/bin/sh



log_dir="/var/log/zopes"



if [ ! -d "$log_dir" ]; then

    mkdir $log_dir

fi



cat /var/log/zopes/syslog_zope_pipe | while read LINE 

do



  # we remove all letters from the left to 'myzope' included



  tmp=${LINE#*myzope}



  # we remove all letters from the right starting at 

  # the first ':' included

  # here we get the sring added in 'format' attribute 

  # of the syslog tag without 'myzope'



  tmp=${tmp%%:*}



  # and here is my string again as file name



  log_file="$log_dir/myzope$tmp.log"



  echo $LINE >> $log_file



done

Now you can restart all your zope instances and check that the system is working as expected.

Notes:



1. Logging to a named pipe is not the best choice. You have to manage the log rotation yourself in this case. When you log to a file, you get the log rotation management for free and syslogd does the job for you.

2. When all zope instances log to the same file, it is easy to find the information you are looking for with a simple grep or tail | grep

3. the best log viewer in the last case is tail and cat. You can access your log file from your local computer with some thing like this:


$ ssh mustapha@my.logging.server.com 'sudo tail -f /var/log/zope.log'


I like to see tail from my gnome system log viewer. To do that, just redirect the out put of tail to a local file like this:


$ ssh mustapha@my.logging.server.com \

     'sudo tail -f /var/log/zope.log' >> remote.zope.log

and open it with your GUI log viewer you get the tail of the log of all servers in real time on your local computer.