1. Andrew's Corner :

Slackware 14.0 and Leafnode-2

This page aims to guide users through the installation and initial configuration of Leafnode-2 under Slackware 14.0 and has been written as support for a SlackBuilds script published on SlackBuilds.org. Leafnode-2 is an NNTP proxy server intended for small sites, where there are few users and little disk space, but where a large number of groups is desired. It can be used to give a regular newsreader off-line functionality as well as significantly enhancing Usenet experience for broadband users. There is also the significant advantage of being able to configure "local" newsgroups accessible only to your private network.

Contents of this Page

I would suggest that this page is best read from start to finish at least initially but I include navigation links below for those who prefer to dabble:

  1. Some Preparation A little preparatory work is required before installing the program.
    1. Setting a Fully Qualified Domain Name Ensure that an unique Fully Qualified Name Name is set for your system
    2. Details of Leafnode-2 Components A quick explanation of the structure of Leafnode-2
  2. Download, Compile and Install the Source Code Downloading the source code for Leafnode-2, compiling and installing it with the aid of a SlackBuild script.
  3. Configure Leafnode-2
    1. Set up the config file Setting up and editing the configuration file for Leafnode-2.
    2. Set up the filter file Setting up and editing the filter file for Leafnode-2.
    3. Set up local groups Setting up "local" newsgroups.
    4. Configure the server Setting the server running.
  4. Set up some cron jobs Setting up cron jobs for Fetchnews and Texpire.
  5. Set up logging Setting up the system logs for news and organising log rotation.
  6. Run the program Fetching the first run of news and finalising the directory structure.

It is a pretty imposing list of steps above but do not fear, Gentle Reader, once it is all in place it will rarely need to be touched. First for some preparation:

Some Preparation

There is a little preparation to do before installing Leafnode-2 but on a well setup Slackware system much of this groundwork will already be in place. There will need to be a user and group called "news" in place but this is the default in Slackware and should already be there. As well the appropriate PCRE regular-expression matching library is a standard part of Slackware and can be found in the "l" series. Next to ensure you have a Fully Qualified Domain Name in place:

Setting a Fully Qualified Domain Name

Next consideration is to ensure that you have an unique Fully Qualified Domain Name (FQDN) for your system. This should have been set during the initial Slackware installation but I fear, Gentle Reader, that this step is often neglected or done without careful thought. Check your own setup with the following command:

$ hostname --fqdn

The name must be unique to avoid the possibility of newsgroup posts with the same Message-ID colliding. There are three good choices here:

  1. If you own your own Domain Name use this in conjunction with a subdomain and the hostname of your computer.
  2. Use a service such as dyndns.org which will give you adomain name for free.
  3. Use the news service individual.net which will give you a free FQDN.

Check that the FQDN is set properly in /etc/HOSTNAME and /etc/hosts and then you are right to continue the setup. If you are still a little puzzled by this requirement I recommend that you read the excellent README-FQDN.pdf that comes with the Leafnode-2 source code. Next for a quick word on the structure of the Leafnode-2 program:

Details of Leafnode-2 Components

This is as good a time as any to mention that Leafnode-2 is not really a single program but functionally it could be thought of as 5 linked applications. These applications are:

Leafnode
This is the actual NNTP server, the daemon that on Slackware we will run as an inetd service.
Fetchnews
This is is a program which submits and retrieves new articles to or from the upstream NNTP server. It can be run manually from the command line or set as a cron job.
Applyfilter
This is a program that applies filter settings to your news spool. It is used if you wish to remove postings from your spool that match a pattern that you have set in /etc/leafnode/filters.
Texpire
This is a program which removes old articles from the local news spool and also rehashes the spool after updates of leafnode.
Newsq
This is a program that outputs a summary of news that is in leafnode's out.going, in.coming and failed.postings news queues.

Each of these applications have their own man page and I encourage you, Gentle Reader, to explore this documentation fully. Perhaps this is a good time to have a look at my favourite man pages cartoon? With that fully understood it is time to download the source code for Leafnode-2, compile and install it:

Download, Compile and Install the Source Code

The source code for Leafnode-2 is still marked as an alpha release but do not fear, Gentle Reader, the code is for the most part quite stable. The cleanest and most system-friendly method of installation of Leafnode-2 on Slackware is by using the SlackBuild script. This is a script I wrote myself but has also been vetted and improved by the SlackBuilds.org team and is hosted on their site. First download the SlackBuild archive and open it:

$ wget http://slackbuilds.org/slackbuilds/14.0/network/leafnode.tar.gz
$ tar xvf leafnode.tar.gz

This will reveal the following structure:

 ./leafnode
  |-- README
  |-- leafnode.info
  |-- leafnode.SlackBuild
  |-- doinst.sh
  |-- slack-desc

You will then need to change to the leafnode directory and download the source code of Leafnode-2 into this directory:

$ cd leafnode
$ wget http://home.pages.de/~mandree/leafnode/beta/leafnode-2.0.0.alpha20110807a.tar.bz2

This is the non-lua version of Leafnode-2 but IMHO it is still the best version to run until the developers have had time to iron out the kinks in the more recent lua-enabled versions. The final step is to ensure that the script is executable and after this run the script as root. Have a look through the script first though as one should always be wary of using another person's script regardless of its pedigree:

$ chmod +x leafnode.SlackBuild
# ./leafnode.SlackBuild

If all is well the package will be built and placed in /tmp and from there simply run installpkg to install the program. If there are any problems with this firstly have a read through the SlackBuilds SlackBuild Usage HOWTO page which speaks of the usage of SlackBuilds scripts in much greater detail than I have given here and then if you are still puzzled contact the maintainer of the script. As I am the maintainer of the script I can guarantee a helpful and warm reception!

Configure Leafnode-2

There are a few different ways to approach the configuration of Leafnode-2, I simply suggest one method. Might I suggest at that even at this late stage you ensure that you have read the documentation that accompanies the Leafnode-2 source code? In particular the file README which covers this and other areas in great detail. But lets get started:

Set up the config file

There is a sample config file that Leafnode-2 installs as /etc/leafnode/config.example and unless you already have a configuration file you will need to do the following :

# cp /etc/leafnode/config.example /etc/leafnode/config

There really is very little that needs to be adjusted here but as a bare minimum you will need to open the file /etc/leafnode/config and adjust the following:

server = your.news.server
username = your.username
password = your.password

You might also put some thought into how many messages you wish to download when first subscribing to a group. I have mine set at 500 which saves me from downloading many 1,000s of messages for a group I may not read for very long:

initialfetch = 500

If you decide that you would like to fill the Leafnode-2 cache at a later stage with say the last 2,000 messages in a particular newsgroup you can run the following:

# fetchnews -vvv -N alt.os.linux.slackware -x 2000

There are a few more well-documented options in this file that I leave you, Gentle Reader, to read at your leisure. Now we set up the filter file:

Set up the filter file

Improved in Leafnode-2 over the original Leafnode is its ability to filter newsgroup postings based on Subject, From, Date, Message-ID, References, Bytes and Lines and then decide to download, score or kill these messages. Again a sample file comes with Leafnode-2 and can simply be renamed:

# cp -v /etc/leafnode/filters.example /etc/leafnode/filters

and then sourced from within /etc/leafnode/config:

filterfile = /etc/leafnode/filters

This file contains many examples of filters and I suggest that you carefully read the vital man pages man filterfile and man pcrepattern before embarking on your own filtering efforts. Fetchnews will filter your incoming posts based on your /etc/leafnode/filters file but if you want to clean your existing local spool you can initially run applyfilter in test mode as follows:

# applyfilter -v  -n  name.of.newsgroup

If there is suitable material to filter then run the command without the -n option and your filter will be applied. Results of filtering will be logged in the file news.info that we will be setting up soon, along with all the other logs. If you would like to see the filters in action try the following command:

# fetchnews -D33 -e

Debugging can be set for filters as well but I shall leave you, Gentle Reader, to explore this on your own.Now to setup some local groups:

Set up local groups

Leafnode-2 has the ability to run newsgroups that are created and run at a local level. In the example I give here two local groups are created, one for general access and one that is controlled by a moderator. If you used the SlackBuild script to install Leafnode-2 you will already have the required file in place but otherwise create this file now as /etc/leafnode/local.groups. Add the following to the file ensuring that the spaces between the groups of words are actually "TAB" spaces using your keyboard:

local.test       y    Local Unmoderated Group
local.moderated  m    Local Moderated Group

In your list of available newsgroups you will now have the group named local.test which can be posted to freely and the group local.moderated in which all posts must go through a moderator. To set the moderator details have to be added to the file /etc/leafnode/moderators which will have been created by the SlackBuild script. For our example the settings would be:

local.moderated:email.address@domain.com

This is enough to allow Leafnode-2 to foward all posts to the group local.moderated directly to the nominated email address. It is probably a good time, Gentle Reader, to check that you either have sendmail correctly configured or you have specified a different MTA in /etc/leafnode/config otherwise Leafnode-2 will not be sending any emails to anybody! To make the "approved" post available to your audience you will have to reinsert the message into the newspool either with your news client or a script, or perhaps a combination of the two, bearing the following header:

Approved: email.address@domain.com

How cool is that! There are great possibilities here for anyone running a small LAN. But now to configure the server itself:

Configure the server

Finally to lay hands on the server itself! Leafnode-2 runs from the BSD Internet super-daemon inetd and must be added to its configuration file. Open the file /etc/inetd.conf and first check that there are no lines starting with "nntp" (without the ""). If they exist you will need to comment them out by adding a hash mark "#" (without the "") at the beginning of the line. Then add the following line:

# Leafnode 2 services:
nntp stream  tcp nowait  news /usr/sbin/tcpd /usr/sbin/leafnode

Just to be sure check the paths of both leafnode and tcpd otherwise none of this will work! A quick word of caution here: inetd is not as commonly used now as it used to be for a variety of security concerns. In fact for many people Leafnode 2 will be the only program that uilises inetd and perhaps these people will emulate my personal practise and comment out all other lines in inetd.conf. Before testing the server we need to ensure that tcpd is policing exactly who can access it. Add the following line to /etc/hosts.allow:

#-- leafnode begin
leafnode: 127.0.0.1
#-- leafnode end

And the following line to /etc/hosts.deny:

#-- leafnode begin
leafnode: ALL
#-- leafnode end

There are other options for /etc/hosts.allow and /etc/hosts.allow that give you a huge degree of flexibility in allowing or disallowing access to your server but I leave you, Gentle Reader, to explore this on your own. But now test this setup by stopping and subsequently restarting the inetd Super Server as follows:

# /etc/rc.d/rc.inetd restart

Then use use telnet to login to the server, and if all is well when you login you should get a response something like this:

# telnet localhost 119
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
200 Leafnode NNTP daemon, version 2.0.0.alpha20080904a at mydomain.org 
quit
205 Always happy to serve!
Connection closed by foreign host.

And your proxy NNTP server is running! To ensure that inetd itself runs on system startup ensure that the permissions are set as follows, with the file executable:

$ ls -l /etc/rc.d/rc.inetd
$ -rwxr-xr-x 1 root root 497 2003-09-12 13:27 /etc/rc.d/rc.inetd

Of course if the file is not executable the following command is required:

# chmod +x /etc/rc.d/rc.inetd

And now the BSD Internet super-daemon inetd will come to life with system startup. This might be a good time as well to set your $NNTPSERVER variable and my preferred method is to add the following to ~/.bashrc:

# Sets $NNTPSERVER for Leafnode-2
export NNTPSERVER=localhost

It only remains to set some cron jobs and organise the logging before we finally run the program. Remember once this is all set in place you will rarely have to touch it so have patience for a little while longer!

Set up some cron jobs

The easiest way to run both Fetchnews and Texpire is from cron. I will demonstrate here a method using crontab that sets Fetchnews to run every hour at 15 past the hour and runs Texpire at 2145 every night. Open or create the news crontab as follows:

# crontab -u news -e

and add the following settings:

# Sets texpire to clean out news at 2145hrs every day:
45  21 * * * /usr/sbin/texpire 1>/dev/null
# Sets fetchnews to run at 15 past every hour:
15 * * * * /usr/sbin/fetchnews  1>/dev/null

These are fairly conservative settings and many people set Fetchnews running much more often that every hour and Texpire perhaps a little less frequently. Experiment a little until you find settings that agree with your Usenet habits. Check your settings after every change as follows:

# crontab -u news -l

If all is well we now move on to setting the logs for Leafnode-2:

Set up logging

It is a good idea to set some logging for Leafnode and we will be doing this by tapping into Linux system logging. First to set the required directory structure and files:

# mkdir -pv /var/log/news/oldnews
# touch /var/log/news/news.info
# touch /var/log/news/news.crit
# touch /var/log/news/news.err
# touch /var/log/news/news.notice

With the directory and file structure in place now open the file /etc/syslog.conf and add the following settings, making sure that there are "TAB" spaces (using your keyboard) separating the file from the path:

# Log files for Leafnode-2 news server:
news.=crit          -/var/log/news/news.crit
news.=err           -/var/log/news/news.err
news.notice         -/var/log/news/news.notice
news.info           -/var/log/news/news.info

And finally we set the program logrotate to rotate the files daily for 7 days and then delete them, saving and compressing the old logs to /var/log/news/oldnews. Open the file /etc/logrotate.conf and add the following lines:

# Rotation for Leafnode-2 news logs:
/var/log/news/* {
    daily
    rotate 7
    olddir /var/log/news/oldnews
    missingok
    postrotate
    killall -HUP syslogd
    endscript
    compress
}

It is best to test all these setting before proceding and this can be done as follows. The first command resets system logging and the second command performs a "dry-run" logrotate without actually changing anything:

# killall -v -HUP syslogd
# logrotate -dv /etc/logrotate.conf

It may seem a little complex but this provides a very detailed logging for Leafnode-2 that is perfect for troubleshooting and again once it is set you will not have to trouble with it again. But now, Gentle Reader, it is finally time to actually run the program:

Run the program

Finally it is time to actually run Leafnode-2! First run this command:

# fetchnews -vvv

This may take some time depending on the speed of your connection and the number of groups being downloaded. When this finished you will need to finish generating the directory structure by running the following command:

# texpire -vvv

And you are done! Open your "subscribed" groups with the newsreader of your choice and you will see something like the following:

Leafnode placeholder for group alt.os.linux.slackware

and mark them as "Read" before running Fetchnews again. Fetchnews will then bring in the news for your subscribed newsgroups. And BTW welcome to the world of Leafnode-2! There is much to explore with this amazing program and much, much more to learn than is seen on this simple page which aims merely to provide a starting point.

And in conclusion...

I owe Garry Knight a special thanks for his page Using Leafnode as a Caching Proxy which was very helpful in research for my own page.Thanks also to Michiel van Wessem and Robbie Workman from Slackbuilds.org for their assistance with the Slackbuild script. Please feel free to contact me with any errors of fact that you have found on this page, any errors of opinion will probably remain uncorrected. In the meantime I am having a great time exploring Leafnode-2, slrn and Usenet, what about you?