Mail2News Mini-Howto Rick Dean v2.0, 1 July 1999 This document describes how to feed a mailing list to a news server. 1. Copyright, Distribution, etc. 1.1. What is this? This document describes how to feed a mailing list to a news server using a Linux box. It is called a mini-HOWTO, specifically the Mail2news mini-HOWTO. 1.2. Copyright and such Copyright (c) 1999, Rick Dean. Copyright (c) 1996, Robert Hart. The authors retain their copyright of this document. You are hereby granted permission to redistribute this document in whole or in part as long as it includes this copyright notice. Commercial redistribution is allowed and encouraged. All translations or derivative works of this document must be covered under this copyright notice, and without additional restrictions on distribution. This arrangement is also known as a copyleft. This copyright notice, itself, is hereby placed in the public domain. You may copy it without atribution. 1.3. Where can I get this HOW-TO? The latest version of this document is available at http://fdd.com/howto/ . Many Linux distributions include mini-HOWTOs in the /usr/doc/HOWTO directory. This mini-HOWTO like most Linux mini-HOWTO's was written initially is SGML. This allows the text to be automatically translated to many formats including text, HTML, PostScript, etc. Those other formats are available somewhere. More information about Linux documentation can be found at the Linux Documentation Project , and so many other places. 1.4. Document history Version 1.0 was written by Robert Hart in 1996 Version 2.0 was written by Rick Dean in 1999 2. Overview 2.1. What? Why? and some definitions This document describes how to feed a mailing list to a news server using a Linux box. It is called a mini-HOWTO, specifically the Mail2news mini-HOWTO. A mailing list (also known as a remailer), is an address where e-mail will be resent to a list of other addresses. This is useful for colaboration of geographically disperse groups. Many standards bodies like the working groups of the IETF use mailing lists. Unfortunately, if one is subscribed to several mailing lists, one's inbox may be routinely flooded. Furthermore, some companies (such as 3Com) specify which e-mail client (such as Lotus Notes) their employees must suffer with. Redirecting these e-mails to a news server frees people to choose a news reader and utilize refined features specifically designed for the task (of deriving signal from noise). News servers started on the Internet long ago, many years before the WWW. They (and the news reader clients) have features such as... threading - responses are grouped with the e-mails they refer to read indication - the computer keeps track of what you have seen (even across newsgroups!) so you don't have to. subject kill - mark threads as read by thread. archiving segregation by newsgroup (i.e. mailing list) a quick standardized method subscribing/unsubscribing and more.. A big focus of news servers is sharing news between servers. The largest of these groups became known as USENET. This mini-HOWTO does not address that. You could share the newsgroups created with this mini-HOWTO on your own, but you will live just fine without it. Like a web server, ubiquitous Internet connectivity has made centralized news servers acceptable. Furthermore, recent benchmarks have shown single-processor Linux boxes can handle 1300+ HTTP hits per second, so scalability is a minor issue. Athough you do not need to own the mailing list to use mail2news, it is a good idea to own the news server. 2.2. Assumed environment This document assumes you are using Linux, but other Unices are nice. Currently only sendmail (for a mail delivery agent) is described, but as qmail grows in popularity (in part because it is easier to configure). Hopefully someone with submit configuration notes for it too. For a news server, this document describes innd. It is pretty dominant as news servers go, but any NNTP compliant one should work. A bit of glue called mail2news.pl is a perl script, thus you need the Perl interpreter, but it is very common and probably already installed. Finally, I assume you are running all this (except the mailing list remailer) on one machine. Dividing it up is left as an exercise for the reader. :-) At the time of this writing this mini-HOWTO was only tested against a RedHat-6.0 distribution. As a good computer scientist, you should not believe anything works until it has been specifically tested. Any feedback or notes relating to other distributions would be welcomed by the author. For most of this mini-HOWTO you will need root access unless otherwise specified. 2.3. Methodology This mini-HOWTO is presented backwards, as this is the easiest way to build and debug it. Backwards means we start with the newsreader and work upstream to the mailing list remailer, the opposite direction of normal data flow. This systems uses several hairy pieces (like sendmail and innd) which are sizeable mini-HOWTO's in thier own right. 2.4. Not covered This mini-HOWTO does not cover... setting up a mailing list remailer. exchanging news between news servers. a complete list of things not covered. Please do not e-mail me about these subjects (or SPAM). 3. The news reader 3.1. Netscape Netscape comes with an integrated news reader. The easy way to subscribe to a group is type in (or click on) a link like news://fuji.sfour.com/ietf.confctrl Once you have subscribed, you only need to go to the message center. This can be done by clicking on the small talk balloons icon in the bottom right of a browser window. 3.2. Free Agent Free Agent is a wonderful news reader by Forte for Windoze. See http://www.forteinc.com/agent/freagent.htm 3.3. trn and friends The old command line newsgreaders like trn and tin are a good standby. Be sure to set the environment variable NNTPSERVER first. For example... export NNTPSERVER=fuji.sfour.com trn 4. The news server 4.1. Installing the news server You need to install a news server. I used inn but others are available (somewhere). Preferably just check the "News Server" box during your initial install, but alternatively if your distribution uses RPM (RedHat package manager), then use something like... rpm -i inn-2.2.9.i386.rpm 4.2. Running the newsserver To manually start or stop the news server, use a command like /etc/rc.d/init.d/innd start or /etc/rc.d/init.d/innd stop or /etc/rc.d/init.d/innd restart To have the news server start at boot you could add a command like this to end of /etc/rc.d/rc.local, but that is not the best. Many distributions have a graphical tool for choosing which daemons run. You can also try the command line program setup or chkconfig --add innd 4.3. Creating the newsgroup innd is pretty picky about permissions and ownership. For much of the news config you will need to be the user news. To become this user from root... su - news Using ctlinnd, create the newsgroup on your news server. Remember, the newsgroup will be local, so start it with a distinctive name so you can filter it out from your news distributions if you do that stuff. I shamelessly named my newsgroup ietf.confctrl. The words from left to right go from less to more specific. You also need to tell innd that the group is moderated (by using ctlinnd). Indicating a moderated group is done by specifying m to the newgroup command. For example... ctlinnd newgroup ietf.confctrl m confctrl@isi.edu The newsgroup is set up as a moderated group, as this allows us to take advantage of the email capabilities of innd. Any messages posted to a moderated group are not immediately submitted to the group. Instead, messages are emailed to the moderator of the group. In our example confctrl@isi.edu is the address which is resent by the remailer. If you are sharing news with other servers, remember to edit your newsfeeds so that this group is not not distributed (unless you specificaly wish this to occur). 4.4. Unrestricting access By default the news server, doesn't let any clients read news, so I needed to disable the user authentication of innd. This was done in the /etc/news/nnrp.access. Check out the nnrp.access man page to learn the syntax of this file. I changed the first non-comment line to ... *:Read Post:::* If you want a username/password, fill in the 3rd and 4th (colin separated) fields. For more information on the syntax, check the man page... man nnrp.access 4.5. Changing permissions of /usr/bin/rnews I had to change the permissions of /usr/bin/rnews. It was not world read/executable, but sendmail runs scripts as nobody. chmod a+rx /usr/bin/rnews 4.6. Testing article posting If you copy the following article to a file named rick.article... ______________________________________________________________________ Path: rick From: rick@fdd.com Message-ID: <199907120548.AAA05475@fdd.com> Subject: test Date: Mon, 12 Jul 1999 00:48:49 -0500 (CDT) Newsgroups: ietf.confctrl Approved: ietf-confctrl@kepler.hedland.edu.au NNTP-Posting-Host: localhost Organisation: (mail2news gateway) test ______________________________________________________________________ Then you should be able to post a file with... /usr/bin/rnews -r localhost ) { last if /^$/; # transform real from: line back to icky style s/^From:\s+(.*) <(.*)>/From: $2 ($1)/; s/Message-Id/Message-ID/; # transform from_ line to path header; also works locally s/^From\s+(\S+)@(\S+).*/Path: $2!$1/ || s/^From\s+(\S+)[^@]*$/Path: $1\n/; print INEWS # if /^(Date|From|Subject|Path|Newsgroups|Organization|Message-ID):/i; if /^(Date|From|Subject|Path|Newsgroups|Message-ID):/i; $saw_subject |= ( $+ eq 'Subject' ); $saw_msgid |= ( $+ eq 'Message-ID' ); # $saw_newsgroup |= ( $+ eq 'Newsgroups' ); } warn "$program: didn't expect newsgroup in both headers and ARGV\n" if $newsgroup && $saw_newsgroup; die "$program: didn't get newsgroup from either headers or ARGV\n" unless $newsgroup || $saw_newsgroup; $approved = $newsgroup; $approved =~ s/\./'-'/eg; ($sec,$min,$hour,$mday,$mon,$year)=localtime(time); $madeupid = "\<$year$mon$mday.$hour$min$sec.$$\@kepler.hedland.edu.au\>"; printf INEWS "Newsgroups: %s\n", $newsgroup if $newsgroup; printf INEWS "Approved: %s\@kepler.hedland.edu.au\n", $approved; print INEWS "Subject: Untitled\n" unless $saw_subject; printf INEWS "Message-ID: %s\n", $madeupid unless $saw_msgid; printf INEWS "NNTP-Posting-Host: %s\n", $postinghost; print INEWS "Organisation: (mail2news gateway)\n"; print INEWS "\n"; print INEWS while ; # gobble rest of message close INEWS; exit $?; ______________________________________________________________________ I saved the script in /usr/local/bin (and will use this path throughout the HOWTO). Be sure to make the script executable by all, but not writable by group or other. Sendmail is picky. chmod a+x /usr/local/bin/mail2news.pl chmod go-w /usr/local/bin/mail2news.pl or chmod 555 /usr/local/bin/mail2news.pl for short. 5.4. How do I know if the script is running? I tested this script by changed my news poster from /usr/bin/rnews to /bin/cat. I then saved an e-mail send to myself in a file. Finally I ran the mail2news.pl on the saved mail and captured the output to a file. /usr/local/bin/mail2news.pl ietf.confctrl /tmp/article 5.5. What is with the Aussie? Yes, the output of your posting script should contain the e-mail address of an austrailian. My guess is that the Austrailian's address is trusted address in your news configuration (althogh I could not find it in mine) for approving moderated postings. 5.6. What do you mean $PATH? If the mail2news.pl script is not in my path, I get the error bash: mail2news.pl: command not found. You will need to either add this directory to your path PATH=$PATH:/usr/local/bin (which only works for the current login) or give an absolute path when you run the script /usr/local/bin/mail2news.pl To view your current path type echo $PATH 6. The local mailer daemon 6.1. Enabling alias files This document only describes modifying sendmail. Many people prefer qmail. Out of the box, sendmail does not come with the "pipe to program" feature enabled. Rightly so, this feature is a security risk. It enambles users to have their mail fed to a program instead of appended to a file. (Users can configure this in thier .forward file.) Please don't believe me. Test your own system first. Much of this chapter might not be needed by you. 6.2. Documentation IMHO, sendmail installs with woefully insufficient documentation. http://www.sendmail.org/ ca/email/doc/op-sh-5.html describes the syntax of sendmail.cf. I hope you don't have to use it. This is a description of the features you are trashing in sendmail.mc http://www.sendmail.org/m4/features.html Sendmail has a man page... man sendmail 6.3. Recompiling your sendmail.cf file To get the "pipe to program" stuff in the aliases file to work you need to modify the default sendmail.cf (by modifying the sendmail.mc) so that the restricted shell is not used. I suppose the proper solution would be to add the one program to the restricted shell list, but their was no man page on smrsh. Strangely, uncommenting the smrsh feature didn't work, I needed to change the shell from /usr/sbin/smrsh to /bin/bash. Yeah, this is slightly risky, but it was not an issue on my machine. Without this change I kept getting a "Service unavailable" error message in the /var/log/maillog file. The header of /etc/senmail.mc of RedHat-6 has a bug. The proper command line is... m4 /etc/sendmail.mc >/etc/sendmail.cf You need to do this when you change sendmail.mc. Hopefully, RedHat will extend the super cool Makefile idea in /etc/mail You will need to install sendmail-cf.something.rpm first. e.g. ... rpm -i sendmail-cf-8.9.3-10.i386.rpm Whenever you modify the sendmail.cf file, you should restart sendmail... /etc/rc.d/init.d/sendmail restart 6.4. Creating a mail alias Instead of creating a new user account, we will only create an alias. When modifying the /etc/aliases file, the double quotes are required. There cannot be a space between the first double quotes and the | (pipe) character, or sendmail will complain "User unknown" Add a line like ... confctrl: "| /usr/local/bin/mail2news.pl ietf.confctrl " Whenever you modify the /etc/aliases file you need to notify sendmail. sendmail -bi 6.5. Did it work? Check the /var/log/maillog to see if it worked, or for error messages. I found it useful to open up another terminal (ssh) window to watch the log with tail -f /var/log/maillog 6.6. Unprivledged If you are having trouble, and create an e-mail alias with a different (simpler) target program to test it, remember that sendmail runs the program as an unprivledged user, who probably doesn't have privledges to write anywhere except globally writable directories such as /tmp. 7. The mailing list remailer 7.1. Subscribing The method of subscribing to each e-mail list is different, although most involve sending some kind of e-mail message. Please be very careful not to send to the replicating address of the mailgroup. This is a sure way to annoy the members of the list. Most remailers (but not all) have a separate address for subscribing. In confctrl, the administration address is confctrl-request@isi.edu Some mailing lists do not let you subscribe an address you are not mailing from. In this case you will need to forge the return address of an e-mail. When you successfully subscribe you should (hopefully) see a welcome message in the newsgroup. 7.2. Forging mail Forgeing mail is easiest to do with an old Netscape client. It would accept anything for a return address. Fortunately, modern browsers require you to retreive mail successfully for an address before they will let you send. Thus you are required to fall back the tried and true method....telnet. 7.2.1. Looking up MX hosts When you speficy a web page, your browser does a DNS lookup to convert the domain name into an IP address. This is a lookup of an "A" record. (Also "CNAME" records are considered.) When sending mail a "MX" record is looked up. If this is missing a "CNAME" or "A" one is used. Thus, to forge mail you need to do an MX lookup. dig mx isi.edu will give... ; <<>> DiG 8.2 <<>> mx isi.edu ;; res options: init recurs defnam dnsrch ;; got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 4 ;; QUERY SECTION: ;; isi.edu, type = MX, class = IN ;; ANSWER SECTION: isi.edu. 1D IN MX 0 tnt.isi.edu. isi.edu. 1D IN MX 10 venera.isi.edu. ;; AUTHORITY SECTION: isi.edu. 1D IN NS venera.isi.edu. isi.edu. 1D IN NS ns.isi.edu. isi.edu. 1D IN NS east.isi.edu. ;; ADDITIONAL SECTION: tnt.isi.edu. 1D IN A 128.9.128.128 venera.isi.edu. 1D IN A 128.9.176.32 ns.isi.edu. 1D IN A 128.9.128.127 east.isi.edu. 1D IN A 38.245.76.2 ;; Total query time: 448 msec ;; FROM: fdd.com to SERVER: default -- 127.0.0.1 ;; WHEN: Sun Jul 25 15:49:32 1999 ;; MSG SIZE sent: 25 rcvd: 182 thus you would use tnt.isi.edu. 7.2.2. SMTP Mail is delivered using the Simple Mail Transport Protocol (SMTP). Like most good Internet protocols, it is ASCII based to make troubleshooting and development easier. I will not explain everything, but simply give an example. Hopefully, this is enough. The protocol is line oriented. Each email as specified in RFC822, is composed of headers and body which are separated by the first blank line (no characters, not even spaces). SMTP specifies the end of an e-mail with a line containing only a period. halyard$ telnet tnt.isi.edu 25 Trying 128.9.128.128... Connected to tnt.isi.edu. Escape character is '^]'. 220 tnt.isi.edu ESMTP Sendmail 8.8.7/8.8.6; Sun, 25 Jul 1999 14:01:25 -0700 (PDT) helo isi.edu 250 tnt.isi.edu Hello rick@node-d8e9822 [216.233.8.34] (may be forged), pleased to meet you mail from: 250 ... Sender ok rcpt to: 250 ... Recipient ok data 354 Enter mail, end with "." on a line by itself From: Subject: help help