1. Introduction

mapSoN is a spam filter that uses a pretty unique approach to keep unsolicited commercial e-mail out of your mailbox. Rather than using a set of configured "bad words", a list of "know spammers", or complicated scoring mechanisms to determine what is spam and what is not, it relies on "known senders" -- or rather "unknown senders".

Every time you receive an e-mail, mapSoN will look-up the sender's e-mail address in a small database file and check whether that address is in there. If it is, the mail is delivered to your mailbox, but if it is not, the e-mail will be stored in a spool directory in your home, using a cryptographic cookie as the filename. Then mapSoN will send a so called request for confirmation to the sender's address, asking him to please confirm his addresses validity by replying and sending the cryptograpic cookie back. When mapSoN receives a mail with such a cookie in it, it will move the corresponding mail from the spool directory to your mailbox and add the sender's address in the mail to the database.

This approach is based on the fact that spammers usually fake the sender address of the spam mail. (In fact, they have to, because sending unsolicited advertisement via e-mail is illegal in most countries.) But because their sender address is invalid, they will never see the request for confirmation, they will never reply, and their spam will sit in that spool file until hell freezes over or an apropriate cron job deletes it. Using this heuristic, mapSoN catches way above 95% of all spam mail I receive.

In order to avoid annoying more "real" people, who are trying to contact you, than necessary, you can import the addresses from your mail archive into the mapSoN database. Furthermore, you can set mapSoN up in a way that will let any mail pass automatically, that is a reply to a mail or a news posting of yours: If you sent someone an e-mail and he replies back, mapSoN won't bother him. It would be pretty inpolite, if it did.

This illustration will probably help not all and only add to the confusion, but I made it, and now I have to include it here -- helpful or not!

2. How effective is mapSoN

You may wonder how effective mapSoN is -- and rightfully so. Of course, being the principal author of this tool, I am biased. But I tried my best to conduct an objective study, determining how many spam mails have been caught, how wany mapSoN did not catch, and how many mails were delayed that weren't spam. The numbers presented below are somewhat skewed, because the logfile I analyzed includes the various test mails I piped into mapSoN in order to test and to debug it, but if at all, they tend to make the result look worse because I did not test the case that a mail comes in and is approved, I tested the case that a mail is deferred.

Anyway, I activated mapSoN for my private mail account on January 11th, 2002. This analysis was made on April 10th the same year, so we have a test period of 90 days -- almost three months. Here are the numbers:

How did I get those numbers? First of all, I counted all logfile entries coming from sendmail, which contained the strings "mailer=local" and "to=<simons". This I assumed to be the number of total mails received. On those entries, I counted how many of those contained the string "to=<simons+" -- what should largely be the mails I receive via mailing lists because I use procmail's argument feature when I subscribe to mailing lists.

All those mails, addressed to "simons+something" bypassed mapSoN automatically. Furthermore, all mails that contained certain headers like In-Reply-To or References bypassed mapSoN entirely because I assume those to be replies to messages of mine. This explains why the number of mails that mapSoN actually processed is much smaller than the number of total mails received: Apparently only 20% of all incoming e-mail were processed by mapSoN at all!

Then I counted the entries in mapSoN's logfile that sayed "passed", which turned out to be 25% of all mails mapSoN saw. The mails that mapSoN did not let through amount to 63% of all mails processed by mapSoN, and I determined those by counting the logfile entries that said "Spooling e-mail".

Of those deferred e-mails, only 6% were confirmed later! So then I waded through the spool directory and moved all "regular" e-mails to a separate folder and counted them: It turned out that regular 14 mails were not confirmed by the sender, that's 0.14% of all deferred mails and 0.0018% of the total number of mails received.

When I looked at those mails in detail, it turned out that 3 of the 14 mails lying in the spool "unwarrantedly" actually had been acknowledged, but that the other person was too dumb to reply correctly: Two of the three sent a new e-mail, which did of course not contain the cookie, and the third person replied to the request for confirmation, but erased the cookie from the mail manually. It's no wonder that the mails they sent me turned out to be of the kind that I don't want anyway.

The remaining 11 mails that were not delivered to me but were not spam either, were some kind of replies I got from Internet sites like amazon.com, other customer service stuff and automated reminders. No personal e-mail. They were delayed because they were sent with incorrect sender addresses that bounced when mapSoN sent the request for confirmation back or that were apparently not read by the other end at alll.

Once I noticed the problem, I got it fixed quickly by using "user+something" addresses at sites like amazon.com, too, so that their mails bypass mapSoN to begin with.

Of course there was a certain amount of spam that got past mapSoN one way or the other. Some used addresses that actually were in my database because I had imported them from my mail archives when I set mapSoN up. Some others were routed past mapSoN by my procmail configuration because they looked like bounces or postings coming from mailing lists. Unfortunately, I cannot determine any exact number without wading through my mail archive manually, and I honestly don't want to do that. After all, the whole point of writing mapSoN was that I do not have to see spam!

To summarize: In the 90 days of testing, mapSoN caught 862 definite spam mails: That's about 10 per day. It reduced the amount of spam in my mailbox to one or two mails once a week. At the same time, only 0.008 percent of all e-mail I receive had to be acknowledged, what I think is an acceptable level of inconvenience for my communication partners, especially given the fact that obviously they were contacting me, not the other way round.

3. Building mapSoN

Compiling the software should be pretty straight-forward; all you'll have to do is the usual routine:

./configure
make
make install

The last step may require super-user privileges, so have the root password ready if you're going for a system-wide installation.

Be adviced that mapSoN needs a fairly recent C++ compiler because it makes full use of the new ISO C++ language features. If you're using the GNU C Compiler version 2.95 or later, you won't have any problems. But other compilers are known not to be ISO C++ compatible. If you're having trouble, send me an e-mail and I'll see what I can do.

The configure script is a standard GNU Autoconf script, which supports all the usual options. If you are not familiar with these scripts, please refer to the "Running configure Scripts" section of the Autoconf user manual, which is available at:

The following list will show only those options that are specific to mapSoN.

Any of the paths you configure here are only used as defaults. You can override them at run-time in the configuration file mapSoN reads at startup.

If your using a fairly recent gcc version to compile this, you might want to try

CXXFLAGS=-frepo LDFLAGS=-frepo ./configure

to configure the build. This will result in an about 30% smaller binary because unused template instances are optimized out.

On some platforms, largefile support doesn't work yet. If you're getting compiler errors, give the configure script the option --disable-largefile and try again.

4. How to Activate mapSoN

Assuming, you have built and installed the mapSoN package sucessfully, you must do two things to activate it: Create the directory ".mapson" in your home directory and tell your Mail Transport Agent to pipe incoming local mail into mapSoN rather than to deliver it to the mailbox directly.

The first step should be manageable without further instructions, but installing mapSoN as local mailer is non-trivial. The rest of this chapter is divided into separate sections that will describe the various possible setups. How mapSoN must be installed depends entirely on the Mail Transport Agent you use, so if your configuration is not discussed in this manual, please consult your MTA's user manual instead. And if you found out how to do it, please write a short paragraph about it and let me know so that I can include it in the next version!

"|exec /usr/local/bin/mapson"

ARGUMENT="$1"

# Have mapSoN accept anything that is a reply to a
# message of mine.
#
:0 w
* ^(In-Reply-To|References|Message-Id):.*example.org
| /usr/local/bin/mapson --accept

# Confirmation mails go into mapSoN.
#
:0
* ARGUMENT ?? [a-f0-9]…repeat 32 times…[a-f0-9]
| /usr/local/bin/mapson --cookie $ARGUMENT

# Forward the mail into mapSoN for approval unless
#   - it has an argument,
#   - is a bounce,
#   - comes from a mailing list, or
#   - is an automatically generated mail.
#
:0 w
* !ARGUMENT ?? ..*
* !^FROM_DAEMON
* !^Precedence: (list|bulk|junk)
* !^Auto-Submitted:
| /usr/local/bin/mapson

:0
* ARGUMENT ?? cat-lovers
/var/spool/mail/user2

From: user+${MD5HASH}@example.com (Real Name's Anti-Spam-Tool)

poll mailserver.example.org mda "/usr/local/bin/procmail -d username"

5. Command Line Syntax

mapSoN understans several optional parameters on the command line, which allow you to override the compiled-in default or the settings in the config file. The standard Unix synopsis line is:

mapson [-h | --help] [--version] [-d | --debug] [-a | --accept] [--cookie cookie] [-c config | --config-file config] [mail...]

Here is a list of all options together with a short description of what the respective option does:

6. The mapSoN Configuration File

At startup, mapSoN will try to read its configuration file at ${HOME}/.mapson/config first of all. If that file does not exist, mapSoN falls back to the system-wide file /usr/local/etc/mapson.config. This means that if mapSoN has been installed well, you don't need a configuration file of your own. It also means, that you may have a configuration file of your own nonetheless.

The config file is read line by line. Empty lines are ignored, as are lines starting with the commend delimiter "#". Anything else is supposed to start with one of the keywords listed below, followed by one or more whitespace characters, followed by the actual data part.

The data part may contain environment variables, which you can use to have one configuration file fit all of your users! See the sample configuration file installed at /usr/local/share/mapson/reqmail.template-sample for such an example.

Valid keywords in the configuration file are:

In order to make the contents of the configuration file as independent from the system's directory structure as possible, mapSoN provides a set of environment variables, which are guaranteed to be defined. You can use them anywhere in the data part of a configuration directive, and you can use the usual manipulations on them.

Environment variables are looked-up case-sensitively, so $home is not the same thing as $HOME. This behavior is different in the request-for-confirmation template, where you can spell the variables upper- or lower-case as you wish. That's because the variables there are not coming from the environmnet, but are mapSoN's internal variables. So be sure not to confuse that, because an undefined variable in this file will cause mapSoN to abort with an error.

Here is the complete list:

7. The Request-for-Confirmation File

When mapSoN issues a request for confirmation, it will try to load the template file containing the text to be used for this purpose. Unless configured otherwise in the configuration file (see Section 6), the first path to look is ${HOME}/.mapson/reqmail.template; ${HOME} meaning the home directory of the user under which's id mapSoN is running under.

If that file does not exist, mapSoN will fall back to the system-wide file at /usr/local/share/mapson/reqmail.template. If this file doesn't exist either, mapSoN will abort with an error.

The request-for-configuration template file is supposed to contain a complete RFC822 message, including headers and everything. The actual request-for-confirmation mail is created by loading the template and expanding the variables contained in it. The result is then piped into the command, you have configured to use to access the Mail Transport Agent.

Here is an example of a request-for-confirmation template you might use:

From: username@example.com (Real Name's Anti-Spam-Tool)
To: ${ENVELOPE:-${RETURN_PATH:-${SENDER}}}
Subject: please confirm [${MD5HASH}]
Precedence: junk
Auto-Submitted: auto-generated
References: $MESSAGEID
In-Reply-To: $MESSAGEID

This is an automated request for confirmation in order to make
sure that the message quote below was actually sent by you. You
don't wanna know the details, trust me. Just press <reply>
and send me a mail back without changing that cookie in the subject
line, that's it. You will never have to do that again -- sorry for
the inconvenience!

Your mail was:

[ | ${HEADER[#]}] |
[${BODY[#]:+ | }${BODY[#]}]{0,5} | \[...\]

mapSoN will replace the variables you see in this example by the actual values from the incoming mail and deliver the confirmation request. Don't panic, there's a pretty good template included in the distribution (see Appendix B) that you can use, you don't have to worry about the variable stuff too much if you don't want to. For those who want to … Here is the complete list of variables provided by mapSoN for this file:

In addition to those, the following arrays are provided:

In addition to those, you can access any environment variable available at run-time. The template file included in this distribution, for example, will use ${USER} in order to make the template file independent of the user who's actually running mapSoN. Don't be to daring, though, not every environment variable you can see in your shell will be set when sendmail, procmail, or whoever calls mapSoN! The only environment variables that are guaranteed to be available are those list in Section 6.

One more thing: The variables listed explicitely in this section can be access case-insensitively. ${bOdY} is the same as ${boDY}, because these are variables provided by mapSoN internally. But environment variables like ${USER} must be accessed in upper-case!

8. Variable-Expression Magic

Throughout mapSoN, the user may specify variables in the text files in order to have their actual contents inserted at the apropriate location. This is a functionality provided by libvarexp. Hence, this section has been inserted verbatim from libvarexp's documentation. Please don't worry if the documentation says things like "implementation defined", etc. Just read about the expressions the library provides you with and how you can use them. Anything you need to know is included in this document.

If you're interested in incorporating libvarexp into programs of your own, though, check out the copy available in the libvarexp directory in the mapSoN distribution or take a look at libvarexp's homepage for further details.

9. Expiring the Mail Spool

to be written

10. Importing Addresses From a Mail Archive

Of course it would be unpolite to have mapSoN send out requests for confirmation to people who you have been communicating with you for months or years, just because you installed a new tool. If you were wise enough to archive your old e-mails, there's a simple way to avoid that happening: Import their addresses into mapSoN's database.

Unfortunately, most mail readers archive old mails in one single file: Each new mail is just appended at the end, just like the mailbox format itself. Currently, mapSoN can not deal with those files. The current version can import addresses only from an archive where a each mail is stored in a separate file, like the archives maintained by the Gnus software, that is part of Emacs, for example.

In this case, though, it's simple enough: Just start mapSoN and give it the file names as parameters on the command line. You might want to enable debugging by giving it the -d flag, so that you can see what's going on:

simons@peti:~/mail-archive$ mapson -d *
1:
12:
    simons@peti.gmd.de................................. new
16:
    simons@peti.gmd.de................................. known
17:
    th@example.com..................................... new
    th@example.com..................................... known
    th@example.com..................................... known
19:
    bscw@cscwmail.example.org.......................... new
    manfred.bogen@gmd.example.org...................... new
53:
    pakhomenko@example.com............................. new
    pakhomenko@example.com............................. known

Depending on the size of your mail archive, this may take a while, but usually mapSoN is pretty quick.

Once that's finished, you'll have a pretty good database to start with, and it's highly unlikely that someone, who has been in contact with you before, will be bothered with an request-for-confirmation mail.

11. What To Do If Something Does Not Work

There's a chance that mapSoN isn't working the way you expect it -- especially in the current unfinished state of the program. Here's a short description of how you can probably figure what's going wrong. The magic word is "log file". mapSoN logs pretty much everything it does to a file, which is per default located at $HOME/.mapson/log.

A typical set of messages found there may look like this:

debug: mapSoN verion 2.0-beta-2 starting up
debug: My configuration:
debug:     Mailbox            = '/var/spool/mail/'
debug:     ConfigFile         = '/home/user/.mapson/config'
debug:     SpoolDir           = '/home/user/.mapson/spool'
debug:     AddressDB          = '/home/user/.mapson/address-db'
debug:     ReqConfirmTemplate = '/home/user/.mapson/reqmail.template: \
    /usr/local/share/mapson/reqmail.template'
debug:     MTA                = '/usr/sbin/sendmail '-f<>' -i -t'
debug:     StrictRFCParser    = 'false'
debug:     PassIncorrectMails = 'true'
debug:     RuntimeErrorRC     = '75'
debug:     SyntaxErrorRC      = '65'
debug:     Debug              = 'true'
error: Runtime error while processing mail 'no-message-id': \
    Can't open address db '/home/user/.mapson/address-db' \
    for reading: No such file or directory

Please note that the backslashes in this example are not actually there, they just denote added line breaks for the layout. In the real file, these split lines are just one one long line.

If you find that your copy of mapSoN does not log the proceedings in this amount of detail, set the Debug directive in the configuration file to yes or add the -d parameter to the command line when calling mapSoN.

By looking at the log file, you can see what exactly mapSoN is doing and why it's doing it. In the example shown above, it fails because of a file permission error.

Of course there are some reasons that may cause mapSoN to behave in a way different from what you execpted that are not directly connected to the mapSoN program itself. Here's a list of popular mistakes:

• Check whether the mailbox file mapSoN uses to deliver passed mails is correct! If it is not, you obviously won't see anything.

• Check whether mapSoN actually sees the incoming mails it is supposed to. Especially when you are using procmail to filter incoming e-mail, make sure that the confirmation mails are passed to mapSoN. You can debug what procmail is doing by adding the lines

  VERBOSE=on
  LOGFILE=$HOME/procmail.log

  to your .procmailrc file. Then look at procmail's log file.