dnswl.py, a Postfix delegation service

  1. Overview
  2. Requirements
  3. Download
  4. Installation
  5. Policy service configuration
  6. Contact

Overview

What is dnswl.py and why would you want to use it?

There are several ways on how to include DNSBL, so called DNS black (-hole) lists) in a MTA like Postfix. Spam is a real big problem these days. All kind of development was focused on how to build rules, to reject unwanted mail (bulk email). At the moment, you can hard blacklist MTAs by using some reject_rbl_client and reject_rhsbl_client rules in smtpd_recipient_restrictions for example, or use a policy service like policyd-weight (which the author likes, too, very much). But there are some interesting facts now, that made the author think different now, on how the perspective of a future mail server might look like.

To understand this, you need to know that Postfix currently gets a new process calles postscreen. This service can (currently) use a sub-service dnsbl, to lookup DNSBL services. This makes the DNSBL part of policyd-weight obsolete. Furthermore, many connections are dropped, before a “real” smtpd process is spawned.

Postscreen does not have whitelisting support, because whitelisting can really be done easyly in a policy service. Maybe it would be possible to include this functionallity into policyd-weight, but the author does not like Perl very much (jokingly).

Why would somebody like to use whitelists? Well, think about IPv6, which is close for production usage. With IPv6, it will really be hard to reject spammers based on blacklists anymore, because they can switch simply to a different IP. So one trend that can be seen at the moment is, that a good MTA reputation could be the key for the future to run proper MTA setups.

What can dnswl.py do now? It can query a set of DNS lists for client addresses that are whitelisted. Normally only “good” MTAs should go into whitelists. If Postfix gets in contact with a whitelisted MTA, it can use dnswl.py to bypass the connecting MTA through some harder checks that could normally make the connection fail (should not be the case!). At the other side, it does not make sense to whitelist the MTA only at the Postfix side, if an amavisd-new process would reject a mail (think of false positives). Therefor dnswl.py also makes use of the soft scoring mechanism in amavisd-new. It populates SQL lookup tables and inserts a boost score, which is freely be configurable for each DNS service, you bind into dnswl.cfg (It also cleans up old records for servers that formerly were whitelisted, but got removed).

If reputation really will be the key for next years MTAs, then it might happen that MTAs will be much more restrictive to “unknown” sending MTAs, which could be a problem of getting false positives. So whitelists could be getting more important and maybe any bigger company that really is dependent on uncomplicated mail transfers, will need to make usage of these services.

But what the future really is going to be, who knows. This project is an attempt of trying to find a solution for the current situation.

How does it work?

    +---------+  1.     +--------------+     +--------------+
25  |         |-------->|              |  2. |              |
--->| Postfix |      4. |   dnswl.py   |---->|  DNS lookup  |
    |         |<--------|              |     |              |
    +---------+ OK or   +--------------+     +--------------+
         |      DUNNO               |
         |      (DEFER_IF_REJECT)   | 3.
       p |                          v
       r |            +--------------------+
       e |            |                    |
         |            |  SQL Server        |    INSERT and DELETE
       q |            |  - prepare DNS-WL  |
       u | 5.         |    soft scoring    |
       e |            |                    |
       u |            +--------------------+
       e |                      ^
         |                      |
         +----+         +-------+
              |         | 6.
              |         |
              v         |
           +---------------+        +-----------+
           |               | 10025  |           |
           |  amavisd-new  |------->|  Postfix  |
           |               | 7.     |           |
           +---------------+        +-----------+

If the DNS is temporarily unavailable, the result is DEFER_IF_REJECT. Make sure, you run a local caching DNS recolver. Dnswl.py only is abale to cache results for records that exist in DNS, not for those that are unavailable (NXDOMAIN)

Requirements

Under Ubuntu (Lucid), simply install the following packages:

  • python-dnspython
  • python-decorator

If you like to have SQL support, you also need

  • python-sqlalchemy
  • python-elixir
  • python-mysqldb — if using MySQL
  • python-psycopg2 — if using PostgreSQL

Download

dnswl.py (version 1.0-stable) and dnswl.cfg

Installation

Put dnswl.py to /usr/local/sbin and dnswl.cfg to /etc/dnswl.cfg

Create a user:

adduser --system --group dnswl

If you like to use upstart, you can put this content into /etc/init/dnswl.conf:

# dnswl - DNS whitelist service (policy-delegation-service for postfix)
#
# Ths service checks against several whitelist DNS services for postfix

description     "DNSWL service"

start on filesystem
stop on runlevel S

expect daemon

exec /usr/local/sbin/dnswl.py

If you want to use SQL, please prepare amavisd-new accordingly (see the SQL documentation; you can use the default tables from the readme file, with the addition shown below). You will need to create a user that is allowed to do SELECT, DELETE and INSERT into the amavisd-new lookup table. The amavisd-new settings could look similar to these. Be careful! The user used by amavisd-new is a different MySQL user, from the one, you just configured to use with dnswl.py:

[...]
$sql_allow_8bit_address = 1;
$timestamp_fmt_mysql = 1;
@storage_sql_dsn =
  ( ['DBI:mysql:database=amavis_rw;host=yoursqlserver;port=3306', 'amavisrw', '********'] );

@lookup_sql_dsn =
  ( ['DBI:mysql:database=amavis_ro;host=yoursqlserver;port=3306', 'amavisro', '********'] );
[...]

The author is using LDAP for lookups as well, so he has a dummy policy preset, with all values unset (default). If you make use of SQL policy presets, see hints below (dnswl.cfg).

Policy service configuration

dnswl.cfg

Currently the main configuration is found in /etc/dnswl.cfg, which is in standard ini file format. Similar to MySQL or Samba. It consists of a global section, which is mandatory. Most important are the settings bind_address or bind_address6, if using IPv6 and a port.

uid and gid default to dnswl/dnswl, but of course you can specify whatever user you like to use. If you followed the instructions above, the defaults are fine.

In the sql section, you can define, if you like the sql manipulation of amavisd-new. This is done with the use_sql boolean value. The verbose_sql is for the debug mode. In this mode, dnswl.py runs in foreground and displays all sql commands.

There are some special parameters, policy_priority, mailaddr_priority and policy_preset, which you might need to modify to match your current setup. The goal of setting priorities is that you can override these values with custom settings. So chosing a lower priority might be a good idea.

Sample output of the table policy (remember, the author uses LDAP for lookups):

MariaDB [amavis_ro]> select * from policy\G
*************************** 1. row ***************************
                          id: 1
                 policy_name: default
                 virus_lover: N
                  spam_lover: N
          banned_files_lover: N
            bad_header_lover: N
         bypass_virus_checks: N
          bypass_spam_checks: N
        bypass_banned_checks: N
        bypass_header_checks: N
          spam_modifies_subj: N
         virus_quarantine_to: NULL
          spam_quarantine_to: NULL
        banned_quarantine_to: NULL
    bad_header_quarantine_to: NULL
         clean_quarantine_to: NULL
         other_quarantine_to: NULL
              spam_tag_level: NULL
             spam_tag2_level: NULL
             spam_kill_level: NULL
       spam_dsn_cutoff_level: NULL
spam_quarantine_cutoff_level: NULL
        addr_extension_virus: NULL
         addr_extension_spam: NULL
       addr_extension_banned: NULL
   addr_extension_bad_header: NULL
              warnvirusrecip: NULL
             warnbannedrecip: NULL
               warnbadhrecip: NULL
              newvirus_admin: NULL
                 virus_admin: NULL
                banned_admin: NULL
            bad_header_admin: NULL
                  spam_admin: NULL
            spam_subject_tag: NULL
           spam_subject_tag2: NULL
          message_size_limit: NULL
            banned_rulenames: NULL

MariaDB [amavis_ro]> select * from users;
+----+----------+-----------+----------------------------+----------+-------+
| id | priority | policy_id | email                      | fullname | local |
+----+----------+-----------+----------------------------+----------+-------+
|  1 |        7 |         1 | christian@roessner-net.com |          | Y     |
+----+----------+-----------+----------------------------+----------+-------+

MariaDB [amavis_ro]> select * from mailaddr limit 1;
+----+----------+--------------------+
| id | priority | email              |
+----+----------+--------------------+
|  1 |        7 | foo@example.org    |
+----+----------+--------------------+

The column policy_id from table users is what the parameter polcy_preset is in dnswl.cfg. Priority maps to policy_priority; The parameter priority from table users is mapped to the parameter mailaddr_priority.

If you decide to use SQL, you need to adjust the dsn setting. dnswl.py makes use of an object relational mapper, which does have support for manny SQL databases like MySQL, PostgreSQL and many others as well. See the configuration for details.

The sections swl-sites and dwl-sites are optional, but of course you will need to set up some services there, else the functionality of dnswl.py is useless.

For each service, you also can (optionaly) define a list of IPs that their DNS servers return in response to queries and a boost value that is used in the wblist, individually for each DNS service, too, optional. If ips is not set, it defaults to 127. A missing boost setting defaults to -10.0.

You can put a suffix behind each IP by putting a slash symbol behind it and specify an individual boost value. This superseeds the global site specific boost level.

See further comments in the sample configuration above. And below you can see some examples of specifying lists:

...
[swl-sites]
sites = swl.spamhaus.org, list.dnswl.org

# This example does not define a side-wide boost value. So it is -10.0 by default
[swl.spamhaus.org]
ips = 127.0.2.2, 127.0.2.3

# You can write lists like in Postfix. But do not miss the comma signs.
# If you like to, you may leave some space between an IP and the individual boost value
[list.dnswl.org]
ips =
        127,
        127.0.2.1 / -1.0,
        127.0.2.2 / -10.0,
        127.0.2.3 / -100.0,
        127.0.3.1 / -1.0,
        127.0.3.2 / -10.0,
        127.0.3.3 / -100.0,
        127.0.4.1 / -1.0,
        127.0.4.2 / -10.0,
        127.0.4.3 / -100.0,
        127.0.5.1 / -1.0,
        127.0.5.2 / -10.0,
        127.0.5.3 / -100.0,
        127.0.6.1 / -1.0,
        127.0.6.2 / -10.0,
        127.0.6.3 / -100.0,
        127.0.7.1 / -1.0,
        127.0.7.2 / -10.0,
        127.0.7.3 / -100.0,
        127.0.8.1 / -1.0,
        127.0.8.2 / -10.0,
        127.0.8.3 / -100.0,
        127.0.9.1 / -1.0,
        127.0.9.2 / -10.0,
        127.0.9.3 / -100.0,
        127.0.10.1 / -1.0,
        127.0.10.2 / -10.0,
        127.0.10.3 / -100.0,
        127.0.11.1 / -1.0,
        127.0.11.2 / -10.0,
        127.0.11.3 / -100.0,
        127.0.12.1 / -1.0,
        127.0.12.2 / -10.0,
        127.0.12.3 / -100.0,
        127.0.13.1 / -1.0,
        127.0.13.2 / -10.0,
        127.0.13.3 / -100.0,
        127.0.14.1 / -1.0,
        127.0.14.2 / -10.0,
        127.0.14.3 / -100.0,
        127.0.15.1 / -1.0,
        127.0.15.2 / -10.0,
        127.0.15.3 / -100.0
boost = -0.1
...

Postfix

dnswl.py is included as a policy service in Postfix. Configure it at least in /etc/postfix/main.cf like shown below:

smtpd_recipient_restrictions =
    ...
    reject_unauth_destination,
    ...
    check_policy_service inet:127.0.0.1:12526

Some suggestions:

Do not put it as first rule under the smtpd_restrictions. Let’s see, how the auther has configured his MTA:

...
# Global variables
default_database_type               = btree
map                                 = ${config_directory}/maps
mapidx                              = ${default_database_type}:${map}
ldap                                = proxy:ldap:${config_directory}/ldap
...
authenticated_smtpd_recipient_restrictions =
    reject_non_fqdn_recipient,
    reject_non_fqdn_sender,
    reject_unknown_recipient_domain,
    reject_unknown_sender_domain,
    permit_mynetworks,
    permit_sasl_authenticated,
    reject
...
smtpd_recipient_restrictions =
    reject_non_fqdn_recipient,
    reject_non_fqdn_sender,
    reject_unknown_recipient_domain,
    reject_unknown_sender_domain,
    reject_unlisted_recipient,
    reject_unauth_destination,
    reject_invalid_helo_hostname,
    reject_non_fqdn_helo_hostname,
    check_sender_access ${mapidx}/sender_access,
    check_client_access pcre:${map}/client_access.pcre,
    check_client_access cidr:${map}/client_access.cidr,
    check_sender_access ${mapidx}/backscatter,
    check_helo_access pcre:${map}/helo_access.pcre,
    check_policy_service inet:[::1]:12526
    check_client_access pcre:${map}/dynamic_ip.pcre,
    reject_unknown_reverse_client_hostname,
    reject_unknown_helo_hostname,
    check_sender_ns_access ${mapidx}/bogus_dns,
    check_recipient_access pcre:${map}/roleaccount_exceptions.pcre,
    check_helo_access ${ldap}/helo_access.cf,
    check_client_access pcre:${map}/greylist.pcre
...

The sample shows a pure MSA/MTA setup. That means that mail traffic arriving on port 25 is _not_ used for submitting client mail. Therefor use the submission service, shown here in /etc/postfix/master.cf:

...
XX.XX.XX.XX:submission inet n - - - - smtpd
    -o myhostname=myabe_another_servername.example.com
    -o content_filter=lmtp-amavis:[::1]:10026
    -o smtpd_tls_cert_file=/ca/newcert.pem
    -o smtpd_tls_key_file=/ca/newkey.pem
    -o smtpd_tls_security_level=encrypt
    -o smtpd_recipient_restrictions=${authenticated_smtpd_recipient_restrictions}
    -o receive_override_options=no_header_body_checks
...

What you can not see here is, where DNS-blackhole lists are used. This is done in Postfix/poscreen service, which is not part of the current stable release, so you might want to use policyd-weight as another policy service inside the smtpd_recipient_restrictions.

Dnswl.py is not intented to whitelist blindly. Every MTA has to be configured with some restrictions and follow some rules that also a MTA with good or brilliant reputation has to follow and must be checked against.

The idea behind dnswl.py now is, to lower the spam scores in amavisd-new. That means that a MTA with unknown or bad reputation can hardly send spam or forged mail. The low score in amavisd-new would radically reject mail (if using smtpd_proxy_filter in Postfix; what the author suggests).

Sample amavisd-new values:

...
$sa_tag_level_deflt             = undef;
$sa_tag2_level_deflt            = 3.63;
$sa_kill_level_deflt            = 3.63;
$sa_dsn_cutoff_level            = 10.0;
$sa_quarantine_cutoff_level     = 15.0;
...

Please! This is not a howto that should be copied and pasted! The author repeats: Please do not copy and paste. You do not know anything about the polcies that this server follows and your setup might differ. But you can use it to understand a use case for dnswl.py.

amavisd-new

For amavisd-new, you need to create the wblist table as folloed:

CREATE TABLE `wblist` (
  `rid` INT(10) UNSIGNED NOT NULL,
  `sid` INT(10) UNSIGNED NOT NULL,
  `wb` VARCHAR(10) CHARACTER SET latin1 NOT NULL,
  `auto` CHAR(1) DEFAULT 'Y',
  PRIMARY KEY (`rid`,`sid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

The only difference to the table shown in the amavisd readme file is the “auto” field. It is used to indicate that rows with a ‘Y’ char set, have been created automatically. This way, you also can have user dependent rows, which might overlap.

If you followed the guidelines in the RELEASE_NOTES of amavisd (upcoming 2.7.0), you will find that MySQL and PostgreSQL both have an optional field “local” in the table “users”. Currently dnswl.py depends on the existens of this field. Maybe at a later time it will find out automatically, if the field exists or not.

Contact

https://lists.roessner-net.de/cgi-bin/mailman/listinfo/dnswl-users

Anything about developing dnswl.py goes here:

https://lists.roessner-net.de/cgi-bin/mailman/listinfo/dnswl-devel


-- Download dnswl.py, a Postfix delegation service als PDF --