SAGATOR

(c) 2003-2004 Jan ONDREJ (SAL) <ondrejj(at)salstar.sk>

 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

Contents
  1. Introduction
    1.1. Features
    1.2 Requirements
    1.3 Tests
  2. Installation
    2.1 Getting started
    2.2 Creating chrooted environment
    2.3 Upgrading from previous versions
  3. Sagator configuration
    3.1 Service configuration
    3.2 Scanner configuration
  4. Detailed scanner configuration
  5. SMTPd configuration
    5.1 Postfix configuration
    5.2 Sendmail configuration
    5.3 Another smtpd configuration
  6. Antivir/Antispam installation
    6.1 ClamAV installation
  7. Staring sagator
  8. Testing
  9. Scanning mailboxes
  10. Retunrning badly quarantined mail to mailq/user mailbox
  11. Removing old files from quarantine
  12. Done

1. Introduction

This program is an email antivirus/antispam gateway. It is an interface
to the postfix or sendmail (or any other smtpd), which runs antivirus
and/or spamchecker. Its modular architecture can use any
combination of antivirus/spamchecker according to configuration.

It currently supports clamav, nod32d, AVGlinux, sophie, trophie,
symantec antivirus scan engine (via ICAP), Bitdefender,
spamassassin, bogofilter and QuickSpamFilter.
It has some internal checkers (string_scanner and regexp_scanner).
Sagator can parse MIME mails and decompress archives, if it is configured so.

---------------------------------------------------------------------

1.1. Features

  - simple chroot support
  - modular antivirus/spamchecker support
     - currently supports only clamav, nod, sophie, trophie, AVGlinux,
         ICAP protocol (Symantec Antivirus)
     - any other antivirus can be implemented
     - spamaassassin support
  - modular architecture
     - you can attach an intrascanner to another intrascanner or realscanner
     - you can combine intrascanners
     - you can combine realscanners
  - you can use simple scanners to determine files which you don't need:
     - mime_ext - define a virus by mime extension
     - file_type - define a virus by file type (content is checked)
     - max_file_size - define max. attachment or max. file size
     - string_scan - define a virus by string
     - regexp_scan - define a virus by regular expression
  - you don't need any perl modules or any other modules, only python2
  - you can return any quarantined mail to mailq/user mailbox

---------------------------------------------------------------------

1.2 Requirements

  - python2
  - clamav, spamassassin, ...
  - SMTP daemon (one from following list)
    - postfix 1.1.11 (or may be later or newer)
       You can use any other SMTPd with special configuration.
    - sendmail with milter and python milters

---------------------------------------------------------------------

1.3 Tests
  Hardware & Software:
    Athlon 2000+, 
    Fedora Core 2
    clamav 0.80rc4 (used as libclam)
  Pattern:
    90000 emails processed in 312 minutes
    speed ~= 4.8 emails/second

---------------------------------------------------------------------

2. Installation

2.1 Getting started

  Unpack the package. Go into main directory and run:
    make
    make install

2.2 Creating chrooted environment.

  Create user vscan:
    useradd vscan -M -s /sbin/nologin

  Use script:
    ./scripts/mkchroot.sh /some/where/new/root/
The directory can be for example: "/var/spool/vscan".
This script creates a small copy of your files, which are required
for antivirus, unpacker, ...

2.3 Upgrading from previous versions

  Upgrade information is available only for sagator 0.5.0 and higher.
In this version you need to add a SRV array in sagator's configuration.
If you are using COLLECTOR_SERVER and BIND_ADDR variables, you need to add:
  SRV=[
    collector(COLLECTOR_SERVER[0],COLLECTOR_SERVER[1]),
    smtpd(SCANNERS,BIND_ADDR[0],BIND_ADDR[1])
  ]

3. Sagator configuration

3.1 Service configuration

  Services are subprograms to communicate with an SMTP daemon or with other
client. For more help see README.services. Use SRV variable in config.py.

  SRV varialble is a python array, defined as:

    SRV=[ service1, service2, ... ]

  Services are defined in separate files, you must include it into
config.py. You can do it with this command:
                                                                                
  from srv import *

Examples:

  a) smtpd service (good for postfix and useable for any smtpd) with
     combination for collector (for statistics):
       SRV=[
         collector('0.0.0.0',28),
         smtpd(SCANNERS,'127.0.0.1',27)
       ]

  b) milter service (for sendmail) with combination with statistics
     collector:
       SRV=[
         collector('0.0.0.0',28),
         milter(SCANNERS,"sagator","inet:3333@127.0.0.1")
       ]

3.2 Scanner configuration

  Edit config.py. There are some parameters, described in config.py file.
Scanner configuration:
  Use SCANNERS variable in config.py. You can use more scanners.
If one of them found a virus, scanning is stopped. If one of them
fails, an error is raised.

  SCANNERS variable is a python array, defined as:

    SCANNERS=[ scanner1, scanner2, ... ]

  Scanners are defined in separate files, you must include it into
config.py. You can do it with:

  from scanners import *

Some examples:
  a) Use only clamscan binary to scan for viruses:
       SCANNERS=[
         clamav(['/usr/bin/clamscan','--stdout','--infected',
                 '--disable-summary','-r','--mbox'])
       ]

  b) Adding spamassassin tester:
       SCANNERS=SCANNERS+[ spamassassind(['localhost',783]) ]

  c) If you want to add some special stuff, like scan for
     names of attachments, and if you don't want any attachments,
     use mime_ext scanner (YOU NEED PYTHON 2.2 FOR THIS@):
       EXECUTABLES='(\.exe|\.COM|\.com|\.COM|\.pif|\.PIF|\.lnk|\.LNK|\.scr|\.SCR)'
       SCANNERS=[ mime_ext(EXECUTABLES) ]

There are also subscanners ... they are scanners, which parses emails
and runs another scanner. For example there are parsemail and
decompress scanners.

Example for parsemail:
  I wan't to test parsed files by clamd. Clamd can parse mail,
but I don't want to use its scanner now.

  parsemail(clamd(['localhost',3310]))

4. Detailed scanner configuration

There are three types of scanners:
  - streamscanner
  - filescanner
  - combination (filescanner+streamscanner)

Scanners are also divided into:
  - realscanners
     These scanners are real scanners, which scans the stream/file
     and then returns its status.
  - interscanners
     These scanners are an interface between main scanning engine
     and realscanners. You also can attach interscanner to another
     interscanner.

By default streamscanner is used. If you wan't to use filescanner,
there is a interscanner named stream2file().

If you need to know, which scanner is which, look at this table:

  ------------------------------------------------------------------
  | name                  | module       | inter | stream |  file  |
  ------------------------------------------------------------------
  | report                | report       |   *   |   *    |   -    |
  | report_recipients     | report       |   *   |   *    |   -    |
  | quarantine            | quarantine   |   *   |   *    |   -    |
  | drop                  | actions      |   *   |   *    |   -    |
  | deliver               | actions      |   *   |   *    |   -    |
  | deliver_to            | actions      |   *   |   *    |   -    |
  | modify_subject        | modify       |   *   |   *    |   -    |
  | modify_header         | modify       |   *   |   *    |   -    |
  | regexp_scan           | basic        |   -   |   *    |   -    |
  | string_scan           | basic        |   -   |   *    |   -    |
  | smtp_comm             | basic        |   -   |   *    |   *    |
  | max_file_size         | basic        |   -   |   *    |   *    |
  | file2stream           | file         |   *   |  -/out |   in/- |
  | stream2file           | file         |   *   | in(m)  |  -/out |
  | stream2mbox           | file         |   *   | in/-   |  -/out |
  | const                 | basic        |   -   |   *    |   *    |
  | match_all             | match        |   *   | in/out |   -    |
  | alternatives          | match        |   *   | in/out |   -    |
  | match_any             | match        |   *   | in/out |   -    |
  | nothing               | match        |   *   | in/out |   -    |
  | recover               | match        |   *   | in/out |   -    |
  | parsemail             | parsemail    |   *   | in/out |   -    |
  | mimeparse             | mimeparse    |   *   | in/out |   -    |
  | mime_ext              | mimeparse    |   -   |   *    |   -    |
  | file_type             | mimeparse    |   -   |   *    |   *    |
  | decompress            | decompress   |   *   |   -    | in/out |
  | stat                  | stats        |   *   | in/out |   -    |
  ------------------------------------------------------------------
  | clamscan              | clamav       |   -   |   -    |   *    |
  | clamd                 | clamav       |   -   |   *    |   -    |
  | libclam               | clamav       |   -   |   *    |   *    |
  | avg                   | avg          |   -   |   -    |   *    |
  | bdc                   | bdc          |   -   |   -    |   *    |
  | kav                   | kav          |   -   |   -    |   *    |
  | nodold                | nod          |   -   |   -    |   *    |
  | nod2                  | nod          |   -   |   -    |   *    |
  | nod2d                 | nod          |   -   |   -    |   *    |
  | dazuko                | dazuko       |   -   |   -    |   *    |
  | savse                 | icap         |   -   |   -    |   *    |
  | sophie                | sophie       |   -   |   -    |   *    |
  | trophie               | sophie       |   -   |   -    |   *    |
  ------------------------------------------------------------------
  | spamassassin          | spamassassin |   -   |   *    |   -    |
  | spamassassind         | spamassassin |   -   |   *    |   -    |
  | bogofilter            | bogofilter   |   -   |   *    |   -    |
  | qsf                   | qsf          |   -   |   *    |   -    |
  ------------------------------------------------------------------

  For detailed scanner description see README.scanners.

There are some opearators, which you can use over scanners:

  --------------------------------------------------------------------
  | operator |  "+"  |  "-"  |  "*"  |  "/"  |  "|"  |  "&"  |  "~"  |
  | S1 | S2  |       |       |       |       |       |       |       |
  --------------------------------------------------------------------
  | OK   OK  | L1+L2   L1-L2   L1*L2   L1/L2    L1      (1)     L1   |
  | F    OK  | Fail    Fail    Fail    Fail     L2     Fail     0.0  |
  | OK   F   | Fail    Fail    Fail    Fail     L1     Fail    NA(2) |
  | F    F   | Fail    Fail    Fail    Fail    Fail    Fail    NA(2) |
  --------------------------------------------------------------------

  S1 - status of first scanner
  S2 - status of secondary scanner
  L1 - returned level of first scanner (on left side of operator)
  L2 - returned level of second scanner (on right side of operator)
  NA - not available, can't occur
  (1) - if L1>=1.0: L1*L2; else 0.0
  (2) - ~ is an unary operator

When you are using interscanners, you can't attach scanners of
wrong type. This means, that you can't attach a scanner with
input of stream to scanner, which has only file output.

For example you can use:
  parsemail(string_scan(...))
but can't:
  parsemail(decompress(...))

5. SMTPd configuration

5.1 Postfix configuration

# /etc/postfix/main.cf
mynetworks = 127.0.0.0/8, ...
content_filter = smtp:[127.0.0.1]:27

# /etc/postfix/master.cf
# Local smtpd without scanning
127.0.0.1:26    inet n - n - 30 smtpd
  -o content_filter=
  -o myhostname=sagator.mydomain.sk
  -o local_recipient_maps=  -o relay_recipient_maps=
  -o mynetworks=127.0.0.0/8  -o mynetworks_style=host
  -o smtpd_restriction_classes=  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject

Myhostname can't be equal with your hostname, otherwise postfix
detects a loop.

  Add sagator to startup scripts. In this package there is
a sagator.init scipt, which can be used in /etc/init.d/ .
You can start it also manually.

5.2 Sendmail configuration

  As first step, please install required packages:
    sendmail with milter support
    python milters (http://www.bmsi.com/python/milter.html)

Configure your sendmail to use sagator's milter. Add these line into
  /etc/mail/sendmail.cf:
    O InputMailFilters=sagator
    Xsagator, S=inet:3333@127.0.0.1, F=T
For more information see milter documentation.

  Configure sagator to run as milter filter. Add this service to SRV in
sagator.conf:
  milter("sagator","inet:3333@127.0.0.1")

Limitations:
  Milter can't modify Subject and other headers, only can add some new.

5.3 Another smtpd configuration.

  For other smtpd, you can use sagator as main smtpd on port 25.
Configure your smtpd on another port as 25. Then configure
sagator to run on port 25 on your IP.

For example:
  Configure smtpd on port 26 on localhost and on your internal ip on port 25.
Allow relaying from your site on internal IP, but don't forget to deny
relaying on localhost:26 or from 127.0.0.0/8.
Use following configuration for sagator:

  SMTP_SERVER=('127.0.0.1',26)
  SRV = [ smtpd('YourExternalIP',25) ]

6. Antivir/Antispam installation

6.1 ClamAV installation

  Download and install clamav - clam antivirus.
URL: http://www.clamav.net/. You will also need some
decompressors, like unrar, unace, unarj, zoo, lha, tar, ...

  If you are using clamd, do not forget to start it in chroot,
and set these parameters in chrooted clamav.conf:
  DatabaseDirectory /var/lib/clamav
  LocalSocket /var/run/clamav/clamd.sock
  StreamSaveToDisk # recomended
  ScanMail # required
  ScanArchive # recomended
If you are using chroot, set in /etc/freshclam.conf:
  DatabaseDirectory /var/spool/vscan/var/lib/clamav
where /var/spool/vscan is your chroot. If you do this, freshclam
will update clamav database in your chroot.

7. Staring sagator

  There is an init script in scripts/ named sagator.init.
Copy it into any init.d directory and set it. For RedHat based system do:
  cp scripts/sagator.init /etc/init.d/sagator
Then edit file /etc/sysconfig/sagator:
  CHROOT="/var/spool/vscan"
  CLAMD="/usr/sbin/clamd -c /etc/clamav.conf"
  NOD32="/usr/local/bin/nod32d"
  AVG="/usr/local/lib/avg7/avgscan -d"

Warning: You must have antivir installed in chroot.

8. Testing

  There some test scripts. Go into test directory and try:

    ./smtptest localhost 25 Eicar
    ./smtptest localhost 25 spam
    ...

  With these scripts you can also test your sagator daemon,
and you don't need to configure content_filter in postfix.
You can test it without real traffic. I recomend for you to
start sagator as command line program, and see its reports first.
Only if it works really, then use it as daemon.

9. Scanning mailboxes

  In some case an virus can pass over sagator (for example, if your
antivir is not actual). If you detect this, you need to find viruses
in user mailboxes. There is an program named sgscan. If scans over
mailboxes and return finded viruses/spams. You can use:

  sgscan /var/spool/mail/*

  If you can scan only for viruses, you can use --av-only command
line parameter. Another parameter is --clean, if you want to create
clean mailboxes as /var/spool/mail/*.clean.
For renaming these mailboxes to real names use a shell script:
  for i in *.clean; do mv $i cho $i | sed 's/\.clean$//'; done
Warning: Don't use --clean and this script, if an user of an SMTPd
         has access to mailbox!!!

10. Retunrning badly quarantined mail to mailq/user mailbox

  If by chance is any mail quarantined, which should not, you can return
to queue or deliver to user mailbox by running command:

  sqback quarantined_file1 [quarantined_file2 [...]]

This can be done also via netcat program:

  nc localhost SMTP_PORT_WITHOUT_FILTER < .../quarantine/filename

11. Removing old files from quarantine

  Use tmpwatch. For example, if you want to delete older files,
than aprox. one month, use crontab command in /etc/crontab:

  0 0 * * * vscan /usr/sbin/tmpwatch 768 /var/spool/vscan/tmp/quarantine

or same line for vscan user's crontab:

  0 0 * * * /usr/sbin/tmpwatch 768 /var/spool/vscan/tmp/quarantine

Replace "/var/spool/vscan/tmp/quarantine" with path to quarantine directory.
For more information read "man tmpwatch" manual page.

12. Done

  Congratulations. You have sucessfully installed sagator. :-)
