Postfix, maildir's, and writing filters

[Da Rock]

On Tue, 2008-09-23 at 08:24 -0400, George Fazio wrote:
> Da Rock wrote:
> > On Mon, 2008-09-22 at 20:53 -0400, George Fazio wrote:
> >   
> >> Da Rock wrote:
> >>     
> >>> Howdy. This may seem simple, but I'm completely green on this: I have a
> >>> postfix server with a courier-imap client frontend using maildir's. I'm
> >>> using imap for an internal mta, but I need to setup a system which
> >>> retains copies of sent emails on the network and not on individual
> >>> workstations (which is what happens currently).
> >>>   
> >>>       
> >> When you say courier-imap client, do you mean you're using maildrop to 
> >> deliver the message to the user's maildir or that there is an end-user 
> >> courier-imap client?  I am only familiar with the maildrop piece of courier.
> >>     
> >>> I've looked at some of the solutions (bcc and send to a psuedo account
> >>> for each user, bcc to the user and filter the incoming mail on this) but
> >>> it seems like a very roundabout way of doing things. I've read up on
> >>> Postfix, and there is support for custom filters, so:
> >>> 1. what does it take to write one?
> >>> 2. how does one copy email from one folder to another in maildirs? Is it
> >>> possible?
> >>>
> >>>   
> >>>       
> >> This is a classic case of over engineering.  You do not want to bcc back 
> >> to the user, or filter the mta, just move the outgoing messages to the 
> >> sent folder.  You might need bcc for the purposes of journaling all 
> >> email, if you have any legal requirement (sox, hippa, etc.) that require 
> >> it.  But, that it another ball of wax entirely.
> >>     
> >>> This idea I have should filter the outgoing mail and copy the messages
> >>> to the sent folder as well as retaining its place in the queue.
> >>>
> >>>   
> >>>       
> >> If the end-user's client is using imap and configured properly, it 
> >> should do this for you.  Thurderbird, the full version of Outlook (and 
> >> probably Express), and many other clients support this natively - you 
> >> just have to make sure the client is configured to do that.  Typically, 
> >> in the configuration of the client, there is something that says 
> >> something like "save a copy of sent messages to <folder of choice>".  I 
> >> don't know what client you're using.  I use Pine/Alpine, Thunderbird, 
> >> and Outlook (when I have no other choice).
> >>
> >> If the end-user's client is using pop, then you have a problem that may 
> >> require a custom solution like you speak of above.
> >>     
> >>> Any ideas? Maybe a link to some good info? I would like to know how to
> >>> do this myself so I can do more in the future so info and pointers would
> >>> be great (if you have a script you'd like to share then please show me
> >>> how it works :) ).
> >>>
> >>> Cheers
> >>>
> >>>   
> >>>       
> >> My mail system is running postfix (mta) w/ dovecot (for imap or pop 
> >> access from the clients), maildrop (for delivering to a maildir), and 
> >> amavis-new (for spam filtering and virus scanning w/ clamav).  My mail 
> >> clients are configured for imap, and they save copies of sent mail to 
> >> the sent folder as expected.  While I am using dovecot, and not courier, 
> >> for my imap server - I cannot imagine that any other imap server would 
> >> handle things any differently ... it's core functionality that ever imap 
> >> server should have imho.
> >>
> >> -George
> >>
> >>     
> >
> > Me too. It may be possible to save a copy in evolution, but I haven't
> > found it in all clients. Plus my system needs to be suitable for a
> > webmail system, and yes some pop clients.
> >
> > You sound like you know maildrop very well, I was considering using it
> > as a part of the solution. If I wrote a milter script for postfix, is it
> > possible to pass the message to maildrop so that it can take care of the
> > formalities such as filenames and formats and tell it to put it in a
> > sent folder? Something like a shell or perl script that uses this line
> > to run maildrop:
> >
> > maildrop -d $user Maildir/.Sent
> >
> > Obviously the message itself will be piped, and the $user will be
> > obtained by copying the from field in the message.
> >
> > Would something like this work? I've been searching on google but
> > haven't found a clear answer, they only mention using maildrop filters
> > and commands there- not actual usage of the maildrop cli.
> >
> > Cheers
> >
> >   
> If you wrote a filter for postfix, which I have no idea how to do, the 
> maildrop command you sight looks good.
> 
> A friend of mine is using round cube for web mail (I don't have webmail 
> setup on my server), and it will also save a copy to the sent folder.  
> Most of the web clients should provide this functionality.  The pop 
> clients are the ones that are going to be an issue.
> 
> I'm not sure what Google is using for gmail.  But, when I send a message 
> using their smtp service, a copy goes into my account.  So, what you're 
> looking to do is obviously possible, but I do not have an experience 
> with it as I have not had need for a solution yet.  I would certainly be 
> interested in what you come up with though.
> 
> Another possible option is to configure the clients to always bcc the 
> sender, and then write a maildrop rule.  Something like "if 
> (/^From:.*user\@domain\.com/) to Maildir/.Sent" might work (I have not 
> tested this rule, it may not function as desired).  That would have to 
> go in the .mailfilter file in each user's home dir.  I think you can 
> have global maildrop rules if I remember correctly.  You might be able 
> to sub user for a variable that contains the user name or the message 
> sender, or something like that, so that you do not have to modify the 
> .mailfilter file for every user.

Interesting possibility here: I have found a program which will directly
write to a maildir via stdin (same as maildrop). You can use the postfix
internal filter to do header checks and filter them on access lists
(which could be sender rather than recipient), and send it to the
"filter", which can be the script which connects to the program.

Now I've adjusted the script to take the $sender parameter from postfix,
and I'm working on 2 aspects: the regex to grab the user details out of
the sender address, and a way to dup the message sent in via stdin and
send it to the stdin of the program as well as back into the mail queue
for further processing (without setting off a loop).

The regex I think I've worked out, but the stdin dup I'm having trouble
with. With regards the queue management, from my understanding postfix
runs several queues which are at various points of process and one can
send using the sendmail command to a local port (I think- anyone know
better?). But how do I dup the message?

I think I can setup a buffer and read the stdin into it first, then type
or cat it into the stdin of the program, then send it back to postfix.
But I'm getting a little lost in coding of the original script, and I'm
not quite sure how to adjust it for this.

Any ideas on this so far? Here is the script: (kept comments for
recognition of original author)

-------------------------------------------------------------------------
# Deliver Maildir - A program to reliably deliver files to Maildir
format directories.
# Copyright (C) 2004  Eric M. Hopper  <hopper at omnifarious.org>
#
# This program is free software; you can redistribute it and/or modify
it under
# the terms of the GNU General Public License as published by the Free
Software
# Foundation; either version 2 of the License, or (at your option) any
later
# version.
#
# This program is distributed in the hope that it will be useful, but
WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS
# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
# details.
#
# You should have received a copy of the GNU General Public License
along with
# this program; if not, write to the Free Software Foundation, Inc., 59
Temple
# Place, Suite 330, Boston, MA 02111-1307 USA

# This shell function takes the mail to be delivered on stdin, and has
# an argument specifying the Maildir to deliver to.  It does wait for
# stdin to be exhausted before sending the SIGUSR1 signal, but it does
# assume that it should always deliver.  This may not be what you want
# in a real shell script.
function dmd {
    if [ "$#" -ne 1 ]; then
        echo "Usage: $0 <maildir>" 1>&2
        return 64
    fi
    $user = ${1%%
@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])}

    $maildir = "/home/$domain/$user/.Maildir"

    # Start a sub-environment so that the cd doesn't change the
    # calling environments current working directory.
    (
        # Die with error if cd fails.
        cd "$maildir" || return 73
        # Make a temporary directory to hold the fifos.
        local tmpdir="$(mktemp -d dmd-XXXXXXXXXX)" || return 73
        # Starts another sub-environment so the temporary directory
        # can always be zapped, no matter what happens. 
        (
            mkfifo "$tmpdir/mail" || return 73
            mkfifo "$tmpdir/results" || return 73
            # Start delivermaildir with stdin and stdout coming from
            # the new fifos.  Save its pid too so we can signal it.
            delivermaildir . <"$tmpdir/mail" >"$tmpdir/results" &
            local pid="$!"
            # Start a sub-environment that has its stdout directed
            # into delivermaildir's input fifo, and has file
            # descriptor 3 reading delivermaildir's stdout fifo.
            (
                # Read in the 'initialized' message.  This could also
                # occur after the 'cat'.  The -u argument to 'read'
                # reads it from file descriptor 3.  Assume that if it
                # can't be read in 30 seconds, that something went
                # wrong.
                local line
                read -rs -u 3 -t 30 line || return 70
                # Print out the line for instructional purposes.
                echo "line: $line" 1>&2
                # It should say 'initialized'.
                if [ "$line" != "initialized" ]; then
                    return 70
                fi
                # Use cat to copy stdin to the sub-environments
                # stdout, which is hooked to delivermaildir's stdin
                # via a fifo.
                cat
                # Now that's done, send the USR1 signal to
                # delivermaildir so it will deliver the mail.
                kill -USR1 "$pid"
                # We don't have to do this, but it's nice to read the
                # output from the signal handler stating that
                # delivermaildir will deliver the mail.
                read -rs -u 3 -t 30 line
                echo "line: $line" 1>&2
                # When the sub-environment exits, it will close its
                # stdout, which will result in delivermaildir getting
                # EOF on its stdin.
            ) >"$tmpdir/mail" 3<"$tmpdir/results"
            local ret="$?"
            if [ "$ret" -ne 0 ]; then
                kill "$pid"
            else
                wait "$pid"
                ret="$?"
                echo "delivermaildir returned $ret" 1>&2
            fi
            return "$ret"
        )
        local ret="$?"
        # Zap the temporary directory.
        rm -rf "$tmpdir"
        return "$ret"
    )
    return "$?"
}
--------------------------------------------------------------------------

The original lacked the definition of the $user and $maildir variables,
and simply took the first argument from the command line at the first cd
command.

In the master.cf for postfix I have included a new interface definition
for this script as follows:

copysent	unix	-	n	n	-	-	pipe
	user=vmail	argv=/path/to/copysent.sh ${sender}

I haven't looked into the main.cf paramenters yet, but so far all of
this looks possible.

Any ideas guys?