Andrew's Corner :

Using Mutt with Gmail

This page is a guide to using the email client mutt, in conjunction with fetchmail, procmail and msmtp, to send, receive and read email under Linux using a Gmail account as a relay. If you need to ask why anybody would want to put such an effort into an undertaking like this, and did not simply use the web interface of Gmail or even mutt with IMAP, perhaps this page is not for you. To use another's words:

Mutt is not designed to suit everyone, including those without basic knowledge about the Mail Concept, or those unwilling to perform configuration. mutt is even not designed for the mass of "average users", although it works well for them, too. For those who prefer a client to "just work somehow" rather than give dedicated best performance, other mail programs are probably better suited.

Perhaps you already feel the lure of the incredibly powerful Linux console applications? If so read on and please use the email link at the base of this page to let me know how you profited, or otherwise, from the information on this page.

Contents of the Page

This is a fairly complex page that I believe would benefit from reading from top to bottom but I include some navigation here for those who would like to sample rather than consume:

Downloading the Mail Receiving mail from Gmail.
- Retrieving the SSL Certificates Retrieving and setting up the required Gmail SSL certificates.
- Setting up fetchmail Setting up the Mail Retrieval Agent (MRA) fetchmail.
- Setting up procmail Setting up the Mail Delivery Agent (MDA) procmail.
Sending the Mail Setting up the Mail Sending Agent (MSA) msmtp.
Reading the Mail Setting up the console Mail User Agent (MUA) mutt.

I will not be dealing with the specifics of downloading and installing the required software for this page as there are too many distro-specific issues there. But obviously you will need to have installed OpenSSL, perhaps cURL or the Debian ca-certificates package, fetchmail, procmail, msmtp and mutt; all fairly standard Linux programs. Whether this is done by using your distro's version or compiling from source the end result should be the same. But before moving on to the actual setup I need to introduce my new friend John.

Introducing John ...

To avoid confusion in editing the many configuration files involved in this setup I will describe the setup of mutt and gmail for my new friend John, who has been created especially for this guide. John's details are as follows:

Gmail Address:        john.example@gmail.com
Gmail Password:       rover
Computer Username:    john

John's details will always be in italics, bold and in red to remind you, Gentle Reader, to substitute your details for his. Hopefully this will lessen the confusion that I have unintentionally created with older versions of this page!

Downloading the Mail

Downloading the mail is perhaps the most complex part of this exercise but take it one step at a time, Gentle Reader, and it becomes quite logical. First step is to download the required SSL certificates, then to setup fetchmail and finally to setup procmail.

Download the SSL Certificates

There are two certificates required to access Gmail and the good news is that they are freely available and easy to setup. The first certificate can be gained directly from Gmail by issueing the following command:

$ openssl s_client -connect smtp.gmail.com:995 -showcerts

Extract the certificate, as shown below, and save it as $HOME/.certs/gmail.pem.

Next is Equifax CA (Certificate Agency) that I note is included in the cURL CA bundle which you should find in /usr/share/curl/curl-ca-bundle.crt. Debian / Ubuntu users can rejoice as both of these certificates will be found in the ca-certificates package in /etc/ssl/certs. I note as well that mutt itself appears to be shipping certificates with its source code as ca-bundle.crt. Extract the certificate, as shown below, and save it as $HOME/.certs/equifax.pem.

The painful bit is almost over and it remains to rehash the certificates and then check that they will authenticate with Gmail:

$ c_rehash $HOME/.certs/
$ openssl s_client -connect pop.gmail.com:995 -CApath $HOME/.certs/

With the following comment from Gmail "+OK Gpop ready for requests..." we can move to lighter areas and start setting up fetchmail.

Setting up fetchmail

Fetchmail is fairly easy to use and setup, particularly as the hard yards have already been done with the SSL certificates. You will need to create the file $HOME/.fetchmailrc and then add the following information for accessing the Gmail server:

poll pop.gmail.com                   
with proto POP3                      
user 'john.example@gmail.com'        
there with password 'rover'        
is 'john' here                              
mda "/usr/bin/procmail -d %T"        
options                                                             
no keep                                 
ssl                                  
sslcertck                            
sslcertpath /home/john/.certs/

It is almost too obvious to point out, Gentle Reader, that you will need to substitute your own details for password, username, email address etc. There remains a final touch, since the username and password are openly in this file you should make the file readable only by the file owner:

$ chmod 600 ~/.fetchmailrc

Now would be a good to time also to make sure you have POP forwarding enabled in your Gmail account. You will find this in: Settings - Forwarding and POP at the web interface of Gmail. Note as well that you cannot remove mail from Gmail servers via POP3 but you can choose to have your messages archived, kept or deleted once they have been downloaded via POP3. This is a Gmail setting hidden in Settings - Forwarding and POP: "When messages are accessed with POP..."

Setting up procmail

procmail is the final link in the chain for downloading mail. Before setting it up there is a little bit of outstanding business to attend to: setting the $MAIL environmental variable so that all software that deals with mail knows where the delivery point is. Add the following to ~/.bashrc making the obvious substitution for username:

# Sets the Mail Environment Variable
MAIL=/var/spool/mail/john && export MAIL

procmail will now know the location of the default mail spool and will deliver all mail to there that has not been sorted to other locations. But the creation of a $HOME/.procmailrc is still required and I give an example below. I have also added a sample filtering recipe for the mutt user mailing list to give the very beginning of what can be a complex process:

PATH=/bin:/usr/bin:/usr/local/bin 
VERBOSE=off  
DEFAULT=/var/spool/mail/john            
MAILDIR=$HOME/mail            
LOGFILE=$HOME/.procmaillog  
# Recipes below this comment:

:0:
* ^TOmutt-user
mutt

The single filtering recipe means that procmail delivers all email addressed to "mutt-user" directly to $HOME/mail/mutt. All other mail goes to the default location /var/spool/mail/username as specified by $MAIL environmental variable. If for some reason you are averse to adding this to ~/.bashrc, Gentle Reader, I have demonstrated above how to add this to ~/.procmailrc with the DEFAULT setting. You will not forget, of course, to make the obvious substitution for "username" in the settings above? For the definitive guide to further recipes, and guidance on much more complex recipes, don't forget to run man procmailex, it all starts to make sense after a while.

Sending the Mail

I have formerly been an advocate for the simple MTA ssmtp, which some would call a Mail Sending Agent (MSA). However I believe that development of ssmtp has ceased and I have moved with some regret on to msmtp. And in a bit of bad news, Gentle Reader, we need to deal with another certificate. This particular certificate is the Thawte Premium Server CA and can again be found in the cURL package and can thus be extracted from /usr/share/curl/curl-ca-bundle.crt. Mychael gives great directions for another method which involves downloading the entire CA package from Thawte and extracting the correct certificate (ThawtePremiumServerCA_b64.txt). You may choose to follow his advice or trust me and profit by my own endeavours in simply presenting the certificate here:

Save this certificate as $HOME/.certs/ThawtePremiumServerCA.crt and then we can move on with the rest of the setup for msmtp. A single configuration file is required $HOME/.msmtprc and the following settings to access Gmail:

account default              
host smtp.gmail.com          
port 587                     
from john.example@gmail.com     
tls on                       
tls_starttls on              
tls_trust_file /home/john/.certs/ThawtePremiumServerCA.crt
auth on                     
user john.example        
password rover          
logfile ~/.msmtp.log

I need not mention again, Gentle Reader, that there should be some fairly obvious changes here to substitute your own username, password and email address? And then the final touch, since the username and password are openly in this file, you should make the file readable only by the file owner:

$ chmod 600 ~/.msmtprc

msmtp is a great program that has many features that quite frankly I am still exploring, feel free to point out anything that I have missed, there is an email link at the base of this page for that purpose. But now finally to Mutt:

Reading the Mail

May 17th 2008 saw the release of a "Development" version of Mutt: 1.5.18 and it is this version that I have used for this page. It may not be available for your distro but can easily compiled from source. Mutt is driven by the file $HOME/.muttrc. I spent some time building this file from scratch but for you, Gentle Reader, I include here a more basic version, similar to the one I started from. Some parts of this, such as aliases and colors, are sourced from their own file so don't forget to create these files.

#======================================================#
# Boring details
set realname = "john"
set from = "john.example@gmail.com"
set use_from = yes
set envelope_from ="yes"

# Use a signature
set signature="~/.signature"

# Use msmtp rather than sendmail. Check that 
# the path is correct for your system:
set sendmail="/usr/local/bin/msmtp"   

# If not set in ~/.bashrc:
set spoolfile = /var/spool/mail/john

#======================================================#
# Folders
set folder="$HOME/mail"      # Local mailboxes stored here
set record="+sent"           # Where to store sent messages
set postponed="+postponed"   # Where to store draft messages
set mbox_type=mbox           # Mailbox type
set move=no                  # Don't move mail from spool

#======================================================#
# Watch these mailboxes for new mail, useful only if 
# Procmail or Maildrop is used to sort mail.
mailboxes ! +slrn +fetchmail +mutt
set sort_browser=alpha    # Sort mailboxes by alpha(bet)

#======================================================#
# What to show and order of headers
ignore *
unignore Date: From: User-Agent: X-Mailer X-Operating-System To: \
         Cc: Reply-To: Subject: Mail-Followup-To:
hdr_order Date: From: User-Agent: X-Mailer X-Operating-System To: \
        Cc: Reply-To: Subject: Mail-Followup-To:
                       
#======================================================#
# which editor do you want to use? 
# vim of course!
set editor="vim -c 'set tw=70 et' '+/^$' " 
set edit_headers=yes      # See the headers when editing

#======================================================#
# Aliases

set alias_file = ~/.mutt/mutt_aliases # In their own file
source ~/.mutt/mutt_aliases           # Source them
set sort_alias=alias                  # Sort alphabetically

#======================================================#
# Colours: defaults are a little bleak so experiment!

source ~/.mutt/mutt_colors            # In their own file 

#======================================================#
# Lists: An example using the mutt-users list:

lists mutt-users
subscribe mutt-users
set followup_to=yes        # Sets 'Mail-Followup-To' header
set honor_followup_to=yes  
fcc-hook mutt-user +mutt   # See your own posts using fcc

#======================================================#
# Odds and ends

set markers          # mark wrapped lines of text in the pager with a +
set smart_wrap       # Don't wrap mid-word
set pager_context=5  # Retain 5 lines of previous page when scrolling.
set status_on_top    # Status bar on top.
push <show-version>  # Shows mutt version at startup

I have been taken to task somewhat by one reader of this page who felt there should be a little more information about Mutt and colors and so ...

Coloring in Mutt

If your terminal supports color, and I believe it is a rarity these days for this not to be case, you can color almost any aspect of the Mutt window. All the fine details are in the Mutt manual: "Section 3: Configuration 8: Using color and mono video attributes". But I can tell you, Gentle Reader, that the basic usage is:

color   object   foreground   background

This can be made a lot more compilcated but a simple start is a good start. The basic colors are white, black, green, magenta, blue, cyan, yellow, red and default but you can also prefix a foreground color with "bright" to make the color bold. Now I personally use a Terminal with a white background so my own colors are:

#---- Mutt Colors for White Background -------
color    hdrdefault    black           default   
color    quoted        red             default   
color    signature     brightblack     default   
color    indicator     brightwhite     red
color    attachment    black           default
color    error         red             default   
color    message       blue            default   
color    search        brightwhite     magenta
color    status        brightyellow    blue
color    tree          red             default   
color    normal        blue            default   
color    tilde         green           default   
color    bold          brightyellow    default   
color    markers       red             default

There should also be a file called colors.default installed as part of your Mutt installation which is intended for Terminals with white backgrounds, this might also furnish a starting point as it did once for me. Just to balance out the equation a little I will also give the colors for those who prefer a black background. This example is taken directly from another sample file colors.linux which should be installed along with Mutt:

#---- Mutt Colors for Black Background -------
color   hdrdefault   blue              black
color   quoted       blue              black
color   signature    blue              black
color   attachment   red               black
color   message      brightred         black
color   error        brightred         black
color   indicator    black             red
color   status       brightgreen       blue
color   tree         white             black
color   normal       white             black
color   markers      red               black
color   search       white             black
color   tilde        brightmagenta     black
color   index        blue              black ~F
color   index        red               black "~N|~O"

If you don't like either of these feel free to experiment a little and come up with your own, most Mutt users will alter these basic colors. I prefer mine simple but if you pull out the manual you will see that you can spend many hours getting it exactly as you want. But now finally to check the mail:

Reward Time!

Finally it is reward time as you open Mutt, type ! to open a shell prompt, type fetchmail -v and start reading the mail! My parting gift to you, Gentle Reader, is a little macro that was written for me by a generous person on the mutt-user mailing list that will actually do this for you when you simply press "I". Place the following in your ~/.muttrc file:

macro index,pager I '<shell-escape> fetchmail -v<enter>'

This ~/.muttrc is intentionally a little basic, although I suspect that it will cover most needs anyway. Don't be afraid however to spend hour after hour painstakingly crafting your own to produce that perfect setup as mutt encourages and rewards such efforts!

And in conclusion...

I wish you all the best with one of the truly great Linux console programs! Please send me an email, using Mutt of course, to let me know if you have found this page at all useful and as well to suggest any corrections that feel should be made. Remember as well the most important thing: "Have Fun!".