Join/Renew Benefits Sage Programs SysAdmin Resources Jobs Board SAGE Home
The USENIX Special Interest Group for Sysadmins
16 June 2004

On IMAP Service for Customers

Phil Pennock, phil dot pennock at globnix dot org

The name of an ISP has been concealed below; in the process, this document could be interpreted as making sweeping generalisations about all ISPs. That is not the intent, and the author apologises for not taking the time to rephrase it appropriately for this different audience. Thanks go to Rob Kolstad for his work on this, but errors and skewed opinions remain mine.

The issue of providing IMAP service to customers has been raised more frequently recently. This missive is an attempt to explain the issues involved, why there's no current product [at my ISP] and the need to be very careful in anything promised to customers (or potential customers).

I think it important for Sales/Product-Management to understand the issues, so that they can be better informed than our competitors. [Sometimes below I say "company" when I mean "organisation"—sorry about that, I just mean "customer accessing mails which are more than this week's jokes and letters from aunty".]

In Internet mail-systems, there's one distinction that most people can happily ignore, even though it affects every mail which they send or receive. It's the concept of "Final Delivery". Email messages can be passed through many systems between the sender and recipient (though some systems will decide that 30 is too many and evidence of a mail-loop). But only one of those, the last, is delivery into the user's real mailbox. That's the Final Delivery.

Up until Final Delivery, the message is in transit. If the message can't be passed on, then a delivery failure notification ("bounce") message is generated and passed back to the sender. Once it reaches the final mailbox, there will be no bounce. There's no guarantee that the mail will actually be read, but it won't be bounced because it's been delivered to "where it was supposed to go".

IMAP is designed as a system to manage mails once they've reached Final Delivery. SMTP is before Final Delivery. POP3 is a slightly weird hybrid, but not really well suited to use after Final Delivery. For many customers using POP3, the POP3 retrieval to their Outlook program is the Final Delivery.

Some ISPs do not provide Final Delivery. They pass mail on to customers. Historically, ISPs just provided SMTP; nowdays you can also find POP3. If a mail is not collected within 35 (or whatever) days, the mail systems delete the mail. If it was read by POP3, it's just deleted; if not read by POP3, a bounce message is sent back.

Once a mail reaches Final Delivery, it can stay in the system for as long as the user wants, really. There's no 35 day limit. If someone has an important mail which they treasure, they might want to keep it for the next 70 years or more—this is to be expected, not something exceptional.

Because ISPs just handle messages "in transit", there are rough limits on how much mail needs to be stored. ISPs engineer the systems for a different set of operating conditions than the conditions for providing Final Delivery.

For instance, systems are extremely unlikely to lose email. That doesn't mean that they can't lose email. An asteroid can strike the planet, destroying Western Europe; enough disks can fail in the NetApp, all at once, to lose messages. These are both possible (and the latter is hopefully more likely). If a customer collects their mail regularly, between a few hours and a couple of days worth of their email might be lost; if the customer doesn't collect regularly, up to 35 days worth might be lost. The design makes this unlikely but there's no possible design which makes losing email impossible. In the worst case business insurance covers this.

The monetary worth of 35 days mail as opposed to 35 or 70 years email is an interesting thing to consider . . .

Because of the issues of volume, many ISPs can't offer an IMAP product that scales to all customers. Individual small products can be offered, though, on the scale of what a company of a couple of hundred employees might implement internally—that's doable.

So, IMAP needs to be considered not as "another way for customers to get their email", but instead as "how customers manage email which they've received". They get multiple mailboxes each, can move mails between them, delete mails, flag mails, add attributes and keywords, search on message content, and much more besides.

The mail-store of the IMAP system will hold immense amounts of information important to an organisation. Only the organisation itself knows how important. But any strategy needs to handle issues such as back-ups, replication, archivals, policies on personal mailboxes of departing staff, etc etc.

But how much information can be held? How much should be held? "All emails" can get to be a very large amount of data, over time. Much of the information will lose relevance; some will not. From an operations point of view, it's certainly useful to me that I can look over the past three or fours years worth of [list] mails, quickly looking for messages matching a few keywords because I vaguely remember that something similar happened before—with paper memos this would not be a productive use of my time and I'd be better off figuring out the solution from scratch (or having decent documentation). Messages can, but do not necessarily, become less relevant over time, losing useful information.

Any "IMAP solution" offered needs to consider not just "mail for a few months". It needs to be "emails useful to the customer, for the lifetime of their usefulness". It needs to provide for backups, in various forms. It needs true Disaster Recovery. It needs a lot which is expensive to provide—probably the reason that customers come to ISPs, after their sysadmin/consultant has told them how much it costs to do things properly.

Any IMAP offering that's cheap to implement carries a potential legal and financial nightmare—I really don't think that most ISPs offering IMAP have properly evaluate this, just as many who rushed to offer free accounts didn't evaluate their long-term business plans there, either. The ISP market is still young enough that it's filled with cowboys who have much financial backing but simply don't understand what they're doing or what the longer-term consequences of their products are, focusing on "get more customers now" instead of "get customers who we'll keep and who won't be sueing us into backruptcy in 3 or 8 years".

Emails have much to offer (including the searchability, mentioned above), but have some challenges for a company which are typically not addressed. IMAP is an important part of the picture, but not the whole picture. Really, what I believe is needed is for organisations to understand how emails are used, understand how other forms of documentation are used, and establish policies for email retention and migration of information into more formal documentation. For instance, it might be something like:

emails more than seven years old are archived onto read-only long-life medium and deleted from the live system; after only six years, someone reviews mails to mailing-lists X, Y and Z for information that looks like it might be still relevant and collects those emails together for review by specialists, who will collate those which still hold relevant important information and ensure that all the information is held in Procedures, Policies, Guidelines or other formal documentation.

Clearly, this ties deeply into internal information management. Not losing information involves some bureaucracy (the NOC will now boo and hiss). Choosing how to handle this is not easy and may well be different for every customer. Their business processes for information management need to be designed to migrate information of any value from ephemeral communications such as emails into more static forms of retaining information such as traditional documentation.

An ISP could offer some standardised services along these lines, with tools and calendaring designed to make it easy for companies to collect the information they need; with automatic burning to DVD (this year; who knows which medium in six years time?) of a customers mails every time a certain volume is accumulated or amount of time passes, with those DVDs being mailed to the customer by recorded delivery or courier. There's a lot that can be done.

But any time that you look at getting this deeply involved in a company's internal processes, what you're actually selling is IT outsourcing services, not Internet Access.

I believe that offering "proper" IMAP access is something which intrudes deeply into the market of some much larger companies; it's not something to be taken lightly, and it's certainly outside our areas of expertise.

Certainly, if one wished to start moving into ASP (ye olde Applications Service Provider, which we heard so much about a couple of years ago) or IT outsourcing, then that's a different matter and IMAP is just one of the technologies which would be used to provide services. But this is not Internet Access. It's not shifting information around, or allowing customers to shift information to others. It's controlling how customers shift information about within their own organisation, which is a fundamentally different animal.

"Just providing IMAP access to a maildrop" is a short-sighted viewpoint; unfortunately, many customers won't understand the issues here and will want "just that". Perhaps this email can be massaged into a suitable position paper for issuing to customers, to make them think more deeply about the issues and realise that we want to work with them to offer good solutions, not just take their money for whatever we can without regard to the consequences.

Some ISPs can offer a small IMAP product in the near future, but it's essential that this isn't hyped and that we are pro-active in ensuring that customers have information on the limitations described here and suggestions that they consider this a stop-gap solution whilst they develop something which integrates better with their business processes.