Archived Forum Post

Index of archived forum posts

Question:

Regarding GMail's "Lock down in Sector 4" error.

Nov 01 '12 at 09:59

Occasionally we get the dreaded "Lock down in Sector 4" error and Gmail prevents access for up to 24 hours.

It seems there are two criteria to cause a lock down: Connection duration and data volume. So my thinking is to use the quickest & lightest Chilkat methods to get in & out quickly.

Originally, I was using the Mailman CopyMail method which pulls down all email and then I perform a comparison to see if I already have the email. This probably creates the longest connection time & most amount of data download.

Now, since I am only interested in mail I have not downloaded yet, I'm thinking a multi-step approach would be lighter:

1) GetUIDLs for all email
(disconnect)

2) Compare against my database
(reconnect)

3) FetchMail for each UIDL I have not downloaded
(disconnect)


Answer

The quick answer is Yes, this sounds like a better approach. Although, I'm not sure it's worth disconnecting and re-connecting between the GetUIDLs and the fetching of the emails.

Here's more information about POP3 and IMAP in general that relates to the issue of retrieving new email.

The fundamental intents of POP3 and IMAP are opposite of each other. (I'm not speaking about Chilkat here, I'm speaking about the email protocols themselves.) The POP3 protocol is very simple, with very little functionality. The POP3 Inbox is intended to be a temporary holding place for newly arrived email. It is expected that an email client (Outlook, Thunderbird) will download the emails and manage them locally. An email client, such as Outlook, has extensive functionality to organize the emails (folders, sub-folders, etc.).

The IMAP protocol however, provides more extensive functionality for managing email on the server (i.e. creating folders, sub-folders, etc.) The fundamental intent of the IMAP protocol is that email will be kept on the server and managed there. Email clients such as Outlook, Thunderbird, etc. can download and manage emails locally, but it's also possible to allow the emails to be kept on the IMAP server and manage them remotely (thus allowing access to the emails from anywhere, or from web-based interfaces).

Many mail servers provide both POP3 and IMAP interfaces (meaning they accept connections on both POP3 and IMAP ports, and "speak" IMAP on the IMAP ports and speak "POP3" on the POP3 ports). The Inbox would be the same when connecting via IMAP or POP3.

It would be bad practice to use the POP3 Inbox as a place of permanent storage for email. POP3 emails can be accessed via sequence numbers or UIDL's. When a POP3 client connects, it has a non-changing snapshot of the emails in the POP3 mailbox. This does not change if new emails arrive. The emails have sequence numbers from 1 to N, where N is the number of emails in the Inbox. The sequence numbers are always from 1 to N, so it's not possible to uniquely identify an email (across sessions) based on sequence number. The sequence numbers are only valid for one session. UIDL's however, are unique across sessions. However, they are not necessarily numerical. They can be long strings with letters, numbers, and other special characters. Therefore, to identify new emails via the POP3 protocol, an app would need to download the entire list of UIDL's and compare each against a locally kept "database" of already-download UIDL's and then only download the unseen emails. You can see how if the POP3 Inbox is treated as a permanent store, the size of the UIDL list would continually grow, and the time to download the list and discover which are as-yet-unseen would continually grow. (I groan when I get a customer email w/ complaints about performance for a POP3 Inbox with (literally) a hundred thousand emails.)

The IMAP protocol however, uses sequence numbers and UID's (Note the difference between "UID" and "UIDL". POP3 uses UIDL's, IMAP uses UID's.) UID's are integers, and each new email is assigned a UID that is one greater than the previous. An app can call the imap.Search method (See IMAP UID Ranges in Search Method ) and then download the emails in the message set returned by the Search method. Also, if your app is the only client reading the IMAP Inbox, it could also download new email based on the SEEN flag. See Download Unread Email from IMAP Mailbox

For GMail, it's probably better to use the IMAP protocol instead of POP3, for efficiency.