Currently I'm using the PHP IMAP libs to connect and download full imap accounts.
Current workflow;
- Cronjob gets all folders with uidvalidity and inserts into DB. Will also list ALL UIDS (queue) found and insert into DB. (Taking 5 to ~100seconds depending on size (1MB to 10GB mail accounts)
- Background process gets 15 UIDs from queue (which are in same imap folder). Connects to server and connects to the folder. Then it will fetch the messages from the (15) uids in the queue with the FetchSingle function ( http://www.chilkatsoft.com/refdoc/phpImapDoc.html#method031 ).
- The background process will extract the needed information (subject, addresses, body text and attachments) from the email object and then save the Email object to an .eml file ( http://www.chilkatsoft.com/refdoc/phpEmailDoc.html#method128 )
Are there any ways to speed up the background process of downloading (this is taking the most time) the e-mails. It must be able to handle big accounts (>25GB, 50k messages). Currently it is just going at about 250 - 1k e-mails per hour.
I'm hoping someone could give me any tips about how to speed this process up.
Currently getting pretty much the same results @ 2 servers (1 x Amazon Ec2 with RDS and 1x a VPS with local mysql db).
Archived Forum Post