Archived Forum Post

Index of archived forum posts

Question:

Optimize HTTP downloading of many files?

Oct 24 '12 at 13:45

Do you have any examples, comments, features, etc. relating to using a persistent connection to optimize multiple downloads? I'm looking at a situation where I may be downloading a lot of small image thumbnails, maybe 100 or 200 at a time. I guess under HTTP 1.1 persistency is sort of assumed, but it isn't clear to me how that would play out with multiple, say, CkHttp_Download() calls.


Answer

A Chilkat HTTP object instance will keep a connection open after sending a request - assuming that it can. Of course, if the server closes its side of the connection, then the HTTP object will automatically re-connect on the next method call that requires a connection. The HTTP object will also close the connection if the HTTP response includes a "Connection: Close" header.

Therefore, if the multiple downloads are from the same web site, then the Chilkat HTTP object will probably use the same connection for all of them. There's no difference in the coding in the application. A call to the http.Download method within a loop will automatically establish the connection if needed, or use the existing connection if possible.

To gain even greater throughput, you could design an app that uses multiple HTTP object instance simultaneously. For example, you could create a multi-threaded program with a main controller thread and multiple background threads. The main thread could have a list of download URLs, and it could pass the next URL to a background thread when it is ready for another. You would have to work out the details, but it shouldn't be that difficult.