Archived Forum Post

Index of archived forum posts

Question:

Good approach to Download many files with sftp.

Aug 27 '13 at 11:06

I'm using a sftp to download file from server. I want to download multiple file at a time, so I need to create many sftp instances and must call connect and initialize. I wonder if this would make many connections to the server, and reconnect with same authentication many times, is there another better or faster way to to do this?


Answer

Each instance of an SFTP object has no knowledge of any other instance. Each instance will have it's own connection and would need to authenticate. I don't think there is a better/faster way.


Answer

Sync would be a one by one download. Async would establish threads to do simultaneous downloads. With a handful, not really any difference, but with hundreds, thruput would be greatly enhanced.


Answer

The API is thread-safe, meaning that for a given object instance, only one call can be active at a time. Therefore, using multiple threads is possible, but each thread should have its own object instance and therefore its own connection.

If, for example, you have 1000 files to download, you could perhaps create 10 threads, each of which would suffer the initial overhead of connecting and authenticating, but once that is done, each thread can download one file after the other without re-connecting or re-authenticating. In other words, it is (of course) possible to download many files one-after-the-other in a single session. You would use multiple threads, each having it's own session, to achieve simultaneous downloads.