There can be several reasons for you to loose the connections with folks.
Having the connection reset by peer means they are the ones with the problem, either they are disconnecting for some reason or they are using a upload manager.
If your sharing plenty of files then it could be an upload manager gone mad even.. they can get a bit picky on ya..
Dropping connections on your side can be caused by flooding, finding sources that are flooding and also from using 3.53 on secondary can cause it also..
You may want to try making sure your sharing plenty of files, use 3.54 if possible and see how you get on. Try using a primary if you can also.
Let me know..