Can we have the exact text of the errors please, aas it's not perfecrtly clear what you mean
virtual memory settings can be adjusted in windows if that's what's being referred to, right click my computer, choose properties, advanced tab, clicking performance options, & then the "change" button.. if you've run out i'd suggest doubling both start & maximum values as a first move.. these changes require a reboot to take effect.
5000 is a kinda limit on a primary connection, it's all that winmx will list on a browse (3000 for secondary), i believe the unlisted files can still turn up in searches, in order to have more shared & still show in browses you'll have to investigate plusmx or similar to run multiple instances of winmx, or run 2 separate copies from separate user accounts
