If you have a Nat Capable Modem and a router there is a bit more to the configurations then there is with only one.. i may be mistaken but it sounds like you have both. Primary with correctly configured ports will usually yield more results.. and if you want to have a go at it.. i have a manual that was made by MicroMecca that may help I will attach below.
As or secondary .. yea I prefer it, my speeds are actually better when ports are configured and things are set right. May still yet be better when the other issue with the modem is settled.