Some routers don't have the throughput to push 50mbps. If you switch the routers (or only use router #2), do you get 50 mb down from it?

It's also possible that the double-NATing is causing inefficiencies, but I think a throughput issue is more likely.