Do you also happen to have a NIC installed in addition to your modem?
If so, it could be waiting to time out, causing the delay. If you don't use it, disable it, or remove it.

-Ryan