Slow Internet - constantly need to restart modem

My internet speed is constantly at around 3 mbps. I restart my modem and my speed increases to 80 - 90 mbps; however, after a few minutes, the speed drops back down to 3 mbps. It does not matter how many times I restart the modem, the high speed only lasts for a few minutes. Is this an issue with my modem or comcast? The modem is a Motorola Surfboard.