Only if you had more than one residential IP from them. If you need more ports for your LAN devices then just connect the switch to one of the LAN ports on one of the routers and use it as an extender.
I just got comcast installed, their modem had a built in wireless router and I was able to connect it to my existing wireless router and 5 port switch. It all worked fine. I decided to pull my linksys wireless router out since I didnt really need it since the modem with the built in router had 4 ports. But I was able to use my wireless router, and a 5 port switch with comcasts modem with built in router.
just to clarify, the 5 port switch was not required to use both routers. We got both the modem set up first, then added my router and verified connection via that and them added the 5 port switch and everything worked fine.