I believe this is more of a firewall rule, but is there a way to setup port forwarding to only allow forwarding if the request is from a certain IP address ?
On higher end home units and entry level buisness units yes absolutley. You can restrict communication to a certain port from an IP / IP range. Not sure if you have a Comcast router or not but if so here is the rules on port forwarding.
From what I can tell it does not look like you ca restrict by IP but I would contact Comcast support to see if they can do it on their end as they have a bit more access to things than the end user.
If you want to do this and Comcast's router does not support it the cheapest option I could think of would be an Ubiquiti Unifi Security Gateway, you would have to get a seperate switch and access point to have a full network just an FYI. https://unifi-sdn.ui.com/