LAN to LAN router setup

That is not just a straight cable modem. It is a combination modem and router gateway device. If you are going to hang a stand-alone / separate router off of a gateway device, and the gateway is not in *bridge mode*, you are cascading 2 routing devices which results in an undesirable "double NAT" condition on your home network.

Google "double NAT" for info about it's potential pitfalls.

You really should put it in to bridge mode. Here's how;

https://www.xfinity.com/support/articles/wireless-gateway-enable-disable-bridge-mode

Only LAN port number 1 will be active after the gateway is bridged.

If bridging is your only answer your wasting my time. Your discription says I lose ports 2-4 and only have the 4 on the router. I found an article on the web about lan to lan and already have it working now I have 7 hardwired lines and no problems. I plugged the second router into port 1 and everything works ok. I wasn't asking how to do it I was only asking which port I should use #1 or the indexed #4 with red line, but # 1 seems ok. 

Sorry to waste your time pal.............. Have a ball.............

Port 4 on that gateway is a 2.5 Gb ethernet port. 

Since you somehow got your setup to work without bridging the gateway, I guess you'll have to experiment and find out if that works.