Ethernet connection failing in XB7

So i recentky moved to a new construction and they gave me different ethernet ports around the house. I mainly want to use 3 (gaming, TV, and Work) but when I connect one the other stops working or sometimes none of them work. It is frustrating because now every time I am going to do something I need to go to the modem and change the cable to the one I need.