Latency at 50 average with 38 upload?

I have decent upload and download speed but my ping is always at 40 or 50 for my games. Im used to getting around 12-20. Is there any known reason for this? I’ve been port forwarding and tried not to. Im not too for sure why. Any suggestions?