Make sure the ISP is not blocking port 22. If they block it, you will need to change the SSH port in sshd_config and then set the router to forward the port to the server's internal IP address. It's a good idea to change the port anyway, in order not to be obvious to script kiddies.