I think you both have to grow up more, your adults, and shouldn't be acting like you both are, if this man loved you he wouldn't have answered so in such a negitive manner, he's telling you no matter what you do it's probably not ever going to work out, if you move to where he wanted to move to, he's telling you it may not work out anyway, when ever you had words between each other, his first retort will be i told you it wouldn't work out, now what, move back, i would stay where you are and find love there !!!
But whatever you decide, good luck, i hope it works out for you !!
Sincerly
Rocco