So I figured out if I create a new ortho camera, rotate it 45 degrees in Y and 30 degrees down (x) and do the same for my reference image, it works out pretty nicely. (My model doesn't exactly match up cause I'm taking the front and side ref images as more accurate).