I think part of the problem is that the displacement map currently has a lot of shadowing in the image, I can see why using it as is could cause accuracy issues. If you have Photoshop, you can try applying a high pass filter with a low radius to the image. I use that all the time for generating the base for bump maps.