Perhaps it's the ratio of illumination on your sun vs the hdri & portal lights? I'm not sure, but look at the illumination and shadow aside from the illumination of coming from the sun. I'd expect more variety throughout the whole room when you change the sun's position. Aside from the obvious sun patch, not much is changing.