Just looking at your settings, I'd have to agree, that's looking a bit odd for a frost glass preset. I've never tried using frost glass with an HDRI image though. Adjusting the refraction does adjust the way the light bends (changes direction) as it goes through the glass but still, I would think the glass would at least keep the frosty look even if the refraction was too high or too low.
Don't be satisfied with what you can do but rather strive to do the things you can't do!
Exceed Expectations!