Looks like you might have the gravity on the cloth to high, try lowering it down a bit.. so the ball would have time to leave before the cloth would react..
Still thinking on other possiblities
I am enough of an artist to draw freely upon my imagination, knowledge is limited, imagination encircles the world. (Albert Einstein)
https://www.artstation.com/kurtb