Usually what you'd do is create a "bound mesh" and a "rig mesh". The rig mesh would be a low poly version of your model, with no skinning. You would just parent constrain the objects to your skeleton. Therefore you can animate as quickly as possible, and not be distracted with skinning.
You would then create a higher resolution version that you would do you final skinning, texturing, and rigging on; then transfer the animation from the rig mesh to the bound.
Would you recommend to crank up the poly counts a tide more for animation? Or is good to go ?No problem, that's what we're here for
From what I see, I'd say it's more than low enough.