Looking at your grid size I wouldn't say it was that big.
I think it would be fair to say that you need some reasonable scale in order to do the calculations. Its the same with painting weights and sculpting too, I've found that too small a model is fiddly to weight stuff as the values are so small on the brush etc it becomes more difficult to paint as with particles too, default is always 1. Also with large landscapes you have to increase your displacement quite alot if you make hills etc.

When I started doing Maya coming from Lightwave we had to work out scale so we used 'ahem' a poser figure as a reference.
Worked it out that 1m(default) grid spaces in lightwave were the equivelent of about a cm in Maya!! So after a bit of working out we had to scale the character 200% in Maya in order for it to work as a world setting!! ie;10grid spaces were the same as 1m. Maya takes this into account so you may have to up your settings accordingly.

Anyway less of my crap, I have had a bit of a go at cloth and it seems ( excuse the pun) that its just slow the more intricate you go and the same ol story of you need more ram and processor power and patience!!. The best one I have used is the Syflex plugin which is really fast compared to Maya.
Are you using stitched panels or is it a poly object? As this may make a difference also!!

_J