Yes, you can set the memory limit, however as I know it works just for rendering via mentalRay, to as much or little as you like, but do not forget to leave something for OS, as well. Under Render options panel you should see Render->Batch Render; click on that small box next to the title; in Memory section tick off Auto Memory Limit and write below how much MB of memory you allow to Maya to use.