I had a kinda similar issue where Maya would start up and just keep eating RAM until it brought the system down, but that was when my video card RAM was crapping out, so it could or could not be hardware issues, run a check.