Huhh. Strange. If you are using the root user, you should be allowed to use that command. I have several times, but then again, I run my MineOS on top of an Ubuntu Server OS.
If Vanilla runs smoothly, I can only suggest that there is one or more of the mods in yout FTB-server that are to resource-heavy for your hardware, and more spesifically: Your CPU. Ram here has little to do with it unfortunately. @tNt may be able to help you streamline your FTB / Modded server, I can only reccomend better hardware if you wish to run FTB…
Sorry.