OS == Operating System. It is the software that runs the system. Windows is an Operating System, as are Mac OS X, Linux, and Solaris. Android devices run the Linux OS. Linux is open source, and Google maintains that openness with Android.
ROM == Read Only Memory. It usually describes a piece of hardware, a memory chip that you normally can't write to. Device manufacturers will typically use ROM to store the OS to help ensure the devices maintains a known state and is thus more reliable and easier to support.
Rooting your phone is a way to give yourself enough access to write to protected parts of the file system and alter the OS. People tend to use the phrase ROM in these types of forums to refer to a complete image that contains an OS which you can load on your phone.