There is something not right for those with issues as bad as I have read in this thread. If the battery life was as bad for everyone as posted here we'd know about it. Either the phone is defective, the battery is defective or there is something chewing up battery that shouldn't (app, OS or even hardware issue). If it were me I would be getting a replacement phone if the issues were that serious. Probably do a factory reset a couple times first. In any case, this is not an android issue or note 4 issue but something wrong with a few specific phones. Happens to all brands, apple, androids, etc.
Before I go jumping off that cliff, I would spend some time looking through the settings like others have said.
This is OP's first Android. It is very possible that all the apps and accounts added to the phone are set to "always sync" type settings that keep the system pulling on the battery.
Google location services draws a lot of battery, and it is enabled unknowingly by many users, experienced and novice alike.
OP, you have gotten a few really good suggestions for settings to check. Look through those to make sure you have not overlooked anything.
I also recommend having it on a charger when you aren't using it whenever possible.
I have a spare battery and charger for my S4. I also keep it on the charger in my car (a requirement for using Waze) and in my office.
I get better battery performance than my wife does with her iPhone. But I still manage the charging as much as I can.
EDIT: OP, check out this article from the Phandroid page:
13 things every Samsung Galaxy Note 4 owner should do
There are a lot of good tips in the article for setting up your phone, especially since this is your first android.