Lots of energy is consumed heating loosely-enclosed spaces like houses and apartments. All of this heat is eventually absorbed into the building materials and then lost to the surrounding environment. The effectiveness of a building's "shell" determines how long this heat transfer takes. The size of the space and the efficiency of the shell around that space determine how much more energy has to be added to maintain a particular temperature.
For a fixed-sized space, the least efficient but cheapest way to heat is to use electric resistance heating (e.g. baseboards connected to household wiring). Burning natural gas / propane is generally more efficient because the heat loss at the thermal power plant is avoided. In moderate to warm climates, using an electric heat pump to move heat in and out of a space can be as efficient or more efficient that burning gas. Using electricity also opens up the possibility of tapping nearby renewable energy sources.
If you can cut down on the amount of space you are heating, less energy will be needed to heat that space. Heating 25% of a space with electric-resistance should be roughly equivalent to heating the whole space with a high-efficiency gas furnace and tightly-sealed duct system. If you duct system is not well-sealed (new ducts usually leak 20+%) or if you have an older furnace (<80% efficient), electric-resistance space heating may save energy and money.
Radiant heating (like heat lamps or "dish" heaters) can help you feel warm for brief periods of time without heating up all the air in a room. In mild climates, hot water bottles, wool socks, good blankets (nothing like a down comforter!), sleeping hats, and Energy Star curtains (and of course sealing and insulation) can all help eliminate the need for night-time heat.