If I put a magnet on my fridge it doesn't wear out and fall off.
This is incorrect. While the process is slow, magnets do wear out over time.
Yeah, I was expecting this.
Magnets degrade, this is different than the magnetic energy being 'used up' when attracting each other or metal objects.
Magnets degrade over time due to entropy as the magnetic domains randomly fall out of alignment, this is not the same as burning fuel or consuming electricity.
Magnets do not expend energy to hold themselves or other objects in place. Again, if you hold two magnets close together in your hands you have to strain to keep them apart, but they are not expending any energy. Just like holding a rock over the surface of the Earth isn't causing the rock to expend energy to pull downward.
Both magnets and rocks will continue to pull without using any energy until another process degrades their physical structures. Magnetic domains flipping, or the atoms spontaneously decaying in normal matter.
Edit:
Actually, forget all that. The magnetic fields do NOT decay for a simple reason. A magnet's field is made up of tiny diopoles all aligned in the same direction. When a magnet 'degrades', those dipoles simply get randomized directions, but the magnetic fields do not go away, they are just disorganized. The magnet will always contain the same amount of magnetic force, it's simply the directions that alter.
So magnetic force is never used up, it's just pointed in other directions. There is no energy expenditure for a magnet to exert a force. I think that's a more concise argument than my first above. This is why waiting to post is never a bad idea...