Soil compaction is one of those steps everyone wants to skip until their fence looks drunk. Without a solid base, rain and frost will push those posts around no matter how deep you dig. Rent a plate compactor for an afternoon instead of pretending to stomp it down with your boots. It's a boring job, but fixing a leaning fence is a much bigger pain. Good foundations are boring until you don't have one.
Honestly how many people actually rent machinery for a fence? Most backyard fences I see are put in with some gravel and a lot of stomping. They seem to hold up fine for years unless the ground is pure swamp. Maybe it's overkill for a simple privacy fence.