During my prime in the 1970s, I always started off the year in great physical shape. I didn't go to the gym during the off-season; few guys back then did. What I did was hunt and fish a lot. I'd plow through marshes in my heavy waders, trudge through snow and across hilly terrain, always making a point of walking as briskly as I could. When I showed up to play in January, my legs were developed and felt strong.
Having my legs in condition helped my swing, improved my balance and gave me stamina. If you want to maintain your conditioning but loathe going to the gym, grab your iPod and try hiking in a wooded area, or at least walking vigorously around your neighborhood.