The time factor of a baseball game is one component but it's also the length of the season. Until the All-star break if you miss one game, a week, or a month the standings are within reach for the bulk of the teams minus the outliers. Life happens and at the end of the day only the most devoted watch even 81 games of the season which is still only half their total. Contrast to the NFL the average fan can easily catch 8 games or more of the regular season.
Some sports such as golf, tennis, or nascar set up better for single events and other must see showdowns.
For many of us, I assume we keep a pulse on our individual secondary sport preferences by default ( general sport talk on the websites/sports talk shows) and tend to tune in around the playoffs. If there is a story line or personal tie in to a player/sport we pay more attention for that period of time then let it go back by the wayside.