Background: Older adults tend to spend more time watching (TV). While frequent TV watching is known to be associated with poor cardiopulmonary function and muscle strength in children and young adults, the relationships between TV viewing and physical fitness and psychological symptoms in older adults have remained understudied. The paper accordingly aimed to explore the relationship of TV viewing time with physical fitness and psychological symptoms among community-dwelling older adults. Methods: In this cross-sectional study, 340 older participants aged ≥65 years were voluntarily recruited during 2014-2015. Based on the average daily TV viewing time, the participants were stratified into two groups: infrequent (≤2 h/day) and frequent (＞2 h/day) TV viewers. Physical fitness of the participants was measured with 30-second chair stand, 30-second arm curl, 2-minute step, 8-foot up and go, back scratch, and one-leg stand tests. Self-reported health and BSRS (brief symptom rating scale) scores were also collected. Results: Frequent TV viewers had a higher body mass index (BMI) than infrequent TV viewers. When controlling age, sex, and weight, the study found frequent TV viewers show a higher BSRS score, experience greater difficulty falling asleep, and sleep less. However, there was no significant difference in physical fitness between the two groups of participants. Conclusions: Longer TV viewing time appears to be associated with higher BMI, less sleep hours, and more psychological symptoms among elderly community dwellers, indicating a need for interventions to prevent excessive TV viewing in older adults.