This paper proposes a hierarchical switching strategy for mobile robots to detect obstacles and to track a given target using vision information only. In the proposed approach, the disparity image obtained from a stereo camera is used to build an obstacle map based on V-disparity. Color information and a self windowing technique are employed to track the target. A switching strategy based on a hierarchical architecture is employed to determine whether the mobile robot should perform obstacle avoidance or target tracking. To verify the effectiveness of the proposed approach, several experiments were conducted. The results indicate that the proposed approach is feasible.