Motion vector plays one significant feature in MPEG-4 video object segmentation. However, the motion vector in this application is required to represent the actual motion displacement, rather than regions of visually significant similarity. In this paper, Optical Flow Back-projection (OFB) which back-projects optical flows in a cluster to restore the region's motion vector from gradient-based optical flows, is proposed to obtain genuine motion displacement. The back-projection is performed based on minimizing the projection mean square errors of the motion vector on gradient directions. As Optical flows of various magnitudes and directions provide various degrees of reliability in the genuine motion restoration, the optical flows to be used in the OFB are selected based on high gradient. With this approach, the interference by similar blocks, which frequently results in irrational motion displacement in traditional block matching approach, can be avoided and more reliable motion vectors can be obtained.