Nowadays there is a vast interest in a self-driving car from both academia and industry. The main reason behind recently enormous progress in deep learning approaches for an autonomous vehicle. The main objective of this research is to propose a deep hybrid encoder-decoder network with input multi-modal data to predict the decision-making task. Therefore, the proposed approaches are tested by both real and simulation data but in the real data single camera image and simulator data three-camera image data. The proposed method analyzes the effects of input data. The experiment results in analyses in terms of Computational time as-well-as parameters in which values of the steering wheel and brake both real and simulated data are (6ms and 9ms) respectively. The analysis shows that our method performs well in driving action prediction.