Mainly due to its propellant non-mixing feature, hybrid rocket propulsion has been demonstrated to be more advantageous in operation safety as compared to its solid and liquid counterparts. The traditionally moderate Isp output of hybrid rockets has been enhanced to be close to the liquid rocket performance in recent years, particularly with the innovative designs employed in this research by using dual-vortical-flow (DVF) chambers. Based on this new approach and cost saving strategy, a multifunction rocket system is designed with the features of high performance hybrid combustion, trajectory following flight controls, enhanced science experiments, and an advanced payload recovery method. High fidelity numerical modeling design approach and hot-fire experiments are employed to assess the overall performance of the DVF hybrid rocket engines that has roll control capability embedded in the engine design. The present hybrid rocket engine designs consider propellant systems of N2O/HTPB, N2O/HDPE and H2O2/HDPE. Pressure-fed system is the baseline for delivering the oxidizer to the combustion chamber while pump-fed system is also considered as a design option, especially for the hydrogen peroxide system. Carbon fiber filament winding pressure tank is incorporated to contain the oxidizer. Pressurant is also employed for better thrust control. To enhance the overall performance and benefits of conducting flight experiments using hybrid rocket, three basic flight trajectory designs are proposed in this study, namely the traditional standard parabolic trajectory, a TASE (Trajectory Augmented Science Experiments) maneuver and a HOOK (Homing Oriented Operation Kernel) maneuver. The TASE maneuver is designed for maximizing the measurement capabilities of the instruments for atmospheric and ionosphere data profiles. The HOOK maneuver is aiming at improving the success in science payload recovery and in reducing the search and recovery efforts. To achieve these goals, a high performance and reliable flight control system is critical, that incorporates the throttling capability of the DVF hybrid rocket engine, which is one of the key development aspects of this study. For the numerical modeling of the internal ballistics of hybrid rocket combustion for flow analysis and design optimization, a multiphysics Navier-Stokes flow solver with finite-rate chemistry, real-fluid properties, turbulence model and radiative transfer model is employed for high resolution computations. This numerical model is also incorporated in analyzing the aerothermodynamics for high-speed ascend and reentry flights. A 6-DOF flight dynamics, navigation and control simulator is employed in assessing the overall performance of the vehicle based on the aerodynamics and propulsion data generated by the flow solver. Results of the numerical analyses are validated by measured data of ground and flight tests.