Training neural networks (NN) over conventional DRAM-based edge devices are highly restricted due to DRAM's leakage-power and limited-density properties. The non-volatile memory (NVM) reveals its potential of providing solutions with the high-density and nearly-zero leakage power features; however, it could lead to severe performance and lifetime issues. This paper focuses on exploring the NVM-aware neural network training design to mitigate performance and lifetime issues caused by the conventional NN approaches. Especially, we aim at exploiting the restrained-and-approximate managements to leverage insignificant data on performance and lifetime optimization. The encouraging results were observed with the comparable validation accuracy in the evaluations.