The defects and optical properties of nonpolar (1010) ZnO and Zn 0.9Mg 0.1O epilayers grown on (100) γ-LiAlO 2 substrates by chemical vapor deposition were investigated by x-ray diffraction, transmission electron microscopy and cathodoluminescence. The x-ray rocking curves of the (1100) reflection for the ZnO and Zn 0.9Mg 0.1O epilayers possessed low FWHM values of 504 and 216 arcsec, respectively. Misfit dislocations in high density exhibited at the epilayer/substrate interface with b= 1/3  for both samples. Basal stacking faults, threading dislocations and inversion domains were observed in the ZnO epilayer with densities on the order of 10 -5 cm -2, 10 -9 cm -2 and 10 -4 cm -1, respectively. On the other hand, only basal stacking faults and threading dislocations were found in the Zn 0.9Mg 0.1O epilayers. The densities of these defects were at least one order of magnitude lower than those of the ZnO one. The Zn 0.9Mg 0.1O epilayers exhibited a near band-edge emission at 3.5 eV whose intensity is about 15 times stronger than that of the ZnO epilayer at 3.3 eV. Monochromatic cathodoluminescence images acquired at the peak energy revealed that the basal stacking faults showed low emission intensities.