With the exceptional mechanical properties, carbon nanotubes (CNTs) are considered to be attractive candidate reinforcements for composite materials and to have potential applications in improving the energy absorption capability of matrix material. However, it is still difficult to reveal the micro-mechanisms of the impact energy absorption of CNT-reinforced composites by experiments, hence, the numerical investigation is helpful. In this paper, a unit cell of single-walled CNTs (SWCNTs) embedded in metal matrix is modeled by nano-scale finite element method. Under impact loads, the failure modes of a single SWCNT and the SWCNT in matrix are predicted, respectively, and several possible energy absorption mechanisms are explained and compared. The investigation shows that, the metal matrix restraints the radial expansion of the SWCNT and therefore improves its crush buckling resistance, and makes it absorb more energy before collapse. The specific energy absorption of SWCNTs-reinforce composites increases with the increasing volume fraction of SWCNTs in both matrixes, and ascends more quickly in magnesium alloy than in aluminum alloy matrix.