Multiple-input multiple-output （MIMO） systems improve the bit error rate （BER） performance

meanwhile

in order to reduce the impact to system capacity

a novel antenna selection algorithm for space-time block coding （STBC） and vertical Bell laboratories layered space time （VBLAST） hybrid coding systems was proposed. Employing the Alamouti STBC code for some VBLAST layers

the STBC and VBLAST hybrid coding MIMO system has the advantages of both STBC and VBLSAT. On the criterion of maximal antenna-subset capacity

the proposed antenna selection algorithm selects those antennas with less contribution to the system capacity to send multilayer STBC signals

so that the system capacity loss by STBC coding can be reduced. Both theoretical analysis and simulation results indicate that the proposed antenna selection algorithm increases the capacity of the STBC and VBLAST hybrid system effectively

and improves the BER remarkably comparing with a VBLAST encoding MIMO system.