Chimeric antigen receptor T (CAR-T) cell immunotherapy is a novel method that is genetically engineered to recruit T cells against malignant disease. Application of CAR-T cell immunotherapy has made great progress in hematological malignancies, hence, it has been proposed for solid tumors like colorectal cancer (CRC) by many researchers. However, the efficacy and application of this method in CRC are limited for the intrinsic challenges posed to CAR-T cells, including the lack of tumor-specific antigens, the tight connection of tumor cells, and unfavorable tumor microenvironment, as well as the side effects. New approaches are proposed to overcome the multiple challenges to alleviate the difficult situation of CAR-T cells in CRC, each of which has its advantages and disadvantages. In this review, we sum up the current stage of knowledge about target-selection, adverse events, the tumor microenvironment and corresponding optimization strategies, and the future prospect of CAR-T cell immunotherapy for solid tumors is also presented.