In this paper, a steganalysis algorithm is proposed based on Modified Graph Clustering Based Ant Colony Optimization (MGCACO) feature selection and Random Forest classifier. First, different features related to the steganalysis problem are extracted from each image, and then an optimal set of the extracted features is selected by using the MGCACO feature selection algorithm, and finally a trained classifier used to separate the clean images from the steganography images. Our proposed algorithm is compared with four steganography algorithms including least significant bit matching (LSB), highly undetectable steganography (HUGO), wavelet obtained weights (WOW) and spatial-universal relative wavelet distortion (S_UNIWARD) with different embedding rates such as 0.1, 0.2, 0.3 and 0.4. Moreover, as a new study, the types of steganography algorithms are identified by using the proposed algorithm. The results of the proposed algorithm show that our approach can distinguish between clean and steganography images acceptably and, in addition, this algorithm can detect the type of steganography algorithm with an average accuracy of 90%.