In this study, cyclic olefin copolymer (COC) composite membranes were prepared with melt processing method by using of various types of graphite nano-sheets namely graphite oxide (GO), octadecylamine (GO-ODA) and stearic acid (GO-SA) modified graphite oxides. Structural and physical characterization of the modified nano-sheets and membranes were conducted by contact angle measurements, XRD, FTIR, TGA, DMA and SEM analysis. Gas permeability (P) and ideal gas selectivity values (a) of membranes for the H-2/CO2 and H-2/CH4 were quantified depending on the structural features and the content of fillers. It was found that increasing amount of filler resulted in reducing of CO2 and CH4 permeability for all series of membranes. On the other hand, the H-2 permeability values of membranes decreased with the increasing amount of filler in the sample series prepared with GO and GO-ODA but increased in the samples containing of GO-SA. It was also found that the ideal selectivity values of the membrane included 5 wt% of GO-SA for the H-2/CH4 and H-2/CO2 were drastically improved to 498 and 38.6, respectively. Consequently, the ideal selectivity of GO-SA base membrane was increased by 57.5% for H-2/CO2 and by 280% for H-2/CH4 separation compared to the COC film. The superior separation performances of the system surpass the current permeability-selectivity tradeoff limits (Robeson, 2008 upper bound) for H-2/CO2 separation. (C) 2016 The Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved.