A novel three-dimensional CFD simulation of H2 gas permeation through dense palladium (Pd) membrane was developed. Due to discontinuity of flow in a membrane, usually, gas diffusion process is simulated by introducing source and sink terms. In a novel approach, an analogy between heat and mass transfer is considered. The most important advantage of this approach is that there is no need to define sink and source terms, and the membrane thickness is considered as a solution domain without separating the geometry adjacent to the membrane. Also, it allows the modeling of multilayer membranes with different mechanisms of diffusion, separately. The effect of membrane geometry on the hydrogen separation was investigated using the straight and helical modules by defining user-defined function (UDF) and user-defined scalars (UDS). The results showed an average flux and H2 recovery enhancement of 20% and 13% for helical configuration, respectively. The influence of the feed gas and sweep gas flow rates, helix pitch, coil diameter, pressure difference, and module temperature on hydrogen separation was also investigated. The proposed simulation model was validated using the experimental data. The results indicated that this method has a maximum error of about 10% for H2 flux.