The onset of miscible viscous fingering in porous media was analyzed theoretically. The linear stability equations were derived in the self-similar domain, and solved through the modal and non-modal analyses. In the nonmodal analysis, adjoint equations were derived using the Lagrangian multiplier technique. Through the non-modal analysis, we show that initially the system is unconditionally stable even in the unfavorable viscosity distribution, and there exists the most unstable initial disturbance. To relate the theoretical predictions with the experimental work, nonlinear direct numerical simulations were also conducted. The present stability condition explains the system more reasonably than the previous results based on the conventional quasi-steady state approximation.