Since the application of structural control systems on nonlinear structures has been limited hence, in this paper designing active mass damper(AMD) for improving the seismic response of nonlinear frames has been studied.To this end, an eight-story steel  frame with bilinear hysteretic behavior has been subjected to white noise excitation and AMD system has been designed by using the nonlinear instantaneous optimal control algorithm.For optimization of control system to minimize the maximum drift of structure, consequently the internal forces of the structural elements  when a limited capacity has been considered for actuator,distributed genetic algorithm(DGA) has been applied. Also some practical limitations such as actuator capacity and AMD stroke length have been considered in design processes.The results of numerical simulations show the effectiveness of the proposed method in designing optimal AMDs for nonlinear frames. Also comparing the performane of AMD sysyem with that of passive mass damper(TMD) and active tendon control(ATC) shows that the AMD has been more effective in reducing the seismic response of nonlinear frames.