In this paper

a novel edge detection algorithm based on cellular neural networks (CNN) is proposed for nucleated cell detection. This new algorithm applies particle swarm optimization (PSO) to design the CNN templates to identify the edge of a nucleated cell. In order to overcome the premature phenomenon of PSO

the variance of populations fitness is calculated

and chaos optimization theory is applied to enhance the PSO’s global optimization. According to the characteristics of nucleated cell

a three-step study strategy is specially designed to obtain the best CNN templates. Experimental results show the new algorithm is effective; its edge fitness rates and checkout rates are better than former algorithms.