Inspired by the lateral inhibition in biological neuron networks, we propose a novel neural network architecture, namely locally connected deep neural network (LCNN), which imposes local connections within the neighboring neurons in the same layer in order to imitate the lateral inhibition.