Memristor-Based Light-Weight Transformer Circuit Implementation for Speech Recognizing

Transformer network (TN) is a promising model widely used for natural language processing (NLP), computer vision (CV), and audio processing (AP). However, the large number of multiples and additions (multi-adds) in TN leads to a massive number of parameters in the TN model, causing an increasing number of hardware resource overhead and thus making it difficult to deploy the TN model to edge devices. To alleviate this problem, this paper proposes a circuit implementation scheme for a memristor-based lightweight transformer network (MLTN). In addition, the speech recognition task based on the speech command dataset is implemented through the modular hardware design approach, which provides an effective lightweight hardware implementation scheme for the edge devices. Further, the retention, noise immunity, variability, and stability of the designed circuit are analyzed by simulation experiments with LTSPICE. The results show that the proposed MLTN circuit not only has a lighter network model but also has significant recognition accuracy with different hardware scales. Specifically, MLTN-40 can achieve 96.18% simulation accuracy, while the energy overhead is only 3.8% of the ARM framework, which is commonly used for edge devices, and 39.9% of the traditional server (GPU), making it a promising solution for implementing speech recognition on edge devices.