Neural Machine Translation Model Optimization via Adversarial Learning

Neural Machine Translation (NMT) is a new approach that has replaced the older statistical method, and it uses deep neural networks to learn how to map source and target languages. At the basic level, NMT systems are based on the idea of sequence-to-sequence learning and usually have an encoder-decoder structure. The encoder in this framework is used to obtain a fixed-length or continuous vector representation of the input sentence that contains the semantic information of the source text. Then, the decoder takes this representation and generates the translation in the target language token by token, predicting the most probable word at each step based on the previous context. The whole model can be optimised at once in this way, and it is better at handling long-range dependencies and complex linguistic structures than the old phrase-based system. Although there have been some developments, the standard NMT model is still often trained by Maximum Likelihood Estimation (MLE), and thus the training goal does not align with the actual index of translation quality. MLE aims to maximise the probability of generating the ground-truth reference sentence, but it is prone to the "exposure bias" problem; that is, errors accumulate during inference because the model has never seen its own predictions in training. In addition, the loss function used for training is different from the evaluation metric BLEU score, so it may not be able to produce very fluent and accurate results [4]. Due to the problems mentioned above, Adversarial Learning has begun to be used for optimisation. Adversarial learning adds a competitive element to the training process and is often based on the concept of Generative Adversarial Networks to address the problem of a mismatch between training goals and evaluation indicators. Therefore, the translation model is a generator that attempts to produce translations that are indistinguishable from human-generated text, and a discriminator network is trained to distinguish between the model's output and real reference sentences. The above process is a minimax problem, so to fool the discriminator, the generator needs to improve both the fluency and accuracy of its output. According to the adversarial feedback, the parameters of the model can be adjusted based on criteria that are closer to human judgment to reduce exposure bias and improve the robustness of the translation system. This optimisation has had some good results in practice and has addressed the problem that NMT systems are weak in handling low-frequency words, rare syntactic structures and domain-specific vocabulary, with demonstrated performance gains of up to 4 BLEU points on low-resource language pairs and strong robustness for zero-shot unsupervised translation scenarios [1]. By increasing the difficulty of the training through adversarial methods, the generalisation ability of the model has been improved, and thus it can be used more reliably for cross-language communication in professional and technical areas.

The theoretical basis of this study is the combination of a sequence-to-sequence architecture and GANs. The typical structure of a standard Neural Machine Translation (NMT) system is an encoder-decoder architecture; that is to say, it can convert a source sentence into a context vector, and then a decoder is used to generate the target sentence one token at a time by maximising the log-likelihood of the correct sequence. The maximum likelihood estimation (MLE) method is relatively simple to use but may be too conservative in translation. Therefore, adversarial learning has added a discriminator network to the sequence-to-sequence model and is formulated as a minimax problem. However, there is a basic difference between adversarial optimisation in discrete translation tasks and the original continuous image generation GANs. Image generation has continuous pixel values and can therefore transmit gradients directly; translation tasks consist of discrete text tokens that are not differentiable, so backpropagation from the discriminator to the generator cannot be performed.

The three problems that have not been solved in the theory are as follows: First, there is a mismatch between the discrete generated tokens and the differentiability requirement of the standard GAN training objective, so the GAN loss function cannot be used directly. Second, there is a problem of mode collapse in adversarial NMT; that is to say, the generator may produce repeated or semantically empty translations to deceive the discriminator and thus reduce linguistic diversity. Third, the theoretical boundary for the application of adversarial regularization to improve generalization performance is not clear, and thus it is difficult to determine how much better adversarial training is than standard MLE in terms of robustness and accuracy.

To solve the problems mentioned above, a formal mathematical problem formulation of the optimised adversarial NMT framework is given in this thesis. Let the generator be $G_{\theta}$ and the discriminator be $D_{\phi}$. The goal is not only to maximise the likelihood but also to find a Nash equilibrium for the value function $V(G, D)$. A standard adversarial loss is employed for sequence-level training. The general optimisation goal of the generator $G$ is to:

$$\min_{\theta} \mathbb{E}_{(X, Y) \sim p_{data}} [L_{NMT}(Y, G_{\theta}(X))] + \lambda \mathbb{E}_{X \sim p_{data}} [(1 - D_{\phi}(G_{\theta}(X)))^2]$$

where $L_{NMT}$ is the traditional cross-entropy loss for sequence generation, $\lambda$ is a weighting factor for adversarial regularisation, and the second term is an adversarial loss that encourages the generator to produce translations that are difficult to distinguish from those written by humans. At the same time, the discriminator $D$ is trained to be less likely to assign a high score to the generated sequence and still give a high score to the real sequence. The above formulation can connect the generation of discrete tokens with differentiable adversarial training and provide a good theoretical basis for the construction of the model and experiments.

The Design of the task-specific adversarial generator is based on the general-purpose adversarial framework that was first used in computer vision. The first problem with directly applying the visual-domain generator is that it is a discrete autoregressive model for sequence-to-sequence translation. Standard Neural Machine Translation (NMT) is not a continuous image pixel space; thus, it is generated step by step in terms of tokens, and standard gradient-based adversarial attacks are not differentiable. Therefore, the proposed generator is to be in line with the autoregressive decoding logic and should not disrupt the sequential generation process due to adversarial perturbations.

Structurally, the proposed model is a modification of the traditional Transformer-based encoder-decoder backbone, and a light adversarial perturbation embedding module is added directly after the decoding input layer. An addition can be made to the generator to add a learnable noise vector to the embedding space and create small changes in the hidden state without changing the meaning of the original sentence. The first way is to apply the adversarial effect at the level of feature representation; it is not to modify the original discrete tokens. A special reward function is employed in the generator to combine the traditional Maximum Likelihood Estimation (MEL) signal with adversarial feedback from the discriminator. The combined objective is to maximise the probability of a correct translation and also to fool the discriminator, so there is a dynamic tension that drives the improvement of the model.

All the forward-propagation steps are to generate many different candidate outputs that are semantically consistent and translationally accurate for the adversarial game. In the decoding stage, the generator will take the source sequence from the encoder and then, step by step, build the target sequence by using perturbation embeddings to generate many candidates. The optimisation goal of the generator is to minimise the negative log-likelihood of the target sequence and at the same time minimise the adversarial loss from the discriminator. The total generator loss function is the sum of the supervised translation loss and the adversarial reward term, and it is formally expressed as the negative expectation of the discriminator's log-probability for the generated sequence. Through strict mathematical derivation, a stable gradient path for sequence-level backpropagation has been obtained; thus, the generator can learn to produce high-fidelity translations that are robust against adversarial judgment and will be used in the next step for integration with the discriminator and gradient penalty regularization.

To solve the problem that the current token-level or partial-sequence discriminators cannot effectively judge whether the whole meaning of the source input sentence is consistent with that of the target generated translation, a special Sentence-Level Semantic Consistency Adversarial Discriminator has been added in this paper. The former way was to focus on the discriminative features of individual words; now, it is to judge the overall semantic consistency of the translated pair. The structure of the discriminator is based on putting both the full source language sentence and the target language candidate translation into the same cross-lingual semantic space. This mapping will place the vector representations of the same meaning in different languages close to each other, and thus it is convenient to compare the semantic content directly instead of just the surface pattern. A multi-layer Transformer interaction network is used in the shared space to learn the deep contextual dependencies and long-range correlations of the two sequences. A global semantic consistency degree is obtained by calculating the interaction of the source and target embeddings in the network; thus, it can be determined how well the target sentence retains the meaning of the source. Based on all the above analysis, the discriminator will output a continuous authenticity score to distinguish high-quality human-authored reference translations from model-generated adversarial translations. A problem in this Design is that the original binary classification goal of GAN discriminators has been removed. It is no longer enough to label the input as real or fake; instead, we want to know how closely the cross-lingual semantics of the whole sentence match. Therefore, the training goal is to minimise the semantic distance between the source and target distributions, and regression-based loss functions or modified adversarial criteria that reflect semantic similarity can be used. The information flow logic starts with the encoding of a whole sentence pair, then goes through the interactive Transformer layers to create a joint representation, and finally reaches the output layer to produce a scalar consistency score. The above feedback mechanism will provide the generator with a small gradient in the direction of the task, and thus the optimisation process will be able to produce translations that are both smooth and accurate to the original text.

To address the problem of training instability in the previously built adversarial Neural Machine Translation (NMT) system, a gradient penalty mechanism has been added. To understand why this optimisation is needed, one must first know the reasons for the instability of the base model. The training process is fundamentally limited by the problem of gradient vanishing or explosion, which is due to the discrete and high-dimensional nature of translation tokens. In addition, the original GAN framework has strict Lipschitz constraints and is difficult to meet with standard weight clipping; thus, it often has irregular gradient behaviour and cannot be trained well.

Therefore, a customised gradient penalty term has been added to the sentence-level semantic discriminator to address the above problems. Instead of just clipping the weights, it is now directly penalising the norm of the discriminator's gradients. The penalty term is the gradient norm of the discriminator's output with respect to the concatenated source-target sentence embedding pairs. Therefore, it can be ensured that the underlying geometric constraints for stable adversarial optimisation are met. A regularisation term is added to keep the gradient norm of the discriminator around 1, and thus it is 1-Lipschitz continuous. Therefore, the rate of change of the function will be relatively small in the data space, and it will not increase too much or decrease to zero during backpropagation.

Therefore, the total loss function of the whole adversarial NMT system is now composed of three parts: the generator's standard translation loss, which ensures the accuracy of the output; the discriminator's semantic matching loss, which determines how natural the generated translations are; and a new gradient penalty term that has been added to control the stability of training. Finally, in order to realise this Design, an alternative two-step training process is employed. In this iteration, the discriminator is trained to distinguish between real data and generated data under the gradient penalty condition, and then the generator is trained to produce translations that can fool the discriminator. The above arrangement will help to maintain a good state of training and improve the quality of translated texts by enhancing stability and language expression.

To guarantee that the empirical verification is stringent, reliable and in line with scientific principles, a full-featured experiment will be set up; first of all, an appropriate dataset covering high-resource and low-resource language pairs should be chosen to test the generalisation ability of the model comprehensively. The WMT 2014 English-German and WMT 2017 English-Chinese benchmarks are used for the high-resource case, and a large number of parallel sentences can be obtained from them to train deep neural networks. TED Talks Urdu-English and IWSLT 2021 Lithuanian-English are the other two datasets that are used to assess the performance of data-scarce environments. The preprocessing workflow is the same for all experiments to guarantee the consistency of the data; that is to say, tokenisation is carried out by language-specific scripts, and then subword segmentation is performed using Byte Pair Encoding (BPE) to reduce the vocabulary size to 32,000 tokens and address the open vocabulary problem by breaking down rare words into subword units. Next, the datasets are strictly divided into the training, validation and test sets; that is to say, the test set will not be used in the training process and will thus provide an objective evaluation of the model.

To measure the quality of translation quantitatively, both automatic and human evaluation indicators are employed to cover all aspects of language. The first automatic index is BLEU (Bilingual Evaluation Understudy); that is to say, it calculates the precision of n-gram overlap between the generated translation and the reference text and has been used as a standard for measuring correspondence. CHRF is also used to get a character-level n-gram precision score; it is very sensitive to changes in morphology and spelling errors, so it can be used with BLEU at the word level. Since the automatic indicators cannot assess the meaning and flow of language, human evaluators are employed to judge the translated text based on criteria such as accuracy and fluency.

Many base models have been set up for comparison, and the proposed method will be compared with these existing architectures. The baselines are the original vanilla Transformer trained by Maximum Likelihood Estimation (MLE), the conventional MLE-trained NMT with label smoothing regularization, and some state-of-the-art adversarial NMT variants that are more robust. To guarantee the fairness of the comparative experiment for all test groups, all models will be run in the same hardware and software environment and use the same GPU resources and deep learning frameworks. In addition, the initialisation settings of hyperparameters, such as learning rate, batch size and optimiser configuration, are standardised to prevent differences in performance caused by tuning variations and thus only the improvements from the proposed optimisation strategy will be considered.

To check whether the optimised adversarial Neural Machine Translation (NMT) model is stable and generalisable, some translation quality tests have been carried out on various low-resource and high-resource language pairs. BLEU, CHRF and human semantic fidelity scores are the quantitative performance indicators that have been used to evaluate the high-resource language pair test set. According to the experimental data, the new model has done better than the two baseline models, which are the standard Transformer and Seq2Seq. To confirm that the above results are reliable, pairwise significance difference analysis was carried out by bootstrap resampling for the proposed model and all baselines. Based on the above statistics, it can be seen that the improvements in translation accuracy and fluency are not due to random variation.

Next, we wanted to see how well the model works for low-resource language pairs in the absence of data. In such cases, the standard NMT model is prone to overfitting because there is a lack of parallel training data and thus performs poorly on new data. According to the results of the experiment, adding adversarial learning and a gradient penalty can solve this problem. Add some noise to the input embeddings and then optimise the worst-case loss; thus, the adversarial framework will be able to learn more robust linguistic representations that are not simply memorised from the training data. A Gradient Penalty term is added to help stabilise the training process and prevent the irregular updates that occur in adversarial training. Therefore, it can be used in the low-resource environment to make full use of the small amount of data and improve the stability of the model.

In addition to the above quantitative indicators, a typical translation output was selected for a detailed case study and its quality was assessed. Based on the analysis, the base NMT model often has the problems of missing translated content, ungrammatical expressions in the target language, and incorrect disambiguation of meaning. On the other hand, the optimised model has reduced these problems. For example, in the case of complex sentence structures, the proposed model was able to maintain good syntactic consistency and correctly handle ambiguous words based on a wider range of context; it outperformed the baseline. The quality of the structure is better, and thus it can be determined that the feedback from the adversarial discriminator has helped improve the generator's output. Finally, according to the results, the adversarial learning framework can increase the metric score and also improve the practical semantic fidelity and grammatical correctness of the translation system significantly.

To check whether the proposed Neural Machine Translation (NMT) system is practically stable and reliable, all sorts of robustness tests have been carried out to see how well the model works in the presence of noise. The reason for this assessment is that in actual application, the source data may be imperfect; that is to say, there may be typos in the user-generated text or errors in the output of the Automatic Speech Recognition (ASR) system. The first purpose of this test is to determine whether the optimisation strategies used can keep the semantic information of the model stable in the face of input perturbations that are not in the same distribution as the clean training data.

To create a noisy input test dataset, four kinds of degradation scenarios in the real world were simulated to test the system thoroughly. First, typographical errors were added by randomly replacing, adding or deleting characters in words to simulate human typing mistakes. Secondly, some word segments were formed by randomly deleting the suffixes or prefixes of words to simulate informal shorthand or abbreviated text. Thirdly, to simulate ASR-generated transcription errors, some words were replaced with phonetically similar or homophonic alternatives to reflect the common mistakes of speech-to-text systems. Finally, in order to set up a control group for sensitivity, some small random noise was added to the embedding space of the source sentences. Many kinds of datasets have been constructed to test the robustness of all kinds of input corruption.

Prepare the dataset and then, as the noise intensity increases, plot the performance degradation curves of the proposed adversarial model in comparison with those of the standard MLE-trained NMT model and the conventional adversarial NMT model. Based on the experimental results, although all models showed a decrease in performance with an increase in noise, the proposed model had a relatively small decline. Even in the presence of high noise, the optimised model still had relatively good translation quality, and its BLEU score was higher than that of the baseline system. The standard MLE model has a relatively poor performance; that is to say, it is too sensitive to the exact shape of the surface, and although the traditional adversarial model is better, it is still not as stable as the one proposed here.

The reason for the relatively good stability of the optimised model is that a sentence-level semantic discriminator has been added and gradient penalty regularisation is used. Therefore, the NMT generator will be forced to learn strong, high-level semantic features of the source language sentences and will not just memorise superficial token-level correlations. Give more weight to the general meaning of the context than to individual word patterns, and the model will be less affected by noise. The purpose of the adversarial training process is to reduce the reliance of the model on fragile surface features and learn internal representations that are invariant to small changes in the input. Thus, it can be seen that the optimised model is able to grasp the meaning of the text and will not be affected by changes in the environment; therefore, it is suitable for a reliable system.

In short, through this study, it has been shown that adversarial learning can be used to improve Neural Machine Translation models. Adversarial learning is a type of training framework that can be used to train the generator network and the discriminator network in this case. The generator tries to produce translation results that are as good as those of professional human translators, and at the same time, the discriminator is trained to identify synthetic translations. Therefore, it is a minimax problem; thus, the translation model should be as accurate as possible and have a good distribution of output that is characteristic of the target language. The three steps of the operation for this method are as follows: First, the NMT model is pre-trained with standard maximum likelihood estimation to obtain a good starting point. Then, the adversarial phase is started; that is to say, the discriminator is trained to distinguish between generated samples and ground-truth data, and then the generator is updated to fool the discriminator by means of gradient-based reinforcement. This loop will run many times, and after some time, the generator will be able to produce text with relatively high semantic and syntactic accuracy that can fool the discriminator consistently.

There are some good applications in practice for this optimisation. Adversarial learning is used to solve the problem of exposure bias, which is that the model performs poorly in the inference environment due to differences from the training environment, and thus improves the robustness of the translation system [3]. It has the problem of the old loss function; that is to say, maximising BLEU score often leads to general or literal translations. Adversarial training can be used to generate more natural, coherent and contextually appropriate sentences that are close to human language. Adversarial learning is also a general way to address the problem of data scarcity for low-resource language pairs in this study. A good way to make full use of the existing dataset and increase the difficulty of negative samples in training is to do so. Finally, the successful integration of adversarial learning into NMT architectures has been an important progress in the field; it provides a regular and effective way to improve the quality of machine-generated translations to be closer to that of human translation. It will provide some support for the construction of robust and intelligent language processing systems in the future.

\[1\]Kumar, Amit, Pratap, A., Singh, Anil Kumar. Exploiting Multilingualism in Low-resource Neural Machine Translation via Adversarial Learning[J]. ArXiv, 2023, abs/2303.18011.

\[2\]Idrissi, Badr Youbi, Clinchant, Stéphane. Masked Adversarial Generation for Neural Machine Translation[J]. 2021.

\[3\]Cheng, Yong, Jiang, Lu, Macherey, Wolfgang. Robust Neural Machine Translation with Doubly Adversarial Inputs[J]. 2019.

\[4\]Yang, Zhen, Chen, Wei, Wang, Feng. Generative Adversarial Training for Neural Machine Translation[J]. Neurocomputing, 2018, 321(DEC.10): 146-155.