Chapter 1 Introduction
Cross-lingual transfer learning is a typical method in modern Natural Language Processing (NLP) that has been introduced to solve the problem of a lack of annotated training data for a particular language. The problem is most severe in the case of low-resource languages; although they are spoken by a large number of people, they do not have the digital infrastructure and linguistic resources of high-resource languages such as English. The first idea of cross-lingual transfer is to use the computational knowledge that has been gained from data-rich source languages to improve the performance of NLP models in data-poor target languages. Instead of training a separate model for each language from scratch, which is often computationally expensive and statistically unreliable due to a lack of data, transfer learning can be used to share the statistical strength and linguistic patterns of different languages. This will help spread the use of language technology and ensure that the advantages of artificial intelligence are not limited to digitally advanced languages but are also enjoyed by all people around the world.
At the level of operation, cross-lingual transfer generally uses multilingual pre-trained language models, such as mBERT and XLM-R. These models have been trained on a large number of texts in hundreds of languages to learn that semantically similar sentences from different languages are close to each other in a high-dimensional vector space. In an ideal case called "zero-shot transfer", a model that has been fine-tuned on a certain task in a high-resource source language, such as named entity recognition or sentiment analysis, can be used for the target language without any additional training. However, there are considerable problems in reality with the "domain shift" problem. Although pre-trained models are a good starting point, the statistical distributions of the source and target languages are often quite different due to typological differences, various syntactic structures and other unique features of morphology. Therefore, the model may not be able to generalise well and will perform poorly on the low-resource target.
To address the above problem and improve the transfer process, adversarial domain adaptation technology will be employed. Domain adversarial training is a method that can be used to learn language-invariant feature representations that are not affected by the difference between the source and target domains; thus, it is difficult to distinguish which domain a particular feature comes from. The two parts of the operating procedure are the feature extractor and the discriminator. The feature extractor is trained to map input sentences from both languages into a common embedding space, and then the discriminator tries to determine which language the embeddings are from. A minimax game dynamic is used to penalise the feature extractor if the discriminator can distinguish the languages, and thus the extractor must produce representations that are indistinguishable across languages. At the same time, a task-specific classifier is used to keep the semantic information needed for a particular downstream task in the embeddings. Remove the language-specific cue and focus on the semantic content of the task to reduce negative transfer and improve the model's robustness in a low-resource environment.
Adversarial methods have shown good results in improving the cross-lingual transfer of NLP models and can be applied to build more robust NLP systems for low-resource languages, including delivering notable performance gains on practical tasks such as slot tagging [1]. The standard transfer method usually needs to be supervised in the target language, but this is often not available. By improving the alignment of the source and target distributions and not needing target labels, this method has reduced the entry barrier for the application of functional language technology. It can quickly build necessary tools, such as machine translation systems, information extraction pipelines and text classifiers, for languages that have long been lacking in technological support. Improve the stability and accuracy of these systems to ensure that the deployed solutions can work stably in important areas such as medical care, education and public welfare. Therefore, the refinement of these transfer mechanisms is not just a theoretical problem; it needs to be done to narrow the digital divide and promote linguistic inclusion in the era of artificial intelligence.
Chapter 2 Optimization Framework Design and Empirical Evaluation of Adversarial Domain Adaptation for Cross-Lingual Transfer in Low-Resource Languages
2.1 Theoretical Limitations of Existing Cross-Lingual Transfer Pipelines for Low-Resource Languages
The basic idea of cross-lingual transfer learning is that, by training on high-resource languages (HRLs) with rich data, models can learn transferable linguistic abstractions that are applicable to low-resource languages (LRLs). However, a close examination of the current pipeline shows that there is a problem in the theory called embedding distribution drift. In the typical workflow of zero-shot or few-shot fine-tuning, it is assumed that the feature space of the source HRL and the target LRL are isomorphic; that is to say, they have the same geometric structure and semantic equivalents are located at the same coordinates. Mathematically, this assumption is often invalid because the marginal probability distributions of the embeddings, and , are very different due to typological differences and an unbalanced pre-training corpus. In the vanilla workflow, only the task-specific loss on the source domain is used for fine-tuning, and the domain discrepancy is not explicitly minimised; thus, there is a misalignment, and the decision boundary learned from HRL data cannot be generalised well to the shifted distribution of LRL data. Therefore, although there is a structural similarity in meaning, the projection of LRL inputs into the shared vector space is in a low-density region relative to the source classifier, and thus the inference performance is poor.
The fixed alignment goals of the original multilingual pre-trained language models (mPLMs), such as multilingual BERT and XLM-RoBERTa, are also a reason for this drift. These models usually have a common subword vocabulary and are trained with a general masked language model objective in all languages. Although it can help to some extent with the cross-lingual alignment of major languages, it is unable to address the specific syntactic and semantic characteristics of less commonly used languages. Because LRLs are relatively rare in the pre-training data, the optimisation landscape of the model is biased towards HRLs. Static alignment has created a "hub language" effect, and the vector space is mainly organised according to the grammar and lexicon of high-resource languages. LRLs often have different forms or are not very common; thus, the fixed mapping is not suitable. The model cannot represent fine-grained language-specific nuances, so the LRL representation has to be approximated by the nearest HRL cluster, and thus some important semantic information needed for precise downstream tasks is lost.
In the past, to solve these problems, Adversarial Domain Adaptation (ADA) has been used, but it has some operational defects that make it unsuitable for LRLs. A first technical flaw is that only one type of gradient penalty is used, and it is generally applied to the distribution of top-layer features. The purpose of the generator in adversarial training is to deceive the discriminator, but if there are too few constraints, it may result in mode collapse. This is because the generator has learned to produce feature outputs that can fool the discriminator by mapping all kinds of LRL inputs to the same general domain-invariant point. Only the top layer is restricted; the lower layers, which contain basic syntactic and morphological information, are not. Therefore, the rare LRL semantic features are already sparse in the model, and they are further reduced by the adversarial optimisation to a shallow alignment that can pass the discriminator test but lacks semantic depth.
At present, there is no systematic way to use the few annotation signals in the existing ADA workflow for cross-lingual tasks. Generally speaking, standard unsupervised domain alignment does not make use of the small number of labelled samples for LRLs in the unsupervised objective to maintain purity, and is only employed for the final evaluation. It is very inefficient. In the case of limited resources, even a small number of annotated data items can provide some structural knowledge about the specific label distribution in the target domain. Since there is no way to add these sparse, unpaired or semi-paired signals, the alignment process will be unaware of the decision boundaries that are relevant to the target task. Adversarial optimisation can align the input distributions globally, but it is not aligned with the task-relevant subspaces that have few labels. Therefore, the existing pipelines cannot meet the semantic alignment requirements of competitive transfer; that is to say, they are unable to correct distributional drift, lack language-specific granularity, have feature erosion in deep networks, and do not make full use of the minimal supervision signals available for low-resource languages.
2.2 Adversarial Domain Adaptation Architecture for Semantic Alignment Between High-Resource and Low-Resource Language Embedding Spaces
2.2.1 Multi-Gradient Penalty Mechanism for Mitigating Mode Collapse in Cross-Lingual Feature Matching
A new multi-gradient penalty method has been introduced to help stabilise the training of adversarial networks for cross-lingual transfer learning and address the problem of mode collapse. Mode collapse is a phenomenon in which the generator, which is used to align the low-resource language (LRL) with the high-resource language (HRL), reduces the semantic representations of the LRL to a small number of points on the HRL manifold. As a result, the distinctive and rare linguistic characteristics of LRL are lost in the general transfer and cannot be used for subsequent work. Therefore, a hierarchical multi-layer discriminator structure is added to the framework; it no longer needs to be a standard Wasserstein GAN-based Adversarial Domain Adaptation (ADA) method, and gradient penalties are only applied to the final task-specific layer. Therefore, it takes the intermediate context embedding outputs of the last four layers of the base multilingual Pre-trained Language Model (mPLM) at the same time.
The basic idea of this Design is that the various layers of a deep transformer model are at different levels of linguistic abstraction. The lower intermediate layer is known to have finer details of the language, such as morphemes and words, and the upper layer contains more general knowledge that is not tied to specific tasks. An adversarial discriminator is employed at various levels to supervise the alignment of the system at different scales explicitly. Therefore, the weighted gradient penalty loss function is used to give more weight to the constraints of the lower intermediate layers. This way of weighting is necessary to impose a stricter Lipschitz condition on the changes of fine-grained linguistic attributes. If the penalty were to be applied uniformly or only at the top layer, the generator could easily meet the demands of the discriminator by smoothing out the unique morphological characteristics of the LRL in the lower layers and thus lose linguistic accuracy for the sake of domain confusion.
The steps in the operation of this mechanism are to derive and integrate a composite loss function. First, calculate the standard adversarial domain discrimination loss for each of the four selected layers. Therefore, the generator will try to make embeddings that the discriminator cannot distinguish from the target HRL distribution. Only using this loss will be unstable. Therefore, a multi-gradient penalty term is added. For each layer, compute the gradient of the discriminator's output with respect to its input. The penalty term is to keep the norm of this gradient close to a target constant (usually 1) and thus ensure that the decision boundary of the discriminator changes smoothly. Importantly, the gradients are multiplied by a weight that is inversely proportional to the layer depth. The combined loss function is the sum of the weighted penalty terms and the base adversarial loss; therefore, the optimisation goal is to reduce domain divergence and at the same time strictly penalise sharp or collapsing gradients in the feature space.
In practice, this mechanism will not have all the rare LRL semantic representations collapsed into the overrepresented HRL distribution manifold by the adversarial generator; it will keep the geometric structure of the LRL feature space at every level of the network. By imposing a large penalty on the irregular gradient in the lower layer, it is ensured that the characteristics of the language are not lost. The flow of these gradients can be regarded as a kind of distributed regulatory force that moves backward in the network. Although the traditional method is to have one main gradient signal that can draw all the LRL inputs into one HRL mode, a multi-layer approach is taken here to create a constrained optimisation problem that needs to meet the structural consistency requirements of lexicon, syntax and semantics at the same time. Therefore, the resulting cross-lingual embeddings will be domain-aligned and structurally rich; thus, they can be used to improve the performance of the model in low-resource languages by retaining the original distribution of features.
2.2.2 Auxiliary Lexical Constraint Module for Integrating Sparse Low-Resource Language Annotation Signals
Due to the lack of linguistic resources in Low-Resource Languages (LRLs), an optimisation framework has been introduced to make use of the sparse annotation signals that are often available in such resource-constrained environments. Since there is no large-scale parallel corpus for LRLs, small, unpaired datasets can be obtained at low cost and effort, such as monolingual part-of-speech tags, bilingual seed lexica with 500-1000 word pairs, and crowdsourced simple semantic labels. First, a small lexical alignment dictionary is built from all the above materials to start the work of this module. It is not an expensive full-parallel corpus construction, and therefore does not need to be aligned at the sentence level; instead, unpaired lexical units are used to build a basic connection between the High-Resource Language (HRL) and the LRL. Normalisation of the raw sparse data is performed to ensure consistency, then the identified lexical pairs are mapped to their corresponding vector representations in the embedding space, and finally, noise is removed to maintain the reliability of the constraints.
A lexical dictionary is created, and then an auxiliary contrastive loss term is added to the module to regulate the geometry of the shared embedding space. The basic idea of this loss function is to change the distances among word vectors according to their meanings. In other words, it reduces the distance and brings the embeddings of semantically equivalent HRL and LRL lexical units in the seed dictionary closer together. At the same time, it increases the distance between representations of semantically different word pairs that frequently appear in the same context but do not have verified translation equivalence. The contrastive mechanism is a strict attraction-repulsion relationship that can refine the boundary of semantic clusters and ensure that the transferred features are in line with the ground-truth meanings specified by sparse annotations.
An important implementation detail is how to add this auxiliary term to the full Adversarial Domain Adaptation (ADA) training objective. Instead of making the lexical constraints hard alignment rules that would cause the model to overfit the small and possibly noisy dictionary, they are added as a soft regularisation signal. In other words, a small weight is added to the auxiliary contrastive loss and then combined with the main adversarial loss. Therefore, the model can address the problem of a large number of domains in adversarial training and also follow the lexical supervision. Since there are few annotations, they are treated as a soft guide, and the framework can still learn from the distributional characteristics of the unlabelled data to avoid being too rigidly tied to a small number of word pairs.
The purpose of the auxiliary module in practice is to address the problem of a lack of information in fully unsupervised adversarial alignment. Unsupervised methods can learn some general domain-invariance, but they are prone to the "hubness" problem and semantic distortion in LRLs because there is no explicit linguistic guidance. The auxiliary lexical constraint module can add some necessary semantic information that has been verified by humans to the training process. The injection can fix the mapping process, anchor the embedding space to stable linguistic landmarks, and thus improve the robustness of cross-lingual transfer significantly. Therefore, the model has good performance on the downstream task of LRL, and it can be seen that even with a small number of annotations, when combined with a reasonable auxiliary loss function, the effectiveness of domain adaptation can be greatly improved.
2.3 Experimental Validation and Performance Comparison on Standard Cross-Lingual NLP Benchmarks
2.3.1 Dataset Configuration and Evaluation Metrics Selection for Target Low-Resource Languages
To verify the effect of the proposed adversarial domain adaptation framework reasonably, good experimental conditions should be created; that is to say, it should be possible to simulate a real low-resource language (LRL) environment and transfer high-resource language (HRL) supervision effectively. The basic idea of the above setup is to determine how well the model can align the embedding spaces of different languages when there is a lack of target data. Therefore, in the construction of the dataset, we have decided to focus on six target languages that have a severe lack of data, specifically those with fewer than 1 million total unlabeled tokens available publicly and no large-scale parallel bilingual corpora. These languages are chosen to ensure typological diversity, and they belong to the Afro-Asiatic (Amharic), Niger-Congo (Wolof, Swahili low-resource dialect variants), Sino-Tibetan (Tibetan), and other families such as Quechua and Haitian Creole. The choice of this way is to test whether the framework can be generalised to many different forms and structures of language, so we know that the optimisation is due to strong feature transfer, not just similarity in language.
After setting the target languages, an experimental framework is employed on the three typical cross-lingual NLP benchmarks to show that it has been applied to many areas: cross-lingual sentiment classification, Named Entity Recognition (NER) and Question Answering (QA). The operation procedure of these tasks is to limit the number of shots for the data split in real life. In this case, the amount of labelled training data for each target LRL is at most 200 annotations per task. A serious restriction is that the model will not be able to learn target-language-specific patterns by heart and must use cross-lingual alignment from the adversarial part; thus, it can be applied to cases where manual annotation is too costly or time-consuming.
To make an all-encompassing and quantitative evaluation of the results, different indicators will be set for various types of benchmarks based on their characteristics. Accuracy is used to determine how well the model can predict the sentiment of text in the sentiment classification problem. The main index of the sequence labelling task for NER is the Macro F1 score; due to the problem of class imbalance, it is not desired that the performance of rare entity types be overlooked because of their small number. For the question-answering task, the exact match (EM) score is used to strictly evaluate how many predicted answers are exactly the same as the ground truth, character for character.
In addition to the accuracy of the task, a distribution distance metric is also used to examine the internal mechanism of transfer learning quantitatively. Specifically, the Maximum Mean Discrepancy (MMD) is used to measure the statistical distance between the HRL source embeddings and the LRL target embeddings in the same feature space. A small MMD value indicates that the data distributions are close, and thus it can be concluded that adversarial training has reduced the domain shift. The combination of task performance indicators and distribution analysis can provide an overall understanding of the system's functions. Finally, in order to ensure that all empirical results are reliable and can be reproduced by other researchers in the future, the experiment has fixed a random seed and used k-fold cross-validation for all few-shot splits; thus, the reported performance is statistically significant and representative of the model's actual ability.
2.3.2 Baseline Model Contrast and Ablation Study of Proposed Framework Components
First, a good comparison system will be set up in the experiment to see how well the optimisation model works. In order to conduct a comprehensive evaluation, many baseline models have been constructed; they are the vanilla zero-shot cross-lingual transfer of XLM-RoBERTa, and the previous state-of-the-art adversarial domain adaptation (ADA) model that uses one gradient penalty. Some good cross-lingual data augmentation methods that have been proposed for low-resource languages (LRLs) have also been added to improve the performance of the model by expanding the data. The six target LRLs were evaluated in three kinds of natural language processing tasks: Named Entity Recognition (NER), Part-of-speech (POS) tagging and Natural Language Inference (NLI). Based on the analysis of the experiment results, the new framework is much better than all the other baselines. Compared with the strongest baseline model, the new method has increased the average macro F1 score by 9.7% and the exact match score by 11.2% for all the tasks. The above indicators show that the framework can be applied to many different types of language and various difficulty levels of tasks, and the optimisation strategies have solved the problem of transferring high-resource and low-resource languages.
Based on the baseline comparison, an organised ablation study was carried out to determine how much each part of the proposal contributes to the whole. Some important parts of this paper have been omitted, such as the multi-gradient penalty mechanism, the auxiliary lexical constraint module and the hierarchical multi-layer discriminator, and thus the performance has dropped. According to the above results, after removing the multi-gradient penalty, the alignment stability has dropped significantly; thus, it is necessary to maintain stability during adversarial training. Similarly, without the auxiliary lexical constraint, there was a relatively large drop in semantic consistency, especially in the syntactically rich tasks of NER and POS tagging; at the same time, the removal of the hierarchical discriminator reduced the model's ability to learn language-invariant features at different levels of the network. All the results of the experiments have been statistically tested for significance to ensure that they are reliable. According to the test results, all the observed improvements in performance are statistically significant at the p < 0.05 level; thus, it can be concluded that these improvements are due to the proposed architectural changes and not due to random fluctuations in training or stochastic initialisation.
To provide more evidence for the theoretical basis of the improvement in performance, a quantitative comparison of the Maximum Mean Discrepancy (MMD) distance between the embedding distributions of the high-resource language (HRL) and the target LRLs is also included in the analysis. MMD is a good indicator of how different the two distributions are in the shared latent space. According to the experimental data, the MMD distance of the proposed framework is relatively small compared with that of the baseline model; thus, the cross-lingual embedding distribution is more concentrated. The decrease in the spread of distribution is related to an improvement in people's work. To reduce the domain discrepancy and apply multiple gradient penalties in the hierarchical discriminator, it can be ensured that the feature representations learned by the HRL are closer to those of the LRL. Therefore, based on this empirical evidence, it can be concluded that the higher macro F1 and exact match scores are due to better semantic alignment; thus, the optimisation framework is both practical and theoretically sound for low-resource situations in real life.
2.4 Error Analysis and Applicability Boundary Discussion for Cross-Lingual Transfer Tasks
To improve the stability and practical application of the optimisation framework that has been put forward, fine-grained error analysis will be carried out to find any deficiencies in the framework and set reasonable limits on its application. First, all the prediction errors of the model on the held-out Low-Resource Language (LRL) test set are divided into different categories according to the reasons for the errors in language and the system. The first type of error is a morphological inflection error, and it is very common in highly agglutinative LRLs. In these languages, words are formed by combining several morphemes, so they have complex structures that the transfer model cannot split properly; thus, the roots and affixes are incorrectly combined. The second type is semantic shift errors, which occur when there are no direct equivalents in High-Resource Language (HRL) for culture-specific LRL concepts. In such cases, the model will attempt to map based on the semantic space of HRL and produce a translation that is grammatically correct but culturally or contextually inappropriate. The third kind is annotation noise propagation errors that arise from sparse input lexical constraints. Because there are only a few bilingual seeds in the framework, any errors or mismatches in these initial pairs will be magnified during the adversarial training process and will result in systematic errors in the output. By counting the proportion of each type of error in the different LRL groups, it can be seen that morphological problems are more frequent in agglutinative languages, and semantic shifts are the main type of error for linguistically distant cultures.
Based on the above error classification, the scope of application for the framework will be determined by examining its performance under three kinds of changes in variables: different quantities of sparse annotation data, various typological distances between source HRLs and target LRLs, and different total amounts of available unlabeled LRL data. According to the results of the experiment, when there are at least 200 unlabeled sentences for semi-supervised learning and the typological distance score from the source HRL is less than 0.85, the framework will have good performance. There are fewer than 200 unlabelled sentences, so the adversarial part cannot converge to a stable domain-invariant representation, and thus the transfer mechanism is not working. If the typological distance is greater than 0.85, it can be considered that the structure has changed significantly; thus, the shared latent space created by the encoder will be too sparse to retain meaningful syntactic information, and the transfer accuracy will drop sharply. The above indicators can help people decide whether a certain pair of languages is suitable for this way of doing things.
Finally, based on the above analysis, there are still some problems that cannot be solved by the current framework; thus, its limitations have been identified. The first is a case of an extremely underdocumented language that does not have any lexical seed pairs because there are no linguistic reference materials or parallel corpora. As the proposed optimisation framework needs a small number of bilingual anchors to start the domain adaptation process, it will not work without any such seeds. In addition, if the LRL has a very different structure from the HRL and is not linear, even with a large amount of data, the assumption of linear transformation in adversarial alignment may no longer hold. Therefore, in the future, research on zero-shot transfer learning that does not require a large number of labelled data will be necessary to extend cross-lingual transfer technology to all the world's languages.
Chapter 3 Conclusion
In short, it has been shown in this study that adversarial domain adaptation can be used to improve the cross-lingual transfer of low-resource languages effectively and stably. The first problem to be addressed in this paper is that there is a lack of training data, and thus, the standard neural machine translation and natural language processing models cannot be applied to languages without large digital corpora. Based on the theory of domain adaptation, the problem of cross-lingual transfer can be transformed into one of domain alignment; thus, the goal is to learn language-agnostic feature representations that are not affected by the specific linguistic attributes of the source and target domains. Therefore, it can generalise well from the high-resource source language to the low-resource target language without needing a large number of parallel data [3].
Adversarial training is used to reduce the domain discrepancy, and it is one of the reasons for this optimisation. The two parts of the system are a feature extractor and a domain discriminator. A feature extractor is trained to map the input sentences of both the source and target languages into the same high-dimensional latent space. At the same time, the domain discriminator is also trying to distinguish the features extracted from the source language and the target language. A minimax game is used to reduce the loss of the feature extractor when the discriminator can determine the origin of a feature, and thus, the extractor needs to produce representations that are difficult for the discriminator to recognise. The two languages' statistical distributions are aligned through this opposition, and therefore, only the semantic information will be used for the next task, not language-specific characteristics.
The path of implementation in this paper has some key technical features that need to be considered for actual use. First, in the preprocessing stage, we will conduct thorough subword segmentation to address out-of-vocabulary words and other forms of the same word; this is especially true for low-resource languages with rich morphology. Secondly, in order to optimise both the feature extraction and domain confusion at the same time, a gradient reversal layer is used in the training process. The first layer is used to invert the gradient signal from the discriminator before it is passed to the feature extractor in backpropagation. As a result, it has been steadily approaching a state where the linguistic features in different areas are the same. A semi-supervised learning part can also be added to use the unlabelled data from the target language and improve the stability of transfer.
In addition to the theoretical interest, the practical applications of these results will also provide some assistance for the preservation and development of low-resource languages in the world's digital ecosystem. By reducing the need for a large number of parallel datasets, this way can lower the entry barrier for the development of functional language technology in less-developed areas. Adversarial adaptation can be employed to carry out cross-lingual transfer, and therefore, many necessary applications of machine translation, sentiment analysis and information retrieval can be realised with a relatively small amount of training data. It will be easier for people to access and will promote the spread of information; it can also provide a model for further research in low-resource NLP. Finally, according to the standard operating procedure in this paper, adversarial domain adaptation is no longer just an experimental method but a basic requirement for high-performance, inclusive language technology.
References
\[1\]He, Keqing, Yan, Yuanmeng, Xu, Weiran. Adversarial Cross-Lingual Transfer Learning for Slot Tagging of Low-Resource Languages[J]. 会议论文, 2020, 000(9200848).
\[2\]He, Keqing, Yan, Yuanmeng, Xu, Weiran. Adversarial Cross-Lingual Transfer Learning for Slot Tagging of Low-Resource Languages[J]. 2020.
\[3\]Huang, Lifu, Ji, Heng, May, Jonathan. Cross-lingual Multi-Level Adversarial Transfer to Enhance Low-Resource Name Tagging[C]. 2019.