Guidance to Pre-tokeniztion for SacreBLEU: Meta-Evaluation in Korean

SacreBLEU, by incorporating a text normalizing step in the pipeline, has been well-received as an automatic evaluation metric in recent years. With agglutinative languages such as Korean, however, the metric cannot provide a conceivable result without the help of customized pre-tokenization. In this regard, this paper endeavors to examine the influence of diversified pre-tokenization schemes –word, morpheme, character, and subword– on the aforementioned metric by performing a meta-evaluation with manually-constructed into-Korean human evaluation data. Our empirical study demonstrates that the correlation of SacreBLEU (to human judgment) fluctuates consistently by the token type. The reliability of the metric even deteriorates due to some tokenization, and MeCab is not an exception. Guiding through the proper usage of tokenizer for each metric, we stress the significance of a character level and the insignificance of a Jamo level in MT evaluation.

Your Brother’s Gatekeeper: How Effects of Evaluation Machineries in Research Are Sometimes Enhanced

Many warnings are issued against the influence of evaluation machineries (such as bibliometric indicators) upon research practices. It is often argued that human judgment can function as a bulwark against constitutive effects of evaluation machineries. Using vignettes (small case narratives) related to the Danish Bibliometric Research Indicator (BRI), this chapter shows that gatekeepers who “know the future” and use this “knowledge” in a preemptive or precautionary way play a key role in the construction of reality which comes out of the BRI. By showing that human judgment sometimes enhances or multiplies the effects of evaluation machineries, this chapter contributes to an understanding of mechanisms which lead to constitutive effects of evaluation systems in research.

A Grotian Moment for Animal Sociability?

In this article, I will revert to the categories of ‘fitness’ and ‘sociability’ to ask whether a ‘Grotian moment for animal sociability’ can be conceptualized. Grotius claims that we share a core of fundamental laws with animals. Building upon a passage from Seneca’s De clementia, Grotius calls these laws ‘commune ius animantium’, i.e. the common law of living beings. These shared legal entitlements, based on a shared sense of innate fitness, show that a certain care of maintaining society (‘animal sociability’) is common to all living beings. However, as I will show, humans, as the only beings capable of speech and moral deliberation, remain the only translators and enforcers of this instinct into a language of rights. From this perspective, it can be argued that a ‘Grotian tradition’ of animal rights exists, as Grotius’s reliance on the ‘common law of living beings’ can be interpreted in a progressive manner. However, I will argue that animal sociability qualifies as a ‘non-Grotian moment’: sociability as owed to animals but only in a thin sense, as it requires human judgment to be enforced into strict right. Such a ‘non-Grotian moment’ reveals that the deeply anthropocentric structure of Grotius’ theory is incapable of triggering any paradigm shift, because animals lack the capacity for judgment that is so essential to be a legal person.

Eliminating Human Agency: Why Does Japan Abandon Predictive Simulations?

Prediction plays a vital role in every branch of our contemporary lives. While the credibility of quantitative simulations through mathematical modeling may seem to be universal, how they are perceived and embedded in policy processes may vary by society. Investigating the ecology of quantitative prediction tools, this article articulates the cultural specificity of Japanese society through the concept of Jasanoff’s “civic epistemology.” Taking COVID-19 and nuclear disasters as examples, this article examines how predictive simulations are mobilized, contested, and abandoned. In both cases, current empirical observation eventually replaces predictive future simulations, and mechanical application of preset criteria substitutes political judgment. These analyses suggest that the preferred register of objectivity in Japan—one of the constitutive dimensions of civic epistemology—consists not in producing numerical results, but in precluding human judgment. Such public calls to eliminate human agency both in knowledge and in policy-making can be a distinct character of Japanese civic epistemology, which may explain why Japan repeatedly withdraws from predictive simulations. It implies the possibility that Western societies’ faith in human judgment should not be taken for granted, but explained.

Controlling hallucinations at word level in data-to-text generation

Data-to-Text Generation (DTG) is a subfield of Natural Language Generation aiming at transcribing structured data in natural language descriptions. The field has been recently boosted by the use of neural-based generators which exhibit on one side great syntactic skills without the need of hand-crafted pipelines; on the other side, the quality of the generated text reflects the quality of the training data, which in realistic settings only offer imperfectly aligned structure-text pairs. Consequently, state-of-art neural models include misleading statements –usually called hallucinations—in their outputs. The control of this phenomenon is today a major challenge for DTG, and is the problem addressed in the paper. Previous work deal with this issue at the instance level: using an alignment score for each table-reference pair. In contrast, we propose a finer-grained approach, arguing that hallucinations should rather be treated at the word level. Specifically, we propose a Multi-Branch Decoder which is able to leverage word-level labels to learn the relevant parts of each training instance. These labels are obtained following a simple and efficient scoring procedure based on co-occurrence analysis and dependency parsing. Extensive evaluations, via automated metrics and human judgment on the standard WikiBio benchmark, show the accuracy of our alignment labels and the effectiveness of the proposed Multi-Branch Decoder. Our model is able to reduce and control hallucinations, while keeping fluency and coherence in generated texts. Further experiments on a degraded version of ToTTo show that our model could be successfully used on very noisy settings.

A method to solve strategy based decision making problems with logarithmic T-spherical fuzzy aggregation framework

T-Spherical fuzzy set (TSFS) is an improved extension in fuzzy set (FS) theory that takes into account four angles of the human judgment under uncertainty about a phenomenon that is membership degree (MD), abstinence degree (AD), non-membership degree (NMD), and refusal degree (RD). The purpose of this manuscript is to introduce and investigate logarithmic aggregation operators (LAOs) in the layout of TSFSs after observing the shortcomings of the previously existing AOs. First, we introduce the notions of logarithmic operations for T-spherical fuzzy numbers (TSFNs) and investigate some of their characteristics. The study is extended to develop T-spherical fuzzy (TSF) logarithmic AOs using the TSF logarithmic operations. The main theory includes the logarithmic TSF weighted averaging (LTSFWA) operator, and logarithmic TSF weighted geometric (LTSFWG) operator along with the conception of ordered weighted and hybrid AOs. An investigation about the validity of the logarithmic TSF AOs is established by using the induction method and examples are solved to examine the practicality of newly developed operators. Additionally, an algorithm for solving the problem of best production choice is developed using TSF information and logarithmic TSF AOs. An illustrative example is solved based on the proposed algorithm where the impact of the associated parameters is examined. We also did a comparative analysis to examine the advantages of the logarithmic TSF AOs.

Sublemma-Based Neural Machine Translation

Powerful deep learning approach frees us from feature engineering in many artificial intelligence tasks. The approach is able to extract efficient representations from the input data, if the data are large enough. Unfortunately, it is not always possible to collect large and quality data. For tasks in low-resource contexts, such as the Russian ⟶ Vietnamese machine translation, insights into the data can compensate for their humble size. In this study of modelling Russian ⟶ Vietnamese translation, we leverage the input Russian words by decomposing them into not only features but also subfeatures. First, we break down a Russian word into a set of linguistic features: part-of-speech, morphology, dependency labels, and lemma. Second, the lemma feature is further divided into subfeatures labelled with tags corresponding to their positions in the lemma. Being consistent with the source side, Vietnamese target sentences are represented as sequences of subtokens. Sublemma-based neural machine translation proves itself in our experiments on Russian-Vietnamese bilingual data collected from TED talks. Experiment results reveal that the proposed model outperforms the best available Russian ⟶ Vietnamese model by 0.97 BLEU. In addition, automatic machine judgment on the experiment results is verified by human judgment. The proposed sublemma-based model provides an alternative to existing models when we build translation systems from an inflectionally rich language, such as Russian, Czech, or Bulgarian, in low-resource contexts.

Securitizing cyberspace: Protecting political judgment

The contemporary debate in democracies routinely refers to online misinformation, disinformation, and deception, as security-issues in need of urgent attention. Despite this pervasive discourse, however, policymakers often appear incapable of articulating what security means in this context. This paper argues that we must understand the unique practical and normative challenges to security actualized by such online information threats, when they arise in a democratic context. Investigating security-making in the nexus between technology and national security through the concept of “cybersovereignty,” the paper highlights a shared blind spot in the envisaged protection of national security and democracy in cyberspace. Failing to consider the implications of non-territoriality in cyberspace, the “cybersovereign” approach runs into a cul de sac. Security-making, when understood as the continuous constitution of “cybersovereign” boundaries presumes the existence of a legitimate securitizing actor; however, this actor can only be legitimate as a product of pre-existing boundaries. In response to the problems outlined, the article proposes an alternative object of protection in the form of human judgment and, specifically, “political judgment” in the Arendtian sense. The turn to political judgment offers a conceptualization of security that can account for contemporary policy practises in relation to security and the online information threat, as well as for the human communicating subject in the interactive and essentially incomplete information and communication environment.

An Oblivious Approach to Machine Translation Quality Estimation

Machine translation (MT) is being used by millions of people daily, and therefore evaluating the quality of such systems is an important task. While human expert evaluation of MT output remains the most accurate method, it is not scalable by any means. Automatic procedures that perform the task of Machine Translation Quality Estimation (MT-QE) are typically trained on a large corpus of source–target sentence pairs, which are labeled with human judgment scores. Furthermore, the test set is typically drawn from the same distribution as the train. However, recently, interest in low-resource and unsupervised MT-QE has gained momentum. In this paper, we define and study a further restriction of the unsupervised MT-QE setting that we call oblivious MT-QE. Besides having no access no human judgment scores, the algorithm has no access to the test text’s distribution. We propose an oblivious MT-QE system based on a new notion of sentence cohesiveness that we introduce. We tested our system on standard competition datasets for various language pairs. In all cases, the performance of our system was comparable to the performance of the non-oblivious baseline system provided by the competition organizers. Our results suggest that reasonable MT-QE can be carried out even in the restrictive oblivious setting.