Dislodged from History, Confronted by Walls : Picturing Migration as a Global Emergency

This essay examines select visual representations of refugees and migrants as embodied subjects in photography, art, and video. It focuses on American asylum politics and explores the questions of free movement, the right to have rights, and the ethics and efficacy of border walls. It argues that the catastrophe of global forced displacement makes the elimination of national borders and the nation state itself a revolutionary necessity.

What You See is What it Means! Semantic Representation Learning of Code based on Visualization and Transfer Learning

Recent successes in training word embeddings for Natural Language Processing ( NLP ) tasks have encouraged a wave of research on representation learning for source code, which builds on similar NLP methods. The overall objective is then to produce code embeddings that capture the maximum of program semantics. State-of-the-art approaches invariably rely on a syntactic representation (i.e., raw lexical tokens, abstract syntax trees, or intermediate representation tokens) to generate embeddings, which are criticized in the literature as non-robust or non-generalizable. In this work, we investigate a novel embedding approach based on the intuition that source code has visual patterns of semantics. We further use these patterns to address the outstanding challenge of identifying semantic code clones. We propose the WySiWiM ( ‘ ‘What You See Is What It Means ” ) approach where visual representations of source code are fed into powerful pre-trained image classification neural networks from the field of computer vision to benefit from the practical advantages of transfer learning. We evaluate the proposed embedding approach on the task of vulnerable code prediction in source code and on two variations of the task of semantic code clone identification: code clone detection (a binary classification problem), and code classification (a multi-classification problem). We show with experiments on the BigCloneBench (Java), Open Judge (C) that although simple, our WySiWiM approach performs as effectively as state-of-the-art approaches such as ASTNN or TBCNN. We also showed with data from NVD and SARD that WySiWiM representation can be used to learn a vulnerable code detector with reasonable performance (accuracy ∼90%). We further explore the influence of different steps in our approach, such as the choice of visual representations or the classification algorithm, to eventually discuss the promises and limitations of this research direction.

Backpropagation-Based Decoding for Multimodal Machine Translation

People are able to describe images using thousands of languages, but languages share only one visual world. The aim of this work is to use the learned intermediate visual representations from a deep convolutional neural network to transfer information across languages for which paired data is not available in any form. Our work proposes using backpropagation-based decoding coupled with transformer-based multilingual-multimodal language models in order to obtain translations between any languages used during training. We particularly show the capabilities of this approach in the translation of German-Japanese and Japanese-German sentence pairs, given a training data of images freely associated with text in English, German, and Japanese but for which no single image contains annotations in both Japanese and German. Moreover, we demonstrate that our approach is also generally useful in the multilingual image captioning task when sentences in a second language are available at test time. The results of our method also compare favorably in the Multi30k dataset against recently proposed methods that are also aiming to leverage images as an intermediate source of translations.

Attention modulates neural representation to render reconstructions according to subjective appearance

Stimulus images can be reconstructed from visual cortical activity. However, our perception of stimuli is shaped by both stimulus-induced and top-down processes, and it is unclear whether and how reconstructions reflect top-down aspects of perception. Here, we investigate the effect of attention on reconstructions using fMRI activity measured while subjects attend to one of two superimposed images. A state-of-the-art method is used for image reconstruction, in which brain activity is translated (decoded) to deep neural network (DNN) features of hierarchical layers then to an image. Reconstructions resemble the attended rather than unattended images. They can be modeled by superimposed images with biased contrasts, comparable to the appearance during attention. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses, modulating neural representations to render reconstructions in accordance with subjective appearance.

Open Challenges and Research Roadmap: Visual Representations aiding Organizational Leadership in Decision Making for New Product Development at the Early Innovation Stage

Innovation is a cornerstone for an organization’s survival and success in the global competitive landscape in the VUCA world. The New Product Development (NPD) process is a crucial part of the portfolio and Innovation Management (IM) process. The leadership of an organization has a disproportionate impact on the outcome of innovation activities. Their involvement in IM and NPD is critical for success, considering they make strategic decisions to allocate resources for business growth. The leadership team demands a holistic picture of ideas before making decisions at early stages. The leadership challenge in decision making is that they have a limited time to make decisions by understanding many related aspects and insights quickly. The visual approaches have been vital in management practices to understand the situation and aid in decision-making by supporting cognitive processes. The fundamental problem in using visual representation is hidden expectations of leadership teams to represent needed elements to aid in strategic decision-making by leadership at the early stage of innovation. Also, the configuration of elements and interplay is another issue. The core challenge lies in understanding the effectiveness of currently used visual representations and then improving them by identifying needed elements and their configuration and placement in the visual representation. The paper takes literature review, expert interviews, and brainstorming approaches to distill the challenges to the concrete research questions and objectives. Providing solutions to the open research questions and challenges can significantly enhance the quality and speed of innovation-related decision-making.

Open Challenges and Research Roadmap: Visual Representations aiding Organizational Leadership in Decision Making for New Product Development at the Early Innovation Stage

Innovation is a cornerstone for an organization’s survival and success in the global competitive landscape in the VUCA world. The New Product Development (NPD) process is a crucial part of the portfolio and Innovation Management (IM) process. The leadership of an organization has a disproportionate impact on the outcome of innovation activities. Their involvement in IM and NPD is critical for success, considering they make strategic decisions to allocate resources for business growth. The leadership team demands a holistic picture of ideas before making decisions at early stages. The leadership challenge in decision making is that they have a limited time to make decisions by understanding many related aspects and insights quickly. The visual approaches have been vital in management practices to understand the situation and aid in decision-making by supporting cognitive processes. The fundamental problem in using visual representation is hidden expectations of leadership teams to represent needed elements to aid in strategic decision-making by leadership at the early stage of innovation. Also, the configuration of elements and interplay is another issue. The core challenge lies in understanding the effectiveness of currently used visual representations and then improving them by identifying needed elements and their configuration and placement in the visual representation. The paper takes literature review, expert interviews, and brainstorming approaches to distill the challenges to the concrete research questions and objectives. Providing solutions to the open research questions and challenges can significantly enhance the quality and speed of innovation-related decision-making.

Building a Bridge for Inclusive Assessment of Newly-Arrived Migrants' Knowledge in Science and Mathematics

The Erasmus+/KA3 project Augmented Assessment “Assessing newly arrived migrants' knowledge in Science and Math using augmented teaching material” aims to address the gap that exists in assessing newly arrived migrant students' prior knowledge in the fields of science and mathematics caused by the linguistic obstacle between them and the teachers. To address this gap, the project will develop the Augmented Assessment Library as well as a teachers' training course focusing on inclusive assessment and augmented reality. The chapter outlines the theoretical orientations of the project (augmented assessment bridges) and discusses the elements that comprise them focusing on the connections among inclusive pedagogy, visual representations in science and math education, multimodality, and augmented reality. It also describes the pedagogical framework underpinning the design of the Augmented Assessment Training Course as well as the main innovation of the project which is the Augmented Assessment Library and its pedagogical value for assessment.