Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

Reinforcement learning (RL) methods usually treat reward functions as black boxes. As such, these methods must extensively interact with the environment in order to discover rewards and optimal policies. In most RL applications, however, users have to program the reward function and, hence, there is the opportunity to make the reward function visible – to show the reward function’s code to the RL agent so it can exploit the function’s internal structure to learn optimal policies in a more sample efficient manner. In this paper, we show how to accomplish this idea in two steps. First, we propose reward machines, a type of finite state machine that supports the specification of reward functions while exposing reward function structure. We then describe different methodologies to exploit this structure to support learning, including automated reward shaping, task decomposition, and counterfactual reasoning with off-policy learning. Experiments on tabular and continuous domains, across different tasks and RL agents, show the benefits of exploiting reward structure with respect to sample efficiency and the quality of resultant policies. Finally, by virtue of being a form of finite state machine, reward machines have the expressive power of a regular language and as such support loops, sequences and conditionals, as well as the expression of temporally extended properties typical of linear temporal logic and non-Markovian reward specification.

Proposal for an IIoT Device Solution According to Industry 4.0 Concept

Today, Industrial Internet of Things (IIoT) devices are very often used to collect manufacturing process data. The integration of industrial data is increasingly being promoted by the Open Platform Communications United Architecture (OPC UA). However, available IIoT devices are limited by the features they provide; therefore, we decided to design an IIoT device taking advantage of the benefits arising from OPC UA. The design procedure was based on the creation of sequences of steps resulting in a workflow that was transformed into a finite state machine (FSM) model. The FSM model was transformed into an OPC UA object, which was implemented in the proposed IIoT. The OPC UA object makes it possible to monitor events and provide important information based on a client’s criteria. The result was the design and implementation of an IIoT device that provides improved monitoring and data acquisition, enabling improved control of the manufacturing process.

Appreciative Learning for Immersive Reward System in Education Game Development

Game replayability is very important in serious game to maximize the understanding for the learning content. The replayability is the result from the gameplay experience. Games have the advantage of providing a fun experience, and immersion is a vital element in game design to produce the experience. However, the design of immersion in games is often not well conceptualized so that it does not produce the expected experience. This study uses Appreciative Learning based reward system, which focuses on positive things such as peak achievements, opportunities, exploration of potential and optimism for the future. The reward activity consists of four stages, namely Discovery, Dream, Design and Destiny. Reward personalization is done by regulating reward behavior using artificial intelligence which runs in all four stages. Appreciative Learning will be used to design immersive experiences consisting of sensory, imaginary and challenge-based immersion, which are the three main elements of immersive games. Intelligent agent behavior is modeled using the Finite State Machine. This study produces an immersive reward design that is applied to the concept of Appreciative Learning in designing a serious game.

APPLICATION OF SOME MODELS OF POWER SUPPLY SYSTEMS WITH AN EXAMPLE OF TAKING INTO ACCOUNT THEIR RELIABILITY FOR INDUSTRIAL ENTERPRISES IN PRACTICE

The publication presents the most common and promising models and schemes for replacing the power supply system of industrial enterprises, and an example of their use in practice for power supply to sewage treatment plants of a chemical enterprise. Various types of models are considered (in tabular form, in the form of an electrical replacement circuit or in the form of a finite-state machine), as well as a logical-probabilistic approach to the representation of the power supply system. The use of automation systems in power supply systems of industrial enterprises can reduce the total number of accidents, prevent their development, reduce the time of disconnections of electrical installations and downtime of mechanisms, and also allows to transfer to work without constant maintenance a significant number of electrical installations of substations, which, in turn, leads to a reduction in the number of service personnel, increase labor productivity and reduce maintenance costs. One of the main tasks on the way to automating the power supply system of enterprises is to present it in the form of a model. The list of replacement models and schemes that can be used to represent the power supply system of enterprises is quite large, and choosing the appropriate model, for that best meets the requirements of the final goal is quite a complex scientific task. The purpose of this article is to analyze the most common and promising models and schemes for replacing the power supply system and provide recommendations on their application in practice for power supply to sewage treatment plants of a chemical enterprise. Presenting the power supply system in the form of a table allows you to simplify and visually perform certification of electrical equipment, identify problem and weak points in the power supply system of industrial enterprises. Such a table proves that a large amount of electrical equipment at lower levels depends on a single electrical device at higher levels. The representation of the power supply system in the form of a three-phase symmetric replacement circuit can be used to calculate short-circuit currents in symmetric and unbalanced modes in the MatLab program and similar. Using the model of the power supply system in the form of a finite-state machine allows you to visually check the reliability of power supply to consumers, simplify the design of the power supply system of an industrial enterprise, select the state of switching devices in the power supply system and take into account the advantages and disadvantages of each state. It is advisable to use the methods discussed in this article to represent the power supply system (in tabular form, in the form of a finite-state machine or a three-phase symmetric replacement scheme) and their further application and use in automated power supply systems of industrial enterprises. The logical-probabilistic method allows us to consider the issue of power supply reliability based on the theory of logical algebra.

Aplikasi Perancang Abstraksi Verilog Mesin Keadaan Terbatas Otomatis

Saat ini, hampir semua perangkat elektronik menggunakan prosesor di dalamnya. Dalam sebuah prosesor, terdapat bagian control unit yang berfungsi mengatur operasi dari komponen-komponen di dalam prosesor. Control unit merupakan sebuah mesin keadaan terbatas atau disebut finite state machine (FSM). Rangkaian FSM dapat disintesis secara manual ataupun secara otomatis menggunakan bahasa abstraksi Verilog. Dalam penelitian ini, dibuat sebuah aplikasi yang dapat membantu pengguna merancang FSM dan selanjutnya menyimpannya dalam format Verilog. Aplikasi yang dibuat secara fungsional dapat berjalan dengan kesesuaian 100% dan mampu untuk membuat rancangan Verilog untuk FSM dengan berbagai model dan teknik pengkodean state. Simulasi modul Verilog yang dihasilkan juga sesuai dengan spesifikasi rangkaian FSM yang dirancang.

Augmenting Customer Support with an NLP-based Receptionist

In this paper, we show how a Portuguese BERT model can be combined with structured data in order to deploy a chatbot based on a finite state machine to create a conversational AI system that helps a real-estate company to predict its client's contact motivation. The model achieves human level results in a dataset that containts 235 unbalanced labels. Then, we also show its benefits considering the business impact comparing it against classical NLP methods.

Efficient parallel derivation of short distinguishing sequences for nondeterministic finite state machines using MapReduce

Distinguishing sequences are widely used in finite state machine-based conformance testing to solve the state identification problem. In this paper, we address the scalability issue encountered while deriving distinguishing sequences from complete observable nondeterministic finite state machines by introducing a massively parallel MapReduce version of the well-known Exact Algorithm. To the best of our knowledge, this is the first study to tackle this task using the MapReduce approach. First, we give a concise overview of the well-known Exact Algorithm for deriving distinguishing sequences from nondeterministic finite state machines. Second, we propose a parallel algorithm for this problem using the MapReduce approach and analyze its communication cost using Afrati et al. model. Furthermore, we conduct a variety of intensive and comparative experiments on a wide range of finite state machine classes to demonstrate that our proposed solution is efficient and scalable.

Digital System Design for Traffic Light Controller System: A Systematic Approach

In this paper, we are going to present the finite state machine, how to implement it via hardware description language (HDL), and how to use it in a real application. At first, the specification and requirements of traffic light controller are stated. Then, the system architecture based on finite state machine (FSM) are conducted. Finally, the way of using HDL as well as the test-bench simulation are given in detail. Keywords : Digital system design, System on chip, Finite State Machine, Digital Design Education, Smart Classroom.