Component-based Analysis of Dynamic Search Performance

In many search scenarios, such as exploratory, comparative, or survey-oriented search, users interact with dynamic search systems to satisfy multi-aspect information needs. These systems utilize different dynamic approaches that exploit various user feedback granularity types. Although studies have provided insights about the role of many components of these systems, they used black-box and isolated experimental setups. Therefore, the effects of these components or their interactions are still not well understood. We address this by following a methodology based on Analysis of Variance (ANOVA). We built a Grid Of Points that consists of systems based on different ways to instantiate three components: initial rankers, dynamic rerankers, and user feedback granularity. Using evaluation scores based on the TREC Dynamic Domain collections, we built several ANOVA models to estimate the effects. We found that (i) although all components significantly affect search effectiveness, the initial ranker has the largest effective size, (ii) the effect sizes of these components vary based on the length of the search session and the used effectiveness metric, and (iii) initial rankers and dynamic rerankers have more prominent effects than user feedback granularity. To improve effectiveness, we recommend improving the quality of initial rankers and dynamic rerankers. This does not require eliciting detailed user feedback, which might be expensive or invasive.

Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to Rank

Learning to rank from logged user feedback, such as clicks or purchases, is a central component of many real-world information systems. Different from human-annotated relevance labels, the user feedback is always noisy and biased. Many existing learning to rank methods infer the underlying relevance of query–item pairs based on different assumptions of examination, and still optimize a relevance based objective. Such methods rely heavily on the correct estimation of examination, which is often difficult to achieve in practice. In this work, we propose a general framework U-rank+ for learning to rank with logged user feedback from the perspective of graph matching. We systematically analyze the biases in user feedback, including examination bias and selection bias. Then, we take both biases into consideration for unbiased utility estimation that directly based on user feedback, instead of relevance. In order to maximize the estimated utility in an efficient manner, we design two different solvers based on Sinkhorn and LambdaLoss for U-rank+ . The former is based on a standard graph matching algorithm, and the latter is inspired by the traditional method of learning to rank. Both of the algorithms have good theoretical properties to optimize the unbiased utility objective while the latter is proved to be empirically more effective and efficient in practice. Our framework U-rank+ can deal with a general utility function and can be used in a widespread of applications including web search, recommendation, and online advertising. Semi-synthetic experiments on three benchmark learning to rank datasets demonstrate the effectiveness of U-rank+ . Furthermore, our proposed framework has been deployed on two different scenarios of a mainstream App store, where the online A/B testing shows that U-rank+ achieves an average improvement of 19.2% on click-through rate and 20.8% improvement on conversion rate in recommendation scenario, and 5.12% on platform revenue in online advertising scenario over the production baselines.

PNTRS

A news recommendation system not only must recommend the latest, trending and personalized news to the users but also give opportunity to know about the people’s opinion on trending news. Most of the existing news recommendation systems focus on recommending news articles based on user-specific tweets. In contrast to these recommendation systems, the proposed Personalized News and Tweet Recommendation System (PNTRS) recommends tweets based on the recommended article. It firstly generates news recommendation based on user’s interest and twitter profile using the Multinomial Naïve Bayes (MNB) classifier. Further, the system uses these recommended articles to recommend various trending tweets using fuzzy inference system. Additionally, feedback-based learning is applied to improve the efficiency of the proposed recommendation system. The user feedback rating is taken to evaluate the satisfaction level and it is 7.9 on the scale of 10.

A Survey of HISs with Information Systems Success Model

Hospital Information System (HIS) is a form of healthcare information system that is globalized and applied in the medical sector. Researchers, doctors, and management are all interested in the rate of success of HISs; therefore it's a continuous study topic. At this research, we created a new tool to assess the success rate of HIS in a medical center based on the perspectives of users. The research was place in Ebnesina and Mashhad, Persia, at the Dr. Hejazi Mental Center and Educational Facility. A self-administered standardized questionnaire based on Information Systems Success Model (ISSM) was used to gather data, and it included seven factors: systems quality, data quality, quality of service, system use, applicability, fulfillment, and positive externalities. An advisory group checked the content's legitimacy. Cronbach alpha was used to test the consistency and stability of dimensions. To examine the importance of relationships between variables, Correlation and regression was determined. On the basis of user feedback, the HIS rate of success has been established. The research included a approximately 125 participants. A content validity index (CVI) of 0.8 and a validity ratio (CVR) of 0.86 were used by an advisory committee to verify the item. The instruments have an overall Cronbach's alpha of 0.9. Between the analyzed dimensions, the Pearson’s correlation coefficient revealed substantial positive connections. In the institution under investigation, the HIS rate of success averaged 65 percent. (CI: 64 percent, 67 percent). The greatest success rates were found in the aspects of "effectiveness," "systems quality," and "positive externalities." Future research might employ the tool used in this research to evaluate HIS. In this research, a technique for calculating the HIS rate of success depending on user feedback was established. This strategy enables institutional HIS chances of success to be compared. Our results also highlight the perspectives of HIS clients in a developing economy.

Therapeutic Gaming for Adolescent Anxiety

Anxiety disorders (AD) are the most prevalent of the mental health conditions and are associated with significant and long-lasting burden of disease both for affected individuals and healthcare systems designed to support them. Despite this, barriers to traditional interventions mean less than half of adolescents experiencing ADs seek-treatment, with less than 20% of treatment-seekers ultimately receiving a scientifically validated intervention. Therapeutic games show significant potential to help reduce AD in adolescents, with some concerns remaining over their abilities to engage users, particularly over time. The chapter presents two studies relating to the development of a new mobile gamified intervention for adolescents with AD. This includes a user-feedback study on currently available games for anxiety and depression, followed by a user-feedback, acceptability, and intention-to-use study of a development version of the new intervention.

Measure the impressions of users of smart applications for performing Umrah, prayers, and visits in the Grand Mosque and the Prophet's Mosque: Field Study: قياس انطباع مستخدمي التطبيقات الذكية الخاصة بأداء العمرة، والصلوات، والزيارة في المسجد الحرام والمسجد النبوي: دراسة ميدانية

The study aimed to measure the impressions of users of smart applications for performing Umrah, prayers, and visits in the Grand Mosque and the Prophet's Mosque. In order to achieve the objectives of the study, the descriptive and analytical approach was used, and the study sample consisted of (412) users of smart applications, and a questionnaire was designed to measure the users ’impression consisting of four main axes, the results showed: The axis (the benefit of users of smart applications in (Creating spacing) and limiting the spread of Corona disease (Covid-19) to a degree (strongly agree), in second place (the impression of users of smart applications for performing Umrah, prayers, and visits in the Grand Mosque and the Prophet's Mosque) with a degree (strongly agree), in the third place came the axis (the extent to which users of special smart applications benefit (by their performance) Umrah, prayers, and visits in the Grand Mosque and the Prophet's Mosque) with a degree (strongly agree), in the last place (Challenges of using smart applications for performing Umrah, prayers, and visiting in the Grand Mosque and the Prophet's Mosque), with a degree of (neutral). The study recommended the need to educate visitors and pilgrims about the importance and how to use the Smart applications for Umrah, pilgrimage and prayers, and the necessity to continuously develop applications based on user feedback and the results of studies to achieve the best service for visitors, and work to integrate all these applications into one application to facilitate its use, and the need for the Ministry of Hajj and Umrah to pay attention to supervising the content provided through the available interactive applications.

Moderator and observer experiences and decision-making: Remote usability testing of a web-based empathy training portal (Preprint)

BACKGROUND COVID 2019 restrictions severely curtailed empirical endeavors that involved in-person human interaction. The pandemic also stimulated our team to embrace technology in a two-fold manner. First, we created novel technology to help us to overcome pandemic restrictions in teaching empathic communication in the traditional classroom. A web-based training portal was created for users to learn empathy in an accessible, compelling, self-directed, and interactive online environment. Second, we harnessed technology to engage in remote usability testing and data collection with prospective users of our training portal. In developing our protocol, we discovered gaps in the literature on moderator and silent observer roles and experiences in conducting remote usability testing. OBJECTIVE The aim of this paper is to share our remote moderator and silent observer experiences and their use of certain tools to capture user feedback and experiences with the app. METHODS The larger-scale project employed a quantitative and think-aloud qualitative problem-discovery usability study design. Three trained research assistants collected and utilized user feedback from eight users who were asked to complete tasks in three sessions. Each research assistant had assigned roles and were asked to qualitatively describe their roles, experiences and reactions to the usability testing protocol, and suggestions for improved techniques and strategies for conducting remote usability testing. RESULTS Major results are described in relation to the research assistant experiences with the study protocol followed by recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel. CONCLUSIONS Our findings highlighted that as we move towards greater remote usability testing, we also need more comprehensive understanding of human-computer interaction and its impact on usability testing outcomes. Our team realized that accessible comprehensive web-conferencing platform to conduct remote sessions is not sufficient. Lead moderator and silent observers offered their insights and recommendations for the ongoing creation and testing of training materials for their respective roles with a focus on: online interpersonal communication skills, conducting user testing protocols, troubleshooting technology and test user issues, proficiency in web-conferencing plus behavior analysis and feedback technologies, and time management. CLINICALTRIAL None.

User Feedback Analysis of an AI-Enabled CBT Mental Health Application (Wysa) (Preprint)

BACKGROUND Digital mental health applications (apps) are rapidly becoming a common source of accessible support across the world, but their effectiveness is often influenced by limited helpfulness and engagement. There is currently a scarcity of research exploring user engagement in digital mental health applications, especially in the space of artificial intelligence (AI) guided applications. OBJECTIVE The study’s primary objective was to analyze feedback content to understand the user’s experiences of engaging with a digital mental health app. As a secondary objective, an exploratory analysis captured the types of mental health app users. METHODS This study utilized a user-led approach to understanding factors for engagement and helpfulness in digital mental health by analyzing feedback (n=7,929) reported on Google Play Store about Wysa, a mental health app (1 year period). The analysis of keywords in user feedback categorized and evaluated the reported user experience into the core domains of acceptability, usability, usefulness, and integration. The study also captured key deficits and strengths of the app, and explored salient characteristics of the types of users who benefit from accessible digital mental health support. RESULTS The analysis of user feedback found the app to be overwhelmingly positively reviewed (84.4% 5-star rating). The themes of engaging exercises, interactive interface and AI-conversational ability indicated the acceptability of the app, while the non-judgementality and ease of conversation highlighted its usability. The app’s usefulness was portrayed by themes such as improvement in mental health, convenient access and cognitive restructuring exercises. Themes of Privacy and Confidentiality underscored users’ preference for the integrated aspects of the app. Further analysis revealed 4 predominant types of individuals who shared app feedback on the store. CONCLUSIONS Users reported therapeutic elements of a comfortable, safe, and supportive environment through using the digital mental health app. Digital mental health apps may expand mental health access to those unable to access traditional forms of mental health support and treatments.