Relevance Assessments for Web Search Evaluation: Should We Randomise or Prioritise the Pooled Documents?

In the context of depth- k pooling for constructing web search test collections, we compare two approaches to ordering pooled documents for relevance assessors: The prioritisation strategy (PRI) used widely at NTCIR, and the simple randomisation strategy (RND). In order to address research questions regarding PRI and RND, we have constructed and released the WWW3E8 dataset, which contains eight independent relevance labels for 32,375 topic-document pairs, i.e., a total of 259,000 labels. Four of the eight relevance labels were obtained from PRI-based pools; the other four were obtained from RND-based pools. Using WWW3E8, we compare PRI and RND in terms of inter-assessor agreement, system ranking agreement, and robustness to new systems that did not contribute to the pools. We also utilise an assessor activity log we obtained as a byproduct of WWW3E8 to compare the two strategies in terms of assessment efficiency. Our main findings are: (a) The presentation order has no substantial impact on assessment efficiency; (b) While the presentation order substantially affects which documents are judged (highly) relevant, the difference between the inter-assessor agreement under the PRI condition and that under the RND condition is of no practical significance; (c) Different system rankings under the PRI condition are substantially more similar to one another than those under the RND condition; and (d) PRI-based relevance assessment files (qrels) are substantially and statistically significantly more robust to new systems than RND-based ones. Finding (d) suggests that PRI helps the assessors identify relevant documents that affect the evaluation of many existing systems, including those that did not contribute to the pools. Hence, if researchers need to evaluate their current IR systems using legacy IR test collections, we recommend the use of those constructed using the PRI approach unless they have a good reason to believe that their systems retrieve relevant documents that are vastly different from the pooled documents. While this robustness of PRI may also mean that the PRI-based pools are biased against future systems that retrieve highly novel relevant documents, one should note that there is no evidence that RND is any better in this respect.

Effects of IL-7 on sepsis: a systematic review

Objective: The aim of this systematic review to evaluate the effects and possible therapeutic relevance of interleukin-7, a novel immunoadjuvant agent, against sepsis. Data Sources: We have searched PubMed, Scopus, LILACS and SciELO for any study which had IL-7 as intervention for sepsis. Study Selection: For each source, a single author was chosen to select all studies based on title and abstract. The remaining two authors then proceeded to read the complete text for relevance assessment. Results: Eleven sources were retrieved, consisting of six studies performed on animals, four on ex vivo human cells, and a single clinical trial. Several parameters were evaluated and reported, such as improved animal survival (3 trials; n= 264); improved cell count, activation, cytokine secretion, proliferation and survival; and higher patient CD4+ and CD8+ lymphocyte count (1 trial; n= 27). Major limitations were the heterogeneity of studies and the complexity of sepsis presentation. Data Extraction: Any data related to IL-7 intervention in septic patients was extracted. Animal and in vitro data was chosen according to outcomes in either overall survival or immune response markers. The data extraction from each study was conducted by a single author and then revised by the other two. Bias was assessed by all three authors together, without any specific methodology. Conclusions: Although IL-7 as a potential therapy option for sepsis still requires further investigation and clinical evaluation, current studies show promising results of how the interleukin could act against the disorder.

Data-Seeking Behaviour in the Social Sciences

Purpose Publishing research data for reuse has become good practice in recent years. However, not much is known on how researchers actually find said data. In this exploratory study, we observe the information-seeking behaviour of social scientists searching for research data to reveal impediments and identify opportunities for data search infrastructure.Methods We asked 12 participants to search for research data and observed them in their natural environment. The sessions were recorded. Afterwards, we conducted semi-structured interviews to get a thorough understanding of their way of searching. From the recordings, we extracted the interaction behaviour of the participants and analysed the spoken words both during the search task and the interview by creating affinity diagrams.Results We found that literature search is more closely intertwined with dataset search than previous literature suggests. Both the search itself and the relevance assessment are very complex, and many different strategies are employed, including the creatively “misuse” of existing tools, since no appropriate tools exist or are unknown to the participants.Conclusion Many of the issues we found relate directly or indirectly to the application of the FAIR principles, but some, like a greater need for dataset search literacy, go beyond that. Both infrastructure and tools offered for dataset search could be tailored more tightly to the observed work processes, particularly by offering more interconnectivity between datasets, literature, and other relevant materials.

Methodology of information relevance assessment

The paper considers a methodology that makes it possible to determine the relationship between useful, distracting and interfering information. The proposed methodology allows you to evaluate the relevance of information supplied to a specialist. The situation when a specialist, based on the information received, makes certain decisions and performs the assigned tasks is considered. The regularity of the change in the probability of performing these functions on the amount of information received and information capacity has been established. The regularity of the change in the probability of performing these functions on the amount of information received and information capacity has been established. It is shown that the equation of information dynamics is an information analogue of T. Malthus's population model. This makes it possible to use well-known methods of analysis and extend the results obtained to other mathematical models. The procedure for selecting models that take into account various restrictive effects is considered. The factor of self-limitation is taken into account. The transition to an informational analogue of the Quetelet-Verhulst population model is justified. The external limiting factor is considered in the context of the influence of distracting and disturbing information. A predictive model for determining the pattern of changes in information ability is proposed. This model takes into account the speed of information flow under ideal conditions and the destructive influence of distracting and interfering information. The procedure for using the model is reviewed. This procedure provides an introduction to the calculations additional variable to take into account the influence of distracting and interfering information. The regularity of the change in the additional value and its relationship with information capabilities have been determined. It allows you to go to the procedure for finding out the potential value of the relevance indicator. A relationship has been established to determine the relevance indicators, which make it possible to take into account the potential amount of useful, distracting and interfering information. The relationship between the current and established values of relevance indicators has been established. The classification of the effects of distracting and interfering information is carried out. Relevance characteristics for the prevailing equal and weak influence of interfering and distracting information compared to useful information are analyzed. Methodology of using the relevance characteristics is considered. The ratios that make it possible to quickly assess the relevance indicator at predetermined control points and to determine the time to reach the control values of the observed values are established. The situation that arises under very strong and very weak influences of distracting and interfering information is analyzed.

Development of TOPSIS Technique under Pythagorean Fuzzy Hypersoft Environment Based on Correlation Coefficient and Its Application towards the Selection of Antivirus Mask in COVID-19 Pandemic

The correlation coefficient between two variables plays an important role in statistics. Also, the accuracy of relevance assessment depends on information from a set of discourses. The data collected from numerous statistical studies are full of exceptions. The Pythagorean fuzzy hypersoft set (PFHSS) is a parameterized family that deals with the subattributes of the parameters and an appropriate extension of the Pythagorean fuzzy soft set. It is also the generalization of the intuitionistic fuzzy hypersoft set (IFHSS), which is used to accurately assess insufficiency, anxiety, and uncertainties in decision-making. The PFHSS can accommodate more uncertainties compared to the IFHSS, and it is the most substantial methodology to describe fuzzy information in the decision-making process. The core objective of the this study is to develop the notion and features of the correlation coefficient and the weighted correlation coefficient for PFHSS and to introduce the aggregation operators such as Pythagorean fuzzy hypersoft weighted average (PFHSWA) and Pythagorean fuzzy hypersoft weighted geometric (PFHSWG) operators under the PFHSS scenario. A prioritization technique for order preference by similarity to the ideal solution (TOPSIS) under PFHSS based on correlation coefficients and weighted correlation coefficients is presented. Through the developed methodology, a technique for solving multiattribute group decision-making (MAGDM) problem is planned. Also, the importance of the developed methodology and its application in indicating multipurpose antivirus mask throughout the COVID-19 pandemic period is presented. A brief comparative analysis is described with the advantages, effectiveness, and flexibility of numerous existing studies that demonstrate the effectiveness of the proposed method.

Climate Change and Child Health - Assessment of Parents’ Perspective and Relevance

BackgroundOngoing climate change has several indirect and direct health consequences. Children are among the most vulnerable group to suffer from these health impacts. Parents as caregivers play a particularly important role in protecting them adequately.ObjectivesThe aim of the study was to investigate how parents perceive the health consequences of climate change. Of particular interest were their information status and already used communication channels, to make them more addressable. In addition, their risk perception and relevance estimation of the health consequences of climate change for their children were investigated.MethodsParents were able to participate anonymously in the study from March to June 2020 by means of an online questionnaire. The study sample consisted of 292 parents living in Germany (age: M = 42.02; SD = 7.73; sex: ♀ = 190; ♂ = 54) with at least one child aged 0 to 18 years. Open-ended questions and closed-ended questions with Likert scales were used. Data were analyzed descriptively. Correlations and regression analysis were performed.FindingsAbout 75% of the respondents knew at least one health consequence of climate change. Heat related illnesses were named as the most important health consequences (74.1%). Parents obtain most of their information from the Internet (73.3%). The increase of air pollutants is estimated as the most relevant risk for the health of their children. Relevance estimation affects risk perception.ConclusionParents are aware of the importance of being informed about the health consequences of climate change. Therefore, knowledge about the health consequences in relation to the relevance assessment must be communicated clearly and sufficiently. Information channels already used should be improved and further multipliers identified.

Clinical relevance assessment of animal preclinical research (RAA) tool: development and explanation

Background Only a small proportion of preclinical research (research performed in animal models prior to clinical trials in humans) translates into clinical benefit in humans. Possible reasons for the lack of translation of the results observed in preclinical research into human clinical benefit include the design, conduct, and reporting of preclinical studies. There is currently no formal domain-based assessment of the clinical relevance of preclinical research. To address this issue, we have developed a tool for the assessment of the clinical relevance of preclinical studies, with the intention of assessing the likelihood that therapeutic preclinical findings can be translated into improvement in the management of human diseases. Methods We searched the EQUATOR network for guidelines that describe the design, conduct, and reporting of preclinical research. We searched the references of these guidelines to identify further relevant publications and developed a set of domains and signalling questions. We then conducted a modified Delphi-consensus to refine and develop the tool. The Delphi panel members included specialists in evidence-based (preclinical) medicine specialists, methodologists, preclinical animal researchers, a veterinarian, and clinical researchers. A total of 20 Delphi-panel members completed the first round and 17 members from five countries completed all three rounds. Results This tool has eight domains (construct validity, external validity, risk of bias, experimental design and data analysis plan, reproducibility and replicability of methods and results in the same model, research integrity, and research transparency) and a total of 28 signalling questions and provides a framework for researchers, journal editors, grant funders, and regulatory authorities to assess the potential clinical relevance of preclinical animal research. Conclusion We have developed a tool to assess the clinical relevance of preclinical studies. This tool is currently being piloted.