论文检索

Qiming Bao, Juho Leinonen, Alex Yuxuan Peng, Wanjun Zhong, Gaël Gendron, Timothy Pistotti, Alice Huang, Paul Denny, Michael Witbrock, Jiamou Liu

Large language models (LLMs) have demonstrated strong capabilities in language understanding and generation, and their potential in educational contexts is increasingly being explored. One promising area is learnersourcing, where students engage in creating their own educational content, such as multiple-choice questions. A critical step in this process is generating effective explanations for the solutions to these questions, as such explanations aid in peer understanding and promote deeper conceptual learning. However, students often find it difficult to craft high-quality explanations due to limited understanding or gaps in their subject knowledge. To support this task, we introduce ``ILearner-LLM,'' a framework that uses iterative enhancement with LLMs to improve generated explanations. The framework combines an explanation generation model and an explanation evaluation model fine-tuned using student preferences for quality, where feedback from the evaluation model is fed back into the generation model to refine the output. Our experiments with LLaMA2-13B and GPT-4 using five large datasets from the PeerWise MCQ platform show that ILearner-LLM produces explanations of higher quality that closely align with those written by students. Our findings represent a promising approach for enriching the learnersourcing experience for students and for leveraging the capabilities of large language models for educational applications.

Kevin Paeth, Daniel Atherton, Nikiforos Pittaras, Heather Frase, Sean McGregor

As artificial intelligence (AI) systems become increasingly deployed across the world, they are also increasingly implicated in AI incidents – harm events to individuals and society. As a result, industry, civil society, and governments worldwide are developing best practices and regulations for monitoring and analyzing AI incidents. The AI Incident Database (AIID) is a project that catalogs AI incidents and supports further research by providing a platform to classify incidents for different operational and research-oriented goals. This study reviews the AIID’s dataset of 750+ AI incidents and two independent taxonomies applied to these incidents to identify common challenges to indexing and analyzing AI incidents. We find that certain patterns of AI incidents present structural ambiguities that challenge incident databasing and explore how epistemic uncertainty in AI incident reporting is unavoidable. We therefore report mitigations to make incident processes more robust to uncertainty related to cause, extent of harm, severity, or technical details of implicated systems. With these findings, we discuss how to develop future AI incident reporting practices.

Sean McGregor, Allyson Ettinger, Nick Judd, Paul Albee, Liwei Jiang, Kavel Rao, William H. Smith, Shayne Longpre, Avijit Ghosh, Christopher Fiorelli 等

In August of 2024, 495 hackers generated evaluations in an open-ended bug bounty targeting the Open Language Model (OLMo) from The Allen Institute for AI. A vendor panel staffed by representatives of OLMo's safety program adjudicated changes to OLMo's documentation and awarded cash bounties to participants who successfully demonstrated a need for public disclosure clarifying the intent, capacities, and hazards of model deployment. This paper presents a collection of lessons learned, illustrative of flaw reporting best practices intended to reduce the likelihood of incidents and produce safer large language models (LLMs). These include best practices for safety reporting processes, their artifacts, and safety program staffing.

Akash V. Maharaj, David Arbour, Daniel Lee, Uttaran Bhattacharya, Anup Rao, Austin Zane, Avi Feller, Kun Qian, Yunyao Li

Enterprise AI Assistants are increasingly deployed in domains where accuracy is paramount, making each erroneous output a potentially significant incident. This paper presents a comprehensive framework for monitoring, benchmarking, and continuously improving such complex, multi-component systems under active development by multiple teams. Our approach encompasses three key elements: (1) a hierarchical ``severity'' framework for incident detection that identifies and categorizes errors while attributing component-specific error rates, facilitating targeted improvements; (2) a scalable and principled methodology for benchmark construction, evaluation, and deployment, designed to accommodate multiple development teams, mitigate overfitting risks, and assess the downstream impact of system modifications; and (3) a continual improvement strategy leveraging multidimensional evaluation, enabling the identification and implementation of diverse enhancement opportunities. By adopting this holistic framework, organizations can systematically enhance the reliability and performance of their AI Assistants, ensuring their efficacy in critical enterprise environments. We conclude by discussing how this multifaceted evaluation approach opens avenues for various classes of enhancements, paving the way for more robust and trustworthy AI systems.

Moomal Farhad, Mohammad Mehedy Masud, Aisha Alnaqbi, Rawan Mubarak, Aaisha Aladawi, Sara Alnaqbi

Children's mental health is crucial for their development, but it's often overlooked, leading to psychological issues. Many children struggle to express their thoughts and feelings effectively. To address this issue, we have proposed a novel approach to analyze children's drawings for psychological screening using artificial intelligence. Specifically, we're focusing on the `draw a person' (DAP) test, where a child's drawing is used to identify potential indicators of their mental and emotional state. Thus, we are introducing an AI-powered technique to automate the psychological screening process for children using the DAP test, which a human professional would traditionally conduct. The screening tool would suggest whether the child needs or doesn’t need further psychological referral. We have collected a dataset consisting of children's drawings and labeled them by experts as either `need' or `no need', indicating whether the child needs or does not need a referral. We have proposed two alternative approaches for the screening process. The first approach consists of extracting features from the drawings following expert guidelines and training a classification model using the features to classify the drawing as either `need' or `no need'. We also propose an out-of-the-box technique applying prompt engineering on state-of-the-art LLMs to automatically extract features from the images. The second approach involves training an image classification model using the drawings. Both approaches are challenged by the issue of class imbalance, as most of the drawings correspond to the `no-need' class. To address this challenge, we introduce Siamese++, a novel Siamese network for image classification, which uses feature embedding and an adaptive distance threshold for classification, instead of the nearest neighbor classification employed by traditional Siamese. Our proposed method achieves a high F1 score (up to 88%) even with a large class imbalance and without the need for any image augmentation. Thus, we have proposed an innovative interdisciplinary integration of AI with psychology and developed novel techniques to solve the real-world problem of psychological screening.

Ping Chen, Jonathan Leung, Yinan Zhang, Zhiqi Shen

Health and longevity are topics of great interest, leading to an exploration of the Japanese concept of ikigai, known for its impact on a fulfilling, extended life. Ikigai levels are dynamic, changing with personal growth and life situations, but traditional assessment methods are time-consuming, discouraging frequent tracking. In this paper, we propose Personalized Optimization and Wellbeing Enhancement Recommendation (POWER), which integrates an ikigai simulator to pre- dict ikigai levels from profile information and a hobby recommender that uses reinforcement learning to adapt recommendations based on continuous user feedback. Our methods, validated through both offline data and an online user study, effectively capture and enhance ikigai.

Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards 等

As cyber threats become increasingly automated and sophisticated, novel solutions must be introduced to improve defence of enterprise networks. Deep Reinforcement Learning (DRL) has demonstrated potential in mitigating these advanced threats. Single DRL Agents have proven utility toward execution of autonomous cyber defence. Despite the success of employing single DRL Agents, this approach presents significant limitations, especially regarding scalability within large enterprise networks. An attractive alternative to the single agent approach is the use of Multi-Agent Reinforcement Learning (MARL). However, developing MARL agents is costly with few options for examining MARL cyber defence techniques against adversarial agents. This paper presents a MARL network security environment, the fourth iteration of the Cyber Autonomy Gym for Experimentation (CAGE) challenges. This challenge was specifically designed to test the efficacy of MARL algorithms in an enterprise network. Our work aims to evaluate the potential of MARL as a robust and scalable solution for autonomous network defence.

Serdar Kadioglu, Elton Yechao Zhu, Gili Rosenberg, John Kyle Brubaker, Martin J. A. Schuetz, Grant Salton, Zhihuai Zhu, Helmut G. Katzgraber

In this tool paper, we design, develop, and release BoolXAI, an interpretable machine learning classification approach for Explainable AI (XAI) based on expressive Boolean formulas. The Boolean formula defines a logical rule with tunable complexity according to which input data are classified. Beyond the classical conjunction and disjunction, BoolXAI offers expressive operators such as AtLeast, AtMost, and Choose and their parameterization. This provides higher expressiveness compared to rigid rules- and tree-based approaches. We show how to train BoolXAI classifiers effectively using native local optimization to search the space of feasible formulas. We provide illustrative results on several well-known public benchmarks that demonstrate the competitive nature of our approach compared to existing methods. Our work is embodied in the open-source BoolXAI library with a high-level user interface to serve researchers and practitioners. BoolXAI can be used either as a standalone interpretable classifier or for post-hoc explanations of other black-box models or observed behavior. We highlight several desirable benefits of our tool, especially in industrial settings where rapid experimentation, reusability, reproducibility, deployment, and maintenance are of great interest. Finally, we showcase a deployed service powered by BoolXAI as an enterprise application.

Evana Gizzi, Timothy Chase Jr, Christian Cassamajor-Paul, Rachael Chertok, Lily Clough, Connor Firth, Alan Gibson, Ibrahim Haroon, James Marshall, Patrick Maynard 等

Infusing artificial intelligence algorithms into production aerospace systems can be challenging due to costs, timelines, and a risk-averse industry. We introduce the Onboard Artificial Intelligence Research (OnAIR) platform, an open-source software pipeline and cognitive architecture tool that enables full life cycle AI research for on-board intelligent systems. We begin with a description and user walk-through of the OnAIR tool. Next, we describe four use cases of OnAIR for both research and deployed onboard applications, detailing their use of OnAIR and the benefits it provided to the development and function of each respective scenario. We conclude with remarks on future work, future planned deployments, and goals for the forward progression of OnAIR as a tool to enable a larger AI and aerospace research community.

Vasil Shteriyanov, Rimma Dzhusupova, Jan Bosch, Helena Holmström Olsson

Within the Engineering, Procurement, and Construction (EPC) industry, engineers manually create documents based on engineering drawings, which can be time-consuming and prone to human error. For example, the expansion of typical assemblies of instrument items (Instrument Typicals) in Piping and Instrumentation Diagrams (P&IDs) is a labor-intensive task. Each Instrument Typical assembly is depicted in the P&IDs via a simplified representation showing only a subset of the utilized instruments. The expansion activity involves recording all utilized instruments to create an instrument item list document based on the P&IDs for a particular EPC project. Fortunately, Artificial Intelligence (AI) could help to automate this process. In this paper, we propose the first method for automating the process of Instrument Typical expansion in P&IDs. The method utilizes computer vision techniques and domain knowledge rules to extract information about the Instrument Typicals from a project's P&IDs and legend sheets. Subsequently, the extracted information is used to automatically generate the listing of all utilized instruments. The effectiveness of our method is evaluated on P&IDs from large industrial EPC projects, resulting in precision rates exceeding 98% and recall rates surpassing 99%. These results demonstrate the suitability of our method for industrial deployment. The successful application of our method has the potential to reduce engineering costs and increase the efficiency of EPC projects. Furthermore, the method could be adapted for additional applications in the EPC industry, which highlights the method's industrial value.

Anthony Palladino, Dana Gajewski, Abigail Aronica, Patryk Deptula, Alexander Hamme, Seiyoung C. Lee, Jeff Muri, Todd Nelling, Michael A. Riley, Brian Wong 等

We present a novel Automatic Target Recognition (ATR) system using open-vocabulary object detection and classification models. A primary advantage of this approach is that target classes can be defined just before runtime by a non-technical end user, using either a few natural language text descriptions of the target, or a few image exemplars, or both. Nuances in the desired targets can be expressed in natural language, which is useful for unique targets with little or no training data. We also implemented a novel combination of several techniques to improve performance, such as leveraging the additional information in the sequence of overlapping frames to perform tubelet identification (i.e., sequential bounding box matching), bounding box re-scoring, and tubelet linking. Additionally, we developed a technique to visualize the aggregate output of many overlapping frames as a mosaic of the area scanned during the aerial surveillance or reconnaissance, and a kernel density estimate (or heatmap) of the detected targets. We initially applied this ATR system to the use case of detecting and clearing unexploded ordinance on airfield runways and we are currently extending our research to other real-world applications.

Alon Oved, Segev Shlomov, Sergey Zeltyn, Nir Mashkif, Avi Yaeli

Predicting the next activity in an ongoing process is one of the most common tasks in the business process management (BPM) domain. It allows businesses to optimize resource allocation, enhance operational efficiency, and aid both in risk mitigation and strategic decision-making. Existing state-of-the-art AI models for BPM do not fully capitalize on available semantic information within process event logs. As current advanced AI-BPM systems provide semantically richer textual data, the need for new adequate models grows. To address this gap, we develop SNAP—a novel system that utilizes LLMs by constructing narratives and semantic contextual stories for historical event logs, which are then used to generate precise and actionable predictions for the ongoing process. SNAP was evaluated on six benchmark datasets, where it demonstrated significant performance improvements over eleven SOTA models, particularly on datasets with high levels of semantic content. This work showcases the potential of integrating LLMs in BPM and outlines a clear path toward future deployment, emphasizing the relevance and innovation of our approach within the broader AI application landscape.

John Murzaku, Zifan Liu, Md Mehrab Tanjim, Vaishnavi Muppala, Xiang Chen, Yunyao Li

We present ECLAIR (Enhanced CLArification for Interactive Responses), a novel unified and end-to-end framework for interactive disambiguation in enterprise AI assistants. ECLAIR generates clarification questions for ambiguous user queries and resolves ambiguity based on the user's response. We introduce a generalized architecture capable of integrating ambiguity information from multiple downstream agents, enhancing context-awareness in resolving ambiguities and allowing enterprise specific definition of agents. We further define agents within our system that provide domain-specific grounding information. We conduct experiments comparing ECLAIR to few-shot prompting techniques and demonstrate ECLAIR's superior performance in clarification question generation and ambiguity resolution.

Aditya Mohan, Andie Gomez-Patron, Matthew Pritchard, Hannah Kerner

Geologists seek to understand the relationship between volcanic unrest and eruptions by identifying subtle Volcanic Thermal Features (VTFs) in high-resolution satellite imagery. This analysis requires the careful curation of large databases of relevant volcanic thermal information. However, volcanic unrest is characterized by highly subtle thermal anomalies. Manual identification on a global scale is highly labor- and time-intensive. We propose Hotspotter: an end-to-end system to automatically detect subtle volcanic thermal anomalies in satellite images and derive relevant thermal statistics. Previous solutions for automated VTF detection have limited data size and geographic diversity. To accommodate an unprecedentedly large and diverse volcanic dataset, we propose an automated pipeline combining unsupervised anomaly detection with supervised classification to filter anomalous regions. Hotspotter gives 90% anomaly detection accuracy and robust generalization to new volcanoes. Our automated approach can accelerate scientists' search for VTFs to help identify relevant thermal precursors and enable more precise forecasts of global volcanic eruptions.

Karol Lynch, Fabio Lorenzi, John D Sheehan, Duygu Kabakci-Zorlu, Bradley Eck

We describe an application that uses large language models to generate structured documents related to industrial equipment, specifically focusing on Failure Modes and Effects Analysis (FMEAs). Our novel application uses techniques in structured document generation, in-context learning, and ensembling to create high-quality structured content that subject matter experts supervise through a user-centric interface that presents FMEA entities as UI elements. Novel evaluation metrics for structured document generation are also proposed. Our empirical results, based on 71 asset evaluations, demonstrate the individual and combined contributions of these techniques, with an overall effectiveness that varies between a recall of 0.669 and a precision of 0.91. Qualitative feedback from target users validates the practicality of the described approach to seamlessly integrate expert supervision with generative AI in a labour-saving workflow.

Niclas Boehmer, Yunfan Zhao, Guojun Xiong, Paula Rodriguez-Diaz, Paola Del Cueto Cibrian, Joseph Ngonzi, Adeline Boatin, Milind Tambe

Maternal mortality remains a significant global public health challenge. One promising approach to reducing maternal deaths occurring during facility-based childbirth is through early warning systems, which require the consistent monitoring of mothers' vital signs after giving birth. Wireless vital sign monitoring devices offer a labor-efficient solution for continuous monitoring, but their scarcity raises the critical question of how to allocate them most effectively. We devise an allocation algorithm for this problem by modeling it as a variant of the popular Restless Multi-Armed Bandit (RMAB) paradigm. In doing so, we identify and address novel, previously unstudied constraints unique to this domain, which render previous approaches for RMABs unsuitable and significantly increase the complexity of the learning and planning problem. To overcome these challenges, we adopt the popular Proximal Policy Optimization (PPO) algorithm from reinforcement learning to learn an allocation policy by training a policy and value function network. We demonstrate in simulations that our approach outperforms the best heuristic baseline by up to a factor of 4.

Sayantan Basu, Mohit Kumar, Sivaramakrishnan Kaveri

Time series data, prevalent in fields like medical, e-commerce, finance, etc., is used for forecasting, such as predicting next quarter’s product demand based on past trends. However, some problems necessitate causal models to answer questions like “What the product demand would have been without a specific intervention (e.g., products with slower delivery time suppressed from the search results)?” Such questions require causal models to estimate unobserved counterfactual outcome. In this paper, we propose a novel Graph Causal Forecasting (GCF) model, that predicts the unobserved demand leveraging the relationship of a product with other similar products in the marketplace (spatial aspect), along with change in demand over time for each product (temporal aspect). The core idea is to estimate the counterfactual outcome using a synthetic control unaffected by the treatment. Our approach uses RGCN-dilated CNN based network, which leverages domain knowledge to automatically design a synthetic control during training. Using GCF for our demand forecasting problem, we achieve 75.3% lower MAPE compared to baseline. We use the forecasted values to recommend high demand products, in terms of our business metric (discussed later) which tracks the quality of these recommendations, we achieve a significant jump of 61.2%. Moreover, it adds 67.8% more high demand products to the marketplace, compared to existing model in production. Deployment of GCF in 2023, led to +1399 bps improvement in number of products with a view from customers, and +310 bps improvement in number of products with a sale. We also compare GCF with state of the art forecasting methods on a semi-synthetic data, created by simulating a treatment on open source traffic data METR-LA. We achieve 30% lower MSE against TGCN, a time series forecasting approach and 30% lower MSE against CRN and 25% lower MSE against Google Causal Impact model, both of which are causal forecasting approaches.

Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Suranjana Samanta, Prateeti Mohapatra, Debanjana Kar, Ruchi Mahindru, Steve Barbier, Eugen Postea, Brad Blancett 等

In the rapidly evolving landscape of site reliability engineering (SRE), the demand for efficient and effective solutions to manage and resolve issues in site and cloud applications is paramount. This paper presents an innovative approach to action automation using large language models (LLMs) for script generation, assessment, and refinement. By leveraging the capabilities of LLMs, we aim to significantly reduce the human effort involved in writing and debugging scripts, thereby enhancing the productivity of SRE teams. Our experiments focus on Bash scripts, a commonly used tool in SRE, and involve the CodeSift dataset of 100 tasks and the InterCode dataset of 153 tasks. The results show that LLMs can automatically assess and refine scripts efficiently, reducing the need for script validation in an execution environment. Results demonstrate that the framework shows an overall improvement of 7-10% in script generation.

Gaowei Zhang, Wei Wang, Yusheng Peng, Yi Wang

Cotton is a critical agricultural product and industrial raw material, playing a key role in the national economies and people's living conditions, particularly in developing countries. However, cotton picking and processing often result in the contamination with various foreign fibers, such as hair, hemp rope, plastic film, and polypropylene rope. These contaminants are difficult to remove during textile processing and tend to break into small fragments, significantly reducing the quality of cotton products and negatively impacting the cotton industry. In this paper, we present an AI-enabled hardware-software integrated system--XCotton, for identifying and removing foreign fibers. Our system has been deployed in actual cotton production environments in the multiple regions in China, Central Asia, and Africa. XCotton achieves a cleaning efficiency of 1000kg/h, representing a 43% improvement, with only 14 kWh energy consumption (63% less). Moreover, XCotton brings significant business values to its manufacturer and clients. XCotton not only enhances the quality of cotton products but also contributes to the value-adding and upgrading of the cotton industry in developing regions, supporting economic growth and improving living conditions.

Qi Ye, Guangya Yu, Jingping Liu, Erzhen Chen, Chenjie Dong, Xiaosheng Lin, Zelei Liu, Han Yu, Tong Ruan

Medical quality control (MQC) indicators are essential for evaluating the performance of healthcare institutions to ensure high-quality patient care. In this paper, we report the design, implementation, and deployment of the Intelligent EMR-LLM platform for Medical Quality Control (IMQC), a large language model (LLM)-empowered system for automatically computing MQC indicators for enhancing the quality of medical services in Shanghai. It consists of an LLM (i.e., EMR-LLM) for processing electronic medical records (EMRs). With EMR-LLM, IMQC translates existing MQC indicators into a standardized representation language and automatically computes them based on EMRs. Since its deployment in February 2024, IMQC has been adopted by the Shanghai Medical Quality Management Center and associated hospitals. So far, it has processed 1,245 medical quality indicators for secondary- and tertiary-level hospitals, achieving an MQC evaluation accuracy of 93.31%, which is comparable to human experts. It has significantly improved efficiency, increasing from 10 EMRs per hour per human expert to over 1,000 EMRs per hour on average using one single H800 GPU. Over the first round of deployment in Shanghai, it is estimated that IMQC saves around 3.42 million RMB per month in manpower costs compared to traditional reporting methods. The successful deployment of IMQC sets a precedence for other regions to adopt similar AI-driven solutions to enhance medical quality control.