论文检索

Sujoy Nath, Arkaprabha Basu, Kushal Bose, Swagatam Das

In the rapidly advancing field of AI-assisted medical diagnosis, the generation of medical reports for Chest X-rays (CXR) has significantly improved with the increased availability of radiographs and their corresponding reports. However, these reports often contain complex medical terminology, making them difficult for patients and non-healthcare professionals to understand. In this study, we introduce a strategy called Chained Prompting for Improved Readability of Medical Reports (CPIR-MR), which translates original medical reports into more comprehensible language. Our primary contribution is the creation of a new extension to the IU X-Ray dataset, providing Simplified Medical Reports (SMRs) generated by CPIR-MR. Additionally, we demonstrate that standard methodologies can effectively produce these simplified reports by proposing a multi-modal text decoder (MTD) that combines BLIP with a classification network to generate simplified medical explanations (SMEs) when fine-tuned on SMRs.

Tanjim Bin Faruk, Abdul Matin, Shrideep Pallickara, Sangmi Lee Pallickara

Masked autoencoders employ random masking to effectively reconstruct input images using self-supervised techniques, which allows for efficient training on large datasets. However, the random masking strategy does not adequately tap into information encapsulated within high-dimensional hyperspectral satellite imagery that is used in several domains. We propose a novel masking strategy, HOGMAE, based on the Histogram of Oriented Gradients that incorporates rich information inherent within satellite images during the mask creation step. Our experiments, over a hyperspectral satellite dataset, demonstrate the effectiveness of our methodology.

Haofeng Zhang

While advances in machine learning and the expansion of massive datasets have significantly improved predictive accuracy, the translation of these predictions into actionable decisions—alongside a robust understanding of associated risks—remains underexplored. My research focuses on developing methodology and theory in data-driven decision-making and uncertainty quantification that effectively address core data challenges. This paper presents two connected pillars of my research: data-driven contextual optimization, uncertainty quantification and reduction.

Kunpeng Xu

Time sequences are essential in fields such as finance, healthcare, and environmental science, where understanding temporal dependencies and making accurate predictions are crucial. These sequences often exhibit complexities like nonlinearity, noise, and concept drift. Traditional models struggle to capture the intricate dynamics of multivariate and co-evolving sequences, particularly in contexts where relationships between variables shift unpredictably. This thesis introduces a range of Kernel Representation Learning (KRL) methodologies to address these challenges. We develop kernel self-representation learning to capture the temporal dependencies and hidden structures, while identifying concept drift in co-evolving sequences. Additionally, we explore theoretical connections between KRL and advanced deep-learning models. The proposed methods are validated through real-world applications, showing improvements in predictive accuracy, interpretability, and robustness.

Yuxing Lu

The advancements in Knowledge Graphs (KGs) and Large Language Models (LLMs) are driving transformative changes across various research fields, including metabolomics. These tools present exceptional opportunities to elucidate complex metabolic pathways and identify biomarkers essential to biological systems. My research focuses on harnessing the potential of KGs and LLMs within metabolomics, specifically making interactions between them and with biological researches. KGs, with their structured representation of metabolic entities and relationships, provide a robust foundation for managing extensive multimodal metabolomic knowledge. Recently, I developed a metabolite-centric knowledge graph and explored innovative methodologies to leverage KGs and LLMs for enhancing predictive modeling in clinical settings. My future research aims to fully exploit the capabilities of KGs and LLMs in metabolomics, advancing our understanding and applications in this field.

Mahammed Kamruzzaman

The rise of large language models (LLMs) has revolutionized natural language processing, offering immense capabilities across various applications. The widespread integration of these models into commonplace technology has brought to light deep concerns about the biases they encompass, which could serve to perpetuate negative preconceptions and social injustices. The scope of my research includes social biases, brand biases, the impact of personas on bias, and stereotypes in low-resource languages. My contributions aim to deepen our understanding of these biases and develop methodologies to mitigate them, enhancing the fairness and utility of LLMs across diverse global applications.

Akash V. Maharaj, David Arbour, Daniel Lee, Uttaran Bhattacharya, Anup Rao, Austin Zane, Avi Feller, Kun Qian, Yunyao Li

Enterprise AI Assistants are increasingly deployed in domains where accuracy is paramount, making each erroneous output a potentially significant incident. This paper presents a comprehensive framework for monitoring, benchmarking, and continuously improving such complex, multi-component systems under active development by multiple teams. Our approach encompasses three key elements: (1) a hierarchical ``severity'' framework for incident detection that identifies and categorizes errors while attributing component-specific error rates, facilitating targeted improvements; (2) a scalable and principled methodology for benchmark construction, evaluation, and deployment, designed to accommodate multiple development teams, mitigate overfitting risks, and assess the downstream impact of system modifications; and (3) a continual improvement strategy leveraging multidimensional evaluation, enabling the identification and implementation of diverse enhancement opportunities. By adopting this holistic framework, organizations can systematically enhance the reliability and performance of their AI Assistants, ensuring their efficacy in critical enterprise environments. We conclude by discussing how this multifaceted evaluation approach opens avenues for various classes of enhancements, paving the way for more robust and trustworthy AI systems.

Daniele Meli, Alberto Castellini, Alessandro Farinelli

Partially Observable Markov Decision Processes (POMDPs) are a powerful framework for planning under uncertainty. They allow to model state uncertainty as a belief probability distribution. Approximate solvers based on Monte Carlo sampling show great success to relax the computational demand and perform online planning. However, scaling to complex realistic domains with many actions and long planning horizons is still a major challenge, and a key point to achieve good performance is guiding the action-selection process with domain-dependent policy heuristics which are tailored for the specific application domain. We propose to learn high-quality heuristics from POMDP traces of executions generated by any solver. We convert the belief-action pairs to a logical semantics, and exploit data- and time-efficient Inductive Logic Programming (ILP) to generate interpretable belief-based policy specifications, which are then used as online heuristics. We evaluate thoroughly our methodology on two notoriously challenging POMDP problems, involving large action spaces and long planning horizons, namely, rocksample and pocman. Considering different state-of-the-art online POMDP solvers, including POMCP, DESPOT and AdaOPS, we show that learned heuristics expressed in Answer Set Programming (ASP) yield performance superior to neural networks and similar to optimal handcrafted task-specific heuristics within lower computational time. Moreover, they well generalize to more challenging scenarios not experienced in the training phase (e.g., increasing rocks and grid size in rocksample, incrementing the size of the map and the aggressivity of ghosts in pocman).

Zheyuan Ryan Shi

AI for public sector research is about using AI to tackle the numerous challenges faced by public sector organizations when they are out there making our world a better place. AI for public sector research is use-inspired research. It differs from traditional AI research first and foremost in its key objective being measurable societal impact. AI for public sector research contributes to the computing community by proposing new problem models, raising complexities that challenge abstractions which often leads to new methodologies, and introducing new contexts for evaluation. However, fulfilling this promise is easier said than done. This talk consists of three parts about our preliminary work in our long-term quest to make AI for public sector operationally scalable, financially sustainable, technically generalizable, and socially responsible. We will cover (1) a concrete AI for public sector project from problem scoping to field trials and deployment, (2) a generalizable algorithm applicable to various public sector domains, and (3) an overview of our work in a wide variety of applications.

Linyi Li

Along with the broad deployment of deep learning (DL) systems, their lack of trustworthiness, such as their lack of robustness, fairness, and numerical reliability, is raising serious social concerns, especially in safety-critical scenarios such as autonomous driving and aircraft navigation. Hence, a rigorous and accurate evaluation of the trustworthiness of DL systems is essential and would be a prerequisite for improving DL trustworthiness. The first part of the talk will be an overview of certified methods for DL trustworthiness. These methods provide computable guarantees for DL systems in terms of worst-case trustworthiness under certain realistic conditions, such as the accuracy lower bound against arbitrary tiny perturbations. Based on our taxonomy and systematization, we illustrate key methodologies, specifically semantic randomized smoothing and branch-and-bound, and their implications for certified DL trustworthiness. As a representative of recent DL breakthroughs, large language models (LLMs) are transforming our lives, but, on the other hand, posing more challenges to trustworthiness. For example, LLMs can be jailbroken with adversarial prompts to output harmful content with bias, harassment, misinformation, and more. The second part of the talk will be an overview of LLM trustworthiness. We will start with sharing hands-on experience in developing fontier LLMs, then illustrate common LLM trustworthiness issues via examples, then demonstrate evaluation challenges, take one benchmark as an example, and conclude by envisioning certifiable trustworthiness for LLMs.

Segev Wasserkrug, Leonard Boussioux, Dick Den Hertog, Farzaneh Mirzazadeh, Ş. İlker Birbil, Jannis Kurtz, Donato Maragno

Many critical business and societal decisions in areas such as supply chain and healthcare involve numerous potential actions, complex constraints, and goals that can be modeled as objective functions. Mathematical optimization, a core area in Operations Research (OR), provides robust, mathematically grounded methodologies to address such decisions and has shown tremendous benefits in many applications. However, its application requires the creation of accurate and efficient optimization models, necessitating rare expertise and considerable time, creating a barrier to widespread adoption in decision-making. Thus, it is a long-standing goal to make these capabilities widely accessible. The advent of Large Language Models (LLMs) has made advanced Artificial Intelligence (AI) capabilities widely accessible through natural language. LLMs can accelerate expert work in creating formal models like computer programs, and emerging research indicates they can also speed up the development of optimization models by OR experts. We, therefore, propose integrating and advancing LLM and optimization modeling to empower organizational decision-makers to model and solve such complex problems without requiring deep expertise in optimization. In this work, we present our vision for democratizing optimization modeling for organizational decision-making by such a combination of LLMs and optimization modeling. We identify a set of fundamental requirements for the vision's implementation and describe the state of the art through a literature survey and some experimentation. We show that a) LLMs already provide substantial novel capabilities relevant to realizing this vision, but that b) major research challenges remain to be addressed. We also propose possible research directions to overcome these gaps. We would like this work to serve as a call to action to bring together the LLM and OR optimization modeling communities to pursue this vision, thereby enabling much more widespread improved decision-making and increasing by orders of magnitude the benefits AI and OR can bring to enterprises and society.

Runlong Yu, Chonghao Qiu, Robert Ladwig, Paul Hanson, Yiqun Xie, Xiaowei Jia

Physics-guided machine learning (PGML) has become a prevalent approach in studying scientific systems due to its ability to integrate scientific theories for enhancing machine learning (ML) models. However, most PGML approaches are tailored to isolated and relatively simple tasks, which limits their applicability to complex systems involving multiple interacting processes and numerous influencing features. In this paper, we propose a Physics-Guided Foundation Model (PGFM) that combines pre-trained ML models and physics-based models and leverages their complementary strengths to improve the modeling of multiple coupled processes. To effectively conduct pre-training, we construct a simulated environmental system that encompasses a wide range of influencing features and various simulated variables generated by physics-based models. The model is pre-trained in this system to adaptively select important feature interactions guided by multi-task objectives. We then fine-tune the model for each specific task using true observations, while maintaining consistency with established physical theories, such as the principles of mass and energy conservation. We demonstrate the effectiveness of this methodology in modeling water temperature and dissolved oxygen dynamics in real-world lakes. The proposed PGFM is also broadly applicable to a range of scientific fields where physics-based models are being used.

Joshua Strong, Qianhui Men, J. Alison Noble

Large language models (LLMs) offer a valuable technology for various applications in healthcare. However, their tendency to hallucinate and the existing reliance on proprietary systems pose challenges in environments concerning critical decision-making and strict data privacy regulations, such as healthcare, where the trust in such systems is paramount. Through combining the strengths and discounting the weaknesses of humans and AI, the field of Human-AI Collaboration (HAIC) presents one front for tackling these challenges and hence improving trust. This paper presents a novel HAIC \textit{guided deferral} system that can simultaneously parse medical reports for disorder classification, and defer uncertain predictions with intelligent guidance to humans. We develop methodology which builds efficient, effective and open-source LLMs for this purpose, for the real-world deployment in healthcare. We conduct a pilot study which showcases the effectiveness of our proposed system in practice. Additionally, we highlight drawbacks of standard calibration metrics in imbalanced data scenarios commonly found in healthcare, and suggest a simple yet effective solution: the Imbalanced Expected Calibration Error.

Marco Minici, Luca Luceri, Francesco Fabbri, Emilio Ferrara

Social media platforms have become vital spaces for public discourse, serving as modern agorás where a wide range of voices influence societal narratives. However, their open nature also makes them vulnerable to exploitation by malicious actors, including state-sponsored entities, who can conduct information operations (IOs) to manipulate public opinion. The spread of misinformation, false news, and misleading claims threatens democratic processes and societal cohesion, making it crucial to develop methods for the timely detection of inauthentic activity to protect the integrity of online discourse. In this work, we introduce a methodology designed to identify users orchestrating information operations, a.k.a. IO drivers, across various influence campaigns. Our framework, named IOHunter, leverages the combined strengths of Language Models and Graph Neural Networks to improve generalization in supervised, scarcely-supervised, and cross-IO contexts. Our approach achieves state-of-the-art performance across multiple sets of IOs originating from six countries, significantly surpassing existing approaches. This research marks a step toward developing Graph Foundation Models specifically tailored for the task of IO detection on social media platforms.

Hongzhao Guan, Nabeel Gillani, Tyler Simko, Jasmine Mangat, Pascal Van Hentenryck

Most US school districts draw geographic "attendance zones" to assign children to schools based on their home address, a process that can replicate existing neighborhood racial/ethnic and socioeconomic status (SES) segregation in schools. Redrawing boundaries can reduce segregation, but estimating expected rezoning impacts is often challenging because families can opt-out of their assigned schools. This paper seeks to alleviate this societal problem by developing a joint redistricting and choice modeling framework, called redistricting with choices (RWC). The RWC framework is applied to a large US public school district to estimate how redrawing elementary school boundaries might realistically impact levels of socioeconomic segregation. The main methodological contribution of RWC is a contextual stochastic optimization model that aims to minimize district-wide segregation by integrating rezoning constraints with a machine learning-based school choice model. The study finds that RWC yields boundary changes that might reduce segregation by a substantial amount (23%) -- but doing so might require the re-assignment of a large number of students, likely to mitigate re-segregation that choice patterns could exacerbate. The results also reveal that predicting school choice is a challenging machine learning problem. Overall, this study offers a novel practical framework that both academics and policymakers might use to foster more diverse and integrated schools.

Niclas Boehmer, Yash Nair, Sanket Shah, Lucas Janson, Aparna Taneja, Milind Tambe

In many applications of AI for Social Impact (e.g., when allocating spots in support programs for underserved communities), resources are scarce and an allocation policy is needed to decide who receives a resource. Before being deployed at scale, a rigorous evaluation of an AI-powered allocation policy is vital. In this paper, we introduce the methods necessary to evaluate index-based allocation policies, which allocate a limited number of resources to those who need them the most. Such policies create dependencies between agents, rendering standard statistical tests invalid and ineffective. Addressing the arising practical and technical challenges, we describe an efficient estimator and methods for drawing valid statistical conclusions. Our extensive experiments validate our methodology in practical settings while also showcasing its statistical power. We conclude by proposing and empirically verifying extensions of our methodology that enable us to reevaluate a past randomized control trial conducted with 10000 beneficiaries for a mHealth program for pregnant women. Our new methodology allows us to draw previously invisible conclusions when comparing two different ML allocation policies.

Nahian Ahmed, Mark Roth, Tyler A. Hallman, W. Douglas Robinson, Rebecca A. Hutchinson

Citizen science biodiversity data present great opportunities for ecology and conservation across vast spatial and temporal scales. However, the opportunistic nature of these data lacks the sampling structure required by modeling methodologies that address a pervasive challenge in ecological data collection: imperfect detection, i.e., the likelihood of under-observing species on field surveys. Occupancy modeling is an example of an approach that accounts for imperfect detection by explicitly modeling the observation process separately from the biological process of habitat selection. This produces species distribution models that speak to the pattern of the species on a landscape after accounting for imperfect detection in the data, rather than the pattern of species observations corrupted by errors. To achieve this benefit, occupancy models require multiple surveys of a site across which the site's status (i.e., occupied or not) is assumed constant. Since citizen science data are not collected under the required repeated-visit protocol, observations may be grouped into sites post hoc. Existing approaches for constructing sites discard some observations and/or consider only geographic distance and not environmental similarity. In this study, we compare ten approaches for site construction in terms of their impact on downstream species distribution models for 31 bird species in Oregon, using observations recorded in the eBird database. We find that occupancy models built on sites constructed by spatial clustering algorithms perform better than existing alternatives.

Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, Pengfei Liu

The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs validity and redundancy to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. We explore different design options for the LLM-based evaluators and empirically demonstrate that ReasonEval, when instantiated with base models possessing strong mathematical knowledge and trained with high-quality labeled data, consistently outperforms baseline methods in the meta-evaluation datasets. We also highlight the strong generalization capabilities of ReasonEval. By utilizing ReasonEval to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We open-source the best-performing model, meta-evaluation script, and all evaluation results to facilitate future research.

Kelsey Kraus, Margaret Kroll

The emergence of powerful LLMs has led to a paradigm shift in Natural Language Understanding and Natural Language Generation. But the properties that make LLMs so valuable for these tasks -- creativity, ability to produce fluent speech, and ability to quickly and effectively abstract information from large corpora -- also present new challenges to evaluating their outputs. The rush to market has led teams to fall back on quick, cost-effective automatic evaluations which offer value, but do not obviate the need for human judgments in model training and evaluation. We argue that when end users need to agree with the decisions made by ML models -- e.g. in toxicity detection or in extraction of main points for summarization -- models should be trained and tested on data that represent the preferences of those users. This paper primarily discusses the role of human feedback in labeling and judgment tasks for model training and evaluation. We first propose methods for disentangling noise from signal in labeling tasks. We show that noise in labeling disagreement can be minimized by adhering to proven methodological best practices, while signal in labeling disagreement can be maximized to play an integral role in model training and evaluation tasks. We illustrate best practices by providing a case study in which two guardrail classifiers are evaluated, using human judgments to align final model behavior to user preferences. We aim for this paper to provide researchers and professionals with guidelines to integrating human judgments into their ML and generative AI evaluation toolkit when working toward achieving accurate and unbiased features that align with users’ needs and expectations.

Maximilian Schäffeler, Mohammad Abdulaziz

We present a methodology based on interactive theorem proving that facilitates the development of verified implementations of algorithms for solving factored Markov Decision Processes. As a case study, we formally verify an algorithm for approximate policy iteration in the proof assistant Isabelle/HOL. We show how the verified algorithm can be refined to an executable, verified implementation. Our evaluation on benchmark problems shows that it is practical. As part of the development, we build verified software to certify linear programming solutions. We discuss the verification process and the modifications we made to the algorithm during formalization.