论文检索

Siddhant Kharbanda, Devaansh Gupta, Erik Schultheis, Atmadeep Banerjee, Cho-Jui Hsieh, Rohit Babbar

Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices. Recent works in this domain have increasingly focused on a symmetric problem setting where both input instances and label features are short-text in nature. Short-text XMC with label features has found numerous applications in areas such as query-to-ad-phrase matching in search ads, title-based product recommendation, prediction of related searches. In this paper, we propose Gandalf, a novel approach which makes use of a label co-occurrence graph to leverage label features as additional data points to supplement the training distribution. By exploiting the characteristics of the short-text XMC problem, it leverages the label features to construct valid training instances, and uses the label graph for generating the corresponding soft-label targets, hence effectively capturing the label-label correlations. Surprisingly, models trained on these new training instances, although being less than half of the original dataset, can outperform models trained on the original dataset, particularly on the PSP@k metric for tail labels. With this insight, we aim to train existing XMC algorithms on both, the original and new training instances, leading to an average 5% relative improvements for 6 state-of-the-art algorithms across 4 benchmark datasets consisting of up to 1.3M labels. Gandalf can be applied in a plug-and-play manner to various methods and thus forwards the state-of-the-art in the domain, without incurring any additional computational overheads. Code has been open-sourced at www.github.com/xmc-aalto/InceptionXML.

Amin Karimi Monsefi, Payam Karisani, Mengxi Zhou, Stacey Choi, Nathan Doble, Heng Ji 0001, Srinivasan Parthasarathy 0001, Rajiv Ramnath

Standard modern machine-learning-based imaging methods have faced challenges in medical applications due to the high cost of dataset construction and, thereby, the limited labeled training data available. Additionally, upon deployment, these methods are usually used to process a large volume of data on a daily basis, imposing a high maintenance cost on medical facilities. In this paper, we introduce a new neural network architecture, termed LoGoNet, with a tailored self-supervised learning (SSL) method to mitigate such challenges. LoGoNet integrates a novel feature extractor within a U-shaped architecture, leveraging Large Kernel Attention (LKA) and a dual encoding strategy to capture both long-range and short-range feature dependencies adeptly. This is in contrast to existing methods that rely on increasing network capacity to enhance feature extraction. This combination of novel techniques in our model is especially beneficial in medical image segmentation, given the difficulty of learning intricate and often irregular body organ shapes, such as the spleen. Complementary, we propose a novel SSL method tailored for 3D images to compensate for the lack of large labeled datasets. Our method combines masking and contrastive learning techniques within a multi-task learning framework and is compatible with both Vision Transformer (ViT) and CNN-based models. We demonstrate the efficacy of our methods in numerous tasks across two standard datasets (i.e., BTCV and MSD). Benchmark comparisons with eight state-of-the-art models highlight LoGoNet's superior performance in both inference time and accuracy. Code available at: https://github.com/aminK8/Masked-LoGoNet.

Rasul Kairgeldin, Miguel Á. Carreira-Perpiñán

Univariate decision trees, commonly used since the 1950s, predict by asking questions about a single feature in each decision node. While they are interpretable, they often lack competitive predictive accuracy due to their inability to model feature correlations. Multivariate (oblique) trees use multiple features in each node, capturing high-dimensional correlations better, but sometimes they can be difficult to interpret. We advocate for a model that strikes a useful middle ground: bivariate decision trees, which use two features in each node. This typically produces trees that not only are more accurate than univariate trees, but much smaller, which offsets the small increase in node complexity and keeps them interpretable. They also help data mining by constructing new features that are useful for discrimination, and by providing a form of supervised, hierarchical 2D visualization that reveals patterns such as clusters or linear structure. We give two new algorithms to learn bivariate trees: a fast one based on CART; and a slower one based on alternating optimization with a feature regularization term, which produces the best trees while still scaling to large datasets.

Jack Julian, Yun Sing Koh, Albert Bifet

Continual learning closely emulates human learning, which allows a model to learn from a stream of tasks sequentially without forgetting previously learned knowledge. Replay-based continual learning methods mitigate forgetting and improve performance by reintroducing data belonging to old tasks, however a replay method's performance may deteriorate when the reintroduced data does not effectively represent all experienced data. To address this concern, we propose the Sketch-based Replay Projection (SRP) method to capture and retain the original data stream's distribution within stored memory. SRP augments existing replay frameworks and introduces a two-fold approach. First, we develop a sketch-based sample selection technique to approximate feature distributions within distinct tasks, thereby capturing a wide distribution of examples for subsequent replay. Second, we propose a data compression method which projects examples into a reduced-dimensional space while preserving inter-example relationships and emphasizing inter-class disparities, encouraging diverse representations of each class while maintaining memory requirements similar to existing replay methodologies. Our experimental results demonstrate that SRP enhances replay diversity and improves the performance of existing replay models.

Xing Jin 0002, Zhihai Gong, Jiuchuan Jiang, Chao Wang, Jian Zhang 0023, Zhen Wang 0013

Sybil attacks are a prevalent concern within the realm of crowdsourcing, underscoring the significance of quality control in this domain. Truth discovery has been extensively studied to deduce the most trustworthy information from conflicting data based on the principle that reliable workers yield reliable answers. However, existing truth discovery approaches overlook the metric of workers' reputations, e.g., workers' historical approval rates on crowdsourcing platforms, despite being inflated and noisy, they offer a rough indication of workers' ability. In this paper, we first refine the approval rate using Wilson Lower Bound to enhance its confidence, and then mitigate its noise and inflation through a method based on ranking similarity. Specifically, we propose a method called RCTD (Reputation-Constrained Truth Discovery), which introduces a similarity metric between the rankings of workers' weights and the refined approval rates. This metric serves as a penalizing factor in the objective function of the truth discovery, restricting workers' weights to avoid excessively deviating from their historical reputation during the weight estimation process. We solve the objective function by introducing the block coordinate descent coupled with heuristics approach method. Experimental results on real-world datasets demonstrate that our approach achieves more accurate inference of true results in the Sybil attack environment compared to the state-of-the-art methods.

Wenyuan Jiang, Wenwei Wu, Le Zhang 0010, Zixuan Yuan, Jian Xiang, Jingbo Zhou 0003, Hui Xiong 0001

In recent years, Graph Neural Networks (GNNs) and Large Language Models (LLMs) have exhibited remarkable capability in addressing different graph learning and natural language tasks, respectively. Motivated by this, integrating LLMs with GNNs has been increasingly studied to acquire transferable knowledge across modalities, which leads to improved empirical performance in language and graph domains. However, existing studies mainly focused on a single-domain scenario by designing complicated integration techniques to manage multimodal data effectively. Therefore, a concise and generic learning framework for multi-domain tasks, i.e., graph and language domains, is highly desired yet remains under-exploited due to two major challenges. First, the language corpus of downstream tasks differs significantly from graph data, making it hard to bridge the knowledge gap between modalities. Second, not all knowledge demonstrates immediate benefits for downstream tasks, potentially introducing disruptive noise to context-sensitive models like LLMs. To tackle these challenges, we propose a novel plug-and-play framework for incorporating a lightweight cross-domain prompting method into both language and graph learning tasks. Specifically, we first convert the textual input into a domain-scalable prompt, which not only preserves the semantic and logical contents of the textual input, but also highlights related graph information as external knowledge for different domains. Then, we develop a reinforcement learning-based method to learn the optimal edge selection strategy for useful knowledge extraction, which profoundly sharpens the multi-domain model capabilities. In addition, we introduce a joint multi-view optimization module to regularize agent-level collaborative learning across two domains. Finally, extensive empirical justifications over 23 public and synthetic datasets demonstrate that our approach can be applied to diverse multi-domain tasks more accurately, robustly, and reasonably, and improve the performances of the state-of-the-art graph and language models in different learning paradigms.

Shen Jiang, Guanghui Zhu, Yue Wang 0084, Chunfeng Yuan, Yihua Huang 0001

Multi-task learning (MTL), which aims to make full use of knowledge contained in multiple tasks to enhance overall performance and efficiency, has been broadly applied in recommendations. The main challenge for MTL models is negative transfer. Existing MTL models, mainly built on the Mixture-of-Experts (MoE) structure, seek enhancements in performance through feature selection and specific expert sharing mode design. However, one expert sharing mode may not be universally applicable due to the complex correlations and diverse demands among various tasks. Additionally, homogeneous expert architectures in such models further limit their performance. To address these issues, in this paper, we propose an innovative automatic MTL framework, AutoMTL, leveraging neural architecture search (NAS) to design optimal expert architectures and sharing modes. The Dual-level Expert Sharing mode and Architecture Navigator (DESAN) search space of AutoMTL can not only efficiently explore expert sharing modes and feature selection schemes but also focus on the architectures of expert subnetworks. Along with this, we introduce an efficient Progressively Discretizing Differentiable Architecture Search (PD-DARTS) algorithm for search space exploration. Extensive experiments demonstrate that AutoMTL can consistently outperform state-of-the-art, human-crafted MTL models. Moreover, the insights obtained from the discovered architectures provide valuable guidance for building new multi-task recommendation models.

Ziyu Jia, Haichao Wang, Yucheng Liu, Tianzi Jiang

Sleep stage classification has important clinical significance for the diagnosis of sleep-related diseases. To pursue more accurate sleep stage classification, multi-channel sleep signals are widely used due to the rich spatial-temporal information contained. However, it leads to a great increment in the size and computational costs, which constrain the application of multi-channel sleep models on hardware devices. Knowledge distillation is an effective way to compress models, yet existing knowledge distillation methods cannot fully extract and transfer the spatial-temporal knowledge in the multi-channel sleep signals. To solve the problem, we propose a general knowledge distillation framework for multi-channel sleep stage classification called spatial-temporal mutual distillation. Based on the spatial relationship of human body and the temporal transition rules of sleep signals, the spatial and temporal modules are designed to extract the spatial-temporal knowledge, thus help the lightweight student model learn the rich spatial-temporal knowledge from large-scale teacher model. The mutual distillation framework transfers the spatial-temporal knowledge mutually. Teacher model and student model can learn from each other, further improving the student model. The results on the ISRUC-III and MASS-SS3 datasets show that our proposed framework compresses the sleep models effectively with minimal performance loss and achieves the state-of-the-art performance compared to the baseline methods.

Jiahao Jiang, Yuheng Jia, Hui Liu 0032, Junhui Hou

This paper studies the semi-supervised partial label learning (SSPLL) problem, which aims to improve the partial label learning (PLL) by leveraging unlabeled samples. Both the existing SSPLL methods and the semi-supervised learning methods exploit the information in unlabeled samples by selecting high-confidence unlabeled samples as the pseudo labels based on the maximum value of the model output. However, the scarcity of labeled samples and the ambiguity from partial labels skew this strategy towards an unfair selection of high-confidence samples on each class, most notably during the initial phases of training, resulting in slower training and performance degradation. In this paper, we propose a novel method FairMatch, which adopts a learning state aware self-adaptive threshold for selecting the same number of high-confidence samples on each class, and uses augmentation consistency to incorporate the unlabeled samples to promote PLL. In addition, we adopt the candidate label disambiguation to utilize the partial labeled samples and mix up the partial labeled samples and the selected high-confidence unlabeled samples to prevent the model from overfitting on partial label samples. FairMatch can achieve maximum accuracy improvements of 9.53%, 4.9%, and 16.45% on CIFAR-10, CIFAR-100, and CIFAR-100H, respectively. The codes can be found at https://github.com/jhjiangSEU/FairMatch.

Shuo Ji 0001, Mingzhe Liu 0002, Leilei Sun, Chuanren Liu, Tongyu Zhu

Dynamic graph learning has attracted much attention in recent years due to the fact that most of the real-world graphs are dynamic and evolutionary. As a result, many dynamic learning methods have been proposed to cope with the changes of node states over time. Among these studies, a critical issue is how to update the representations of nodes when new temporal events are observed. In this paper, we provide a novel memory structure - Memory Map (MemMap) for this problem. MemMap is an adaptive and evolutionary latent memory space, where each cell corresponds to an evolving "topic" of the dynamic graph. Moreover, the representation of a node is generated from its semantically correlated memory cells, rather than linked neighbors of the node. We have conducted experiments on real-world datasets and compared our method with the SOTA ones. It can be concluded that: 1) By constructing an adaptive and evolving memory structure during the dynamic learning process, our method can capture the dynamic graph changes, and the learned MemMap is actually a compact evolving structure organized according to the latent "topics" of the graph nodes. 2) Our research suggests that it is a more effective and efficient way to generate node representations from a latent semantic space (like MemMap in our method) than from directly connected neighbors (like most of the previous graph learning methods). The reason is that the number of memory cells in latent space could be much smaller than the number of nodes in a real-world graph, and the representation learning process could well balance the global and local message passing by leveraging the semantic similarity of graph nodes via the correlated memory cells.

Jintian Ji, Songhe Feng, Yidong Li

The Unaligned Multi-view Clustering (UMC) problem is currently receiving widespread attention, focusing on clustering unaligned multi-view data generated in real-world applications. Although some algorithms have emerged to address this issue, there still exist the following drawbacks: 1) The fully unknown correspondence of samples across views can significantly limit the exploration of consistent clustering structure. 2) The fixed representation space makes it difficult to mine the comprehensive information in the original data. 3) Unbiased tensor rank approximation is desired to capture the high-order correlation among different views. To address these issues, we proposed a novel UMC framework termed Tensorized Unaligned Multi-view Clustering with Multi-scale Representation Learning (TUMCR). Specifically, TUMCR designs a multi-scale representation learning and alignment framework, which constructs multi-scale representation spaces to comprehensively explore the unknown correspondence across views. Then, a tensorial multi-scale fusion module is proposed to fuse multi-scale representations and explore the high-order correlation hidden in different views, which utilizes the Enhanced Tensor Rank (ETR) to learn the low-rank structure. Furthermore, TUMCR is solved by an efficient algorithm with good convergence. Extensive experiments on different types of datasets demonstrate the effectiveness and superiority of our TUMCR compared with state-of-the-art methods. Our code is publicly available at: https://github.com/jijintian/TUMCR.

Sheo Yon Jhin, Seojin Kim, Noseong Park

Time series forecasting has been an essential field in many different application areas, including economic analysis, meteorology, and so forth. The majority of time series forecasting models are trained using the mean squared error (MSE). However, this training based on MSE causes a limitation known as prediction delay. The prediction delay, which implies the ground-truth precedes the prediction, can cause serious problems in a variety of fields, e.g., finance and weather forecasting --- as a matter of fact, predictions succeeding ground-truth observations are not practically meaningful although their MSEs can be low. This paper proposes a new perspective on traditional time series forecasting tasks and introduces a new solution to mitigate the prediction delay. We introduce a continuous-time gated recurrent unit (GRU) based on the neural ordinary differential equation (NODE) which can supervise explicit time-derivatives. We generalize the GRU architecture in a continuous-time manner and minimize the prediction delay through our time-derivative regularization. Our method outperforms in metrics such as MSE, Dynamic Time Warping (DTW) and Time Distortion Index (TDI). In addition, we demonstrate the low prediction delay of our method in a variety of datasets.

Olivier Jeunen, Ivan Potapov, Aleksei Ustimenko

Approaches to recommendation are typically evaluated in one of two ways: (1) via a (simulated) online experiment, often seen as the gold standard, or (2) via some offline evaluation procedure, where the goal is to approximate the outcome of an online experiment. Several offline evaluation metrics have been adopted in the literature, inspired by ranking metrics prevalent in the field of Information Retrieval. (Normalised) Discounted Cumulative Gain (nDCG) is one such metric that has seen widespread adoption in empirical studies, and higher (n)DCG values have been used to present new methods as the state-of-the-art in top-n recommendation for many years. Our work takes a critical look at this approach, and investigates when we can expect such metrics to approximate the gold standard outcome of an online experiment. We formally present the assumptions that are necessary to consider DCG an unbiased estimator of online reward and provide a derivation for this metric from first principles, highlighting where we deviate from its traditional uses in IR. Importantly, we show that normalising the metric renders it inconsistent, in that even when DCG is unbiased, ranking competing methods by their normalised DCG can invert their relative order. Through a correlation analysis between off- and on-line experiments conducted on a large-scale recommendation platform, we show that our unbiased DCG estimates strongly correlate with online reward, even when some of the metric's inherent assumptions are violated. This statement no longer holds for its normalised variant, suggesting that nDCG's practical utility may be limited.

Jihyeong Jeon, Jiwon Park, Chanhee Park, U Kang

How can we leverage inherent frequency features of stock signals for effective portfolio optimization? Portfolio optimization in the domain of finance revolves around strategically allocating assets to maximize returns. Recent advancements highlight the efficacy of deep learning and reinforcement learning (RL) in capturing temporal asset patterns for portfolio optimization. However, previous methodologies focusing on time-domain often fail to detect sudden market shifts and abrupt events because their models are overly tailored to prevalent patterns, resulting in significant losses. In this paper, we propose FreQuant (Adaptive Portfolio Optimization via Multi-Frequency Quantitative Analysis), an effective deep RL framework for portfolio optimization that fully operates in the frequency domain, tackling the limitations of time domain-focused models. By bringing the analysis into the frequency domain with the Discrete Fourier Transform, our framework captures both prominent and subtle market frequencies, enhancing its adaptability and stability in response to market shifts. This approach allows FreQuant to adeptly identify primary asset patterns while also effectively responding to less common and abrupt market events, providing a more accurate and comprehensive asset representation. Empirical validation on diverse real-world trading datasets underscores the remarkable performance of FreQuant, showing its superiority in terms of profitability. Notably, FreQuant achieves up to 2.1x higher Annualized Rate of Return and 2.9x higher Portfolio Value than the best-performing competitors.

Md Mouinul Islam, Soroush Vahidi, Baruch Schieber, Senjuti Basu Roy

We investigate the problem of finding winner(s) given a large number of users' (voters') preferences casted as ballots, one from each of the m users, where each ballot is a ranked order of preference of up to ℓ out of n items (candidates). Given a group protected attribute with k different values and a priority that imposes a selection order among these groups, the goal is to satisfy the priority order and select a winner per group that is most representative. It is imperative that at times the original users' preferences may require further manipulation to meet these fairness and priority requirement. We consider manipulation by modifications and formalize the margin finding problem under modification problem. We study the suitability of Instant Run-off Voting (IRV) as a preference aggregation method and demonstrate its advantages over positional methods. We present a suite of technical results on the hardness of the problem, design algorithms with theoretical guarantees and further investigate efficiency opportunities. We present exhaustive experimental evaluations using multiple applications and large-scale datasets to demonstrate the effectiveness of IRV, and efficacy of our designed solutions qualitatively and scalability-wise.

Makoto Imamura, Takaaki Nakamura

Over the past two decades, time series motif discovery has become a crucial subroutine for many time series data mining tasks; concurrently, it has been established that Dynamic Time Warping (DTW) outperforms other similarity measures like Euclidean Distance in most scenarios. Against this backdrop, a DTW motif discovery algorithm was recently developed; however, it is confined to working with fixed-length subsequences. In this work, we propose a novel approach that allows us to find motifs under both length differences and warping. Our algorithm exploits a promising time series representation called Spikelets and introduces the first lower bound for DTW in the Spikelet space. Extensive empirical studies demonstrate that our method scales effectively across various real-world datasets and efficiently identifies DTW motif pairs of different lengths.

Bulat Ibragimov, Anton Vakhrushev

The Gradient Boosting machine learning ensemble algorithm, well-known for its proficiency and superior performance in intricate machine learning tasks, has encountered limited success in the realm of uplift modeling. Uplift modeling is a challenging task that necessitates a known target for the precise computation of the training gradient. The prevailing two-model strategies, which separately model treatment and control outcomes, are encumbered with limitations as they fail to directly tackle the uplift problem. This paper presents an innovative approach to uplift modeling that employs Gradient Boosting. Unlike previous works, our algorithm utilizes multioutput boosting model and calculates the uplift gradient based on intermediate surrogate predictions and directly models the concealed target. This method circumvents the requirement for a known target and addresses the uplift problem more effectively than existing solutions. Moreover, we broaden the scope of this solution to encompass multitreatment settings, thereby enhancing its applicability. This novel approach not only overcomes the limitations of the traditional two-model strategies but also paves the way for more effective and efficient uplift modeling using Gradient Boosting.

Bulat Ibragimov, Gleb Gusev

Gradient Boosting is a leading learning method that builds ensembles and adapts their sizes to particular tasks, consistently delivering top-tier results across various applications. However, determining the optimal number of models in the ensemble remains a critical yet underexplored aspect. Traditional approaches assume a universal ensemble size effective for all data points, which may not always hold true due to data heterogeneity. This paper introduces an adaptive approach to early stopping in Gradient Boosting, addressing data heterogeneity by assigning different stop moments to different data regions at inference time while still training a common ensemble on the entire dataset. We propose two methods: Direct Supervised Partition (DSP) and Indirect Supervised Partition (ISP). The DSP method uses a decision tree to partition the data based on learning curves, while ISP leverages the dataset's geometric and target distribution characteristics. An effective validation protocol is developed to determine the optimal number of early stopping regions or detect when the heterogeneity assumption does not hold. Experiments using state-of-the-art implementations of Gradient Boosting, LightGBM, and CatBoost, on standard benchmarks demonstrate that our methods enhance model precision by up to 2%, underscoring the significance of this research direction. This approach does not increase computational complexity and can be easily integrated into existing learning pipelines.

Seonghyeon Hwang, Minsu Kim 0006, Steven Euijong Whang

We study the problem of robust data augmentation for regression tasks in the presence of noisy data. Data augmentation is essential for generalizing deep learning models, but most of the techniques like the popular Mixup are primarily designed for classification tasks on image data. Recently, there are also Mixup techniques that are specialized to regression tasks like C-Mixup. In comparison to Mixup, which takes linear interpolations of pairs of samples, C-Mixup is more selective in which samples to mix based on their label distances for better regression performance. However, C-Mixup does not distinguish noisy versus clean samples, which can be problematic when mixing and lead to suboptimal model performance. At the same time, robust training has been heavily studied where the goal is to train accurate models against noisy data through multiple rounds of model training. We thus propose our data augmentation strategy RC-Mixup, which tightly integrates C-Mixup with multi-round robust training methods for a synergistic effect. In particular, C-Mixup improves robust training in identifying clean data, while robust training provides cleaner data to C-Mixup for it to perform better. A key advantage of RC-Mixup is that it is data-centric where the robust model training algorithm itself does not need to be modified, but can simply benefit from data mixing. We show in our experiments that RC-Mixup significantly outperforms C-Mixup and robust training baselines on noisy data benchmarks and can be integrated with various robust training methods.

Yihong Huang 0001, Yuang Zhang, Liping Wang 0012, Fan Zhang 0036, Xuemin Lin 0001

Unsupervised Outlier Detection (UOD) is an important data mining task. With the advance of deep learning, deep Outlier Detection (OD) has received broad interest. Most deep UOD models are trained exclusively on clean datasets to learn the distribution of the normal data, which requires huge manual efforts to clean the real-world data if possible. Instead of relying on clean datasets, some approaches directly train and detect on unlabeled contaminated datasets, leading to the need for methods that are robust to such challenging conditions. Ensemble methods emerged as a superior solution to enhance model robustness against contaminated training sets. However, the training time is greatly increased by the ensemble mechanism. In this study, we investigate the impact of outliers on training, aiming to halt training on unlabeled contaminated datasets before performance degradation. Initially, we noted that blending normal and anomalous data causes AUC fluctuations-a label-dependent measure of detection accuracy. To circumvent the need for labels, we propose a zero-label entropy metric named Loss Entropy for loss distribution, enabling us to infer optimal stopping points for training without labels. Meanwhile, a negative correlation between entropy metric and the label-based AUC score is demonstrated by theoretical proofs. Based on this, an automated early-stopping algorithm called EntropyStop is designed to halt training when loss entropy suggests the maximum model detection capability. We conduct extensive experiments on ADBench (including 47 real datasets), and the overall results indicate that AutoEncoder (AE) enhanced by our approach not only achieves better performance than ensemble AEs but also requires under 2% of training time. Lastly, loss entropy and EntropyStop are evaluated on other deep OD models, exhibiting their broad potential applicability.