Omnipredictors for Regression and the Approximate Rank of Convex Functions
Consider the supervised learning setting where the goal is to learn to predict labels y given points x from a distribution. An omnipredictor...
Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
Despite the successes of large language models (LLMs), they exhibit significant drawbacks, particularly when processing long contexts. Their inference cost scales quadratically with...
How Smooth Is Attention?
Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz...
Optimization Without Retraction on the Random Generalized Stiefel Manifold
Optimization over the set of matrices X that satisfy X^TBX = Ip, referred to as the generalized Stiefel manifold, appears in many applications...
Careful With That Scalpel: Improving Gradient Surgery With an EMA
Beyond minimizing a single training loss, many deep learning estimation pipelines rely on an auxiliary objective to quantify and encourage desirable properties of...
Accurate Knowledge Distillation via N-best Reranking
We propose utilizing n-best reranking to enhance Sequence-Level Knowledge Distillation (Kim and Rush, 2016) where we extract pseudo-labels for student model’s training data...
Transfer Learning for Structured Pruning under Limited Task Data
This paper was accepted at the Efficient Natural Language and Speech Processing (ENLSP-III) Workshop at NeurIPS.
Large, pre-trained models are problematic to use in...
Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question...
Open Domain Question Answering (ODQA) within natural language processing involves building systems that answer factual questions using large-scale knowledge corpora. Recent advances stem...
Applying RLAIF for Code Generation with API-usage in Lightweight LLMs
This paper was accepted at the Natural Language Reasoning and Structured Explanations workshop at ACL 2024.
Reinforcement Learning from AI Feedback (RLAIF) has demonstrated...
Revisiting Non-separable Binary Classification and its Applications in Anomaly Detection
The inability to linearly classify XOR has motivated much of deep learning. We revisit this age-old problem and show that linear classification of...
