Long Document Classification Benchmark 2025: XGBoost vs BERT vs Traditional ML [Complete Guide]

What is Long Document Classification?

Long document classification is a specialized subfield of document classification within Natural Language Processing (NLP) that focuses on categorizing documents containing 1,000+ words (2+ pages), such as academic papers, legal contracts, and technical reports. Unlike short-text classification, long documents present challenges like input length constraints (e.g., 512 tokens in BERT), loss of contextual coherence when splitting the document, high computational cost, and the need for complex label structures like multi-label or hierarchical classification.

Executive Summary

This benchmark study evaluates various approaches for classifying lengthy documents (7,000-14,000 words ≈ 14-28 pages ≈ short to medium academic papers) across 11 academic categories. XGBoost [1] emerged as the most versatile solution, achieving F1-scores (balanced measure combining precision and recall) of 0.75-0.86 with reasonable computational requirements. Logistic Regression [2] provides the best efficiency-performance trade-off for resource-constrained environments, training in under 20 seconds with competitive accuracy. Surprisingly, RoBERTa-base [3] significantly underperformed despite its general reputation, while traditional machine learning approaches proved highly competitive against advanced transformer models.

Our experiments analyzed 27,000+ documents across four complexity categories from simple keyword matching to large language models, revealing that traditional ML methods often outperform sophisticated transformers while using 10x fewer computational resources. These counter-intuitive results challenge common assumptions about the necessity of complex models for long document classification tasks.

Quick Recommendations

• Best Overall: XGBoost (F1: 0.86, fast training)
• Most Efficient: Logistic Regression (trains in <20s)
• GPU Available: BERT-base [4] (F1: 0.82, but slower)
• Avoid: Keyword-based methods, RoBERTa-base

Study Methodology & Credibility

• Dataset Size: 27,000+ documents across 11 academic categories [Download]
• Hardware Specification: 15x  vCPUs, 45GB RAM, NVIDIA Tesla V100S 32GB
• Reproducibility: All code and configurations are available on GitHub

Key Research Findings (Verified Results)

• XGBoost achieved an 86% F1-score on 27,000 academic documents
• Traditional ML methods train 10x faster than transformer models
• BERT requires 2GB+ GPU memory vs 100MB RAM for XGBoost
• RoBERTa-base achieved just a 57% F1-score, falling short of expectations in low-data settings
• Training transformer-based models on the full dataset isn’t justifiable due to the extremely long training time (over 4 hours). Notably, as the data volume grows, model complexity increases and training time rises exponentially

How to Choose the Right Document Classification Method for Long Documents with a Small Number of Samples (~100 to 150 Samples)

Table of Contents

1. Introduction
2. Classification Methods: Simple to Complex
3. Technical Specifications
4. Results and Analysis
5. Deployment Scenarios
6. Frequently Asked Questions
7. Conclusion

1. Introduction

Long document classification is a specialized subfield of document classification within Natural Language Processing (NLP). At its core, document classification involves assigning one or more predefined categories or labels to a given document based on its content. This is a fundamental task for organizing, managing, and retrieving information efficiently in various domains, from legal and healthcare to news and customer reviews.

In long document classification, the term “long” refers to the substantial length of the documents involved. Unlike short texts—such as tweets, headlines, or single sentences—long documents can span multiple paragraphs, full articles, books, or even legal contracts. This extended length presents unique challenges that traditional text classification methods often struggle to address effectively.

Main Challenges in Long Document Classification

• Contextual Information: Long documents contain much richer and more complex context. Accurately understanding and classifying them requires processing information spread across multiple sentences and paragraphs, not just a few keywords.
• Computational Complexity: Many advanced NLP models, especially Transformer-based ones like BERT, have limits on the maximum input length they can handle efficiently. Their self-attention mechanisms, while powerful for capturing word relationships, become computationally expensive (O(N²) complexity – grows exponentially with document length) and memory-intensive when dealing with very long texts.
• Information Density and Sparsity: Although long documents contain a lot of information, the most important features for classification are often sparsely distributed. This makes it challenging for models to detect and focus on these key signals amidst large amounts of less relevant content.
• Maintaining Coherence: A common approach is to split long documents into smaller chunks. However, this can break the flow and context, making it harder for models to grasp the overall meaning and make accurate classifications.

Study Objectives

In this benchmark study, we evaluate various long document classification methods from a practical and software development standpoint. Our objective is to identify which approach best addresses the unique challenges of long document processing, based on the following criteria:

1. Efficiency: Models should be able to process long documents efficiently in terms of time and memory
2. Accuracy: Models should be able to accurately classify documents, even with long lengths
3. Robustness: Models should be robust to varying document lengths and different types of information organization

2. Classification Methods: Simple to Complex

This section presents four categories of classification methods, ranging from simple keyword matching to sophisticated language models. Each method represents different trade-offs between accuracy, speed, and implementation complexity.

2.1 Simple Methods (No Training Required)

These methods are quick to implement and perform well when the documents are relatively simple and not structurally complex. Typically rule-based, pattern-based, or keyword-based, they require no training time, which makes them especially robust to changes in the number of labels.

When to use: Known document structures, rapid prototyping, or when training data is unavailable.
Main advantage: Zero training time and high interpretability.
Main limitation: Poor performance on complex or nuanced classification tasks.

Keyword-Based Classification

The process begins by extracting a set of representative keywords for each category from the document set. During testing (or prediction), the classification follows these basic steps:

1. Tokenize the document
2. Count the number of keyword matches for each category
3. Assign the document to the category with the highest match count or keyword density

More advanced tools like YAKE (Yet Another Keyword Extractor) [5] can be used to automate keyword extraction. Additionally, if category names are known in advance, external keywords—those not found within the documents—can be added to the keyword sets with the help of intelligent models.

TF-IDF (Term Frequency-Inverse Document Frequency) + Similarity

While it uses TF-IDF vectors, it doesn’t require training a machine learning model. Instead, you select a few representative documents for each category—often just 2 or 3 examples per category are sufficient—and compute their TF-IDF vectors, which reflect the importance of each word within the document relative to the rest of the corpus.

Next, for each category, you calculate a mean TF-IDF vector to represent a typical document in that class. During testing, you convert the new document into a TF-IDF vector and compute its cosine similarity with each category’s mean vector. The category with the highest similarity score is selected as the predicted label.

This approach is particularly effective for long documents, as it takes into account the entire content rather than focusing on a limited set of keywords. It is also more robust than basic keyword matching and still avoids the need for supervised training.

Next steps: If simple methods meet your accuracy requirements, proceed with keyword extraction using YAKE or manual selection. Otherwise, consider intermediate methods for better performance.

Key Takeaway: Simple methods offer rapid implementation and zero training time but suffer from poor accuracy on complex classification tasks. Best suited for well-structured documents with clear keyword patterns.

2.2 Intermediate Methods (Traditional ML)

Having covered simple methods, we now examine intermediate approaches that require training but offer significantly better performance.

When to use: When you have labeled training data and need reliable, fast classification.
Main advantage: Excellent balance of accuracy, speed, and resource requirements.
Main limitation: Requires feature engineering and training data.

One of the most accessible and time-tested approaches for document classification — especially as a baseline — is the combination of TF-IDF vectorization with traditional machine learning classifiers such as Logistic Regression, Support Vector Machines (SVMs), or XGBoost. Despite its simplicity, this method continues to be a competitive option for many real-world applications, especially when interpretability, speed, and ease of deployment are prioritized.

Method Overview

The technique is straightforward: the document text is converted into a numerical representation using TF-IDF, which captures how important a word is relative to a corpus. This produces a sparse vector of weighted word counts.

The resulting vector is then passed into a classical classifier, typically:

• Logistic Regression for linear separability and fast training
• SVM for more complex boundaries
• XGBoost for high-performance, tree-based modeling

The model learns to associate word presence and frequency patterns with the desired output labels (e.g., topic categories or document types).

Handling Long Documents

By default, TF-IDF can process the entire document at once, making it suitable for long texts without the need for complex chunking or truncation. However, if documents are extremely lengthy (e.g., exceeding 5,000–10,000 words), it may be beneficial to:

1. Chunk the document into smaller segments (e.g., 1,000–2,000 words)
2. Classify each chunk individually
3. And then aggregate results using majority voting or average confidence scores

This chunking strategy can improve stability and mitigate sparse vector issues while still remaining computationally efficient.

Next steps: Start with Logistic Regression for baseline performance, then try XGBoost for optimal accuracy. Use 5-fold cross-validation with stratified sampling for robust evaluation.

Key Takeaway: Intermediate methods demonstrate the best balance of accuracy and efficiency. XGBoost consistently delivers top performance while Logistic Regression excels in resource-constrained environments.

2.3 Complex Methods (Transformer-Based)

Moving beyond traditional approaches, we explore transformer-based methods that leverage pre-trained language understanding.

When to use: When maximum accuracy is needed and GPU resources are available.
Main advantage: Deep language understanding and high accuracy potential.
Main limitation: Computational intensity and 512-token limit requiring chunking.

For many classification tasks involving moderately long documents — typically in the range of 300 to 1,500 words — fine-tuned transformer models like BERT, DistilBERT [6], and RoBERTa represent a highly effective and accessible middle-ground solution. These models strike a balance between traditional machine learning approaches and large-scale models like Longformer or GPT-4.

Architecture and Training

At their core, these models are pretrained language models that have learned general linguistic patterns from large corpora such as Wikipedia and BookCorpus. When fine-tuned for document classification, the architecture is extended by adding a simple classification head — usually a dense layer — on top of the transformer’s pooled output.

Fine-tuning involves training this extended model on a labeled dataset for a specific task, such as classifying reports into categories like Finance, Sustainability, or Legal. During training, the model adjusts both the classification head and (optionally) the internal transformer weights based on task-specific examples.

Handling Length Limitations

One key limitation of standard transformers like BERT and DistilBERT is that they only support sequences up to 512 tokens. For long documents, this constraint must be addressed by:

• Truncation: Simply cutting off the text after the first 512 tokens. Fast, but may ignore critical information later in the document.
• Chunking: Splitting the document into overlapping or sequential segments, classifying each chunk individually, and then aggregating predictions using majority vote, average confidence, or attention-based weighting.
• Preprocessing and data preparation: In this approach, long documents are first broken down into shorter texts (up to 512 tokens) using preprocessing techniques like keyword extraction or summarization. While these methods may sacrifice some coherence between segments, they offer faster training and classification times.

While chunking adds complexity, it enables these models to handle documents up to several thousand words while maintaining reasonable performance.

Next steps: Begin with DistilBERT for faster training, then upgrade to BERT-base if accuracy gains justify the computational cost. Implement overlapping chunking strategies for documents exceeding 512 tokens.

Key Takeaway: Transformer methods offer high accuracy but require significant computational resources. BERT-base performs well while RoBERTa-base surprisingly underperforms, highlighting the importance of empirical evaluation over reputation.

2.4 Most Complex Methods (Large Language Models)

Finally, we examine the most sophisticated approaches using large language models for instruction-based classification.

When to use: Zero-shot classification, extremely long documents, or when training data is limited.
Main advantage: No training required, handles very long contexts, high accuracy.
Main limitation: High API costs, slower inference, and requires internet connectivity.

These methods are powerful models that are able to understand complex documents with minimal or no training. They are suitable for tasks such as instruction-based or zero-shot classification.

API-Based Classification

OpenAI GPT-4 / Claude / Gemini 1.5: This approach leverages the instruction-following capability of models like GPT-4, Claude, and Gemini through API calls. These models can handle long-context inputs—up to 128,000 tokens in some cases (which is roughly equivalent to 300+ pages of text ≈ multiple academic papers).

The method is simple in concept: you provide the model with the document text (or a substantial chunk of it) along with a prompt like:

“You are a document classification assistant. Given the text below, classify the document into one of the following categories: [Finance, Legal, Sustainability].”

Once prompted, the LLM analyzes the document in real time and returns a label or even a confidence score, often with an explanation.

RAG-Enhanced Classification

LLMs Combined with RAG (Retrieval-Augmented Generation): Retrieval-Augmented Generation (RAG) is a more advanced architectural pattern that combines a vector-based retrieval system with an LLM. Here’s how it works in a classification setting:

• First, the long document is split into smaller, semantically meaningful chunks (e.g., by section, heading, or paragraph)
• Each chunk is embedded into a dense vector using an embedding model (like OpenAI’s text-embedding or SentenceTransformers)
• These vectors are stored in a vector database (like FAISS or Pinecone)
• When classification is needed, the system retrieves only the most relevant document chunks and passes them to an LLM (like GPT-4) along with a classification instruction

This method allows you to process long documents efficiently and scalably, while still benefiting from the power of large models.

Next steps: Start with simpler prompting strategies before implementing RAG. Consider cost-effectiveness compared to fine-tuned models for your specific use case.

Key Takeaway: LLM methods provide powerful zero-shot capabilities for long documents but come with high API costs and latency. Best suited for scenarios where training data is limited or extremely long context processing is required.

2.5 Model Comparison Summary

The following table provides a comprehensive overview of all classification methods, comparing their capabilities, resource requirements, and optimal use cases to help guide your selection process.

Key Insight: Traditional ML (XGBoost) often outperforms advanced transformers while using 10x fewer resources.

2.6 Referenced Datasets & Standards

The following datasets provide excellent benchmarks for testing long document classification methods:

Context: All datasets are publicly available with proper licensing agreements. Training times vary from 2 hours (small datasets) to 2 days (large datasets) on standard hardware.

3. Technical Specifications

3.1 Evaluation Criteria

Accuracy Assessment: Using accuracy, precision (true positives / predicted positives), recall (true positives / actual positives), and F1-score (harmonic mean of precision and recall) criteria.

Resource and Time Assessment: The amount of time and resources used during training and testing.

3.2 Experiment Settings

Hardware Configuration: 15x  vCPUs, 45GB RAM, NVIDIA Tesla V100S 32GB.

Evaluation Methodology: 5-fold cross-validation with stratified sampling was used to ensure robust statistical evaluation.

Software Libraries: scikit-learn 1.3.0, transformers 4.38.0, PyTorch 2.7.1, XGBoost 3.0.2

3.2.1 Dataset Selection

We use the ArXiv Dataset with 11 labels that have the largest length variation across academic domains.

Document Length Context: To better contextualize these word counts, we can convert them to number of pages, using the standard estimate of 500 words per page for double-spaced academic text (14,000 words ≈ 28 pages ≈ short academic paper). By this measure:

• math.ST averages around 28 pages
• math.GR and cs.DS are about 25–26 pages
• cs.IT and math.AC average roughly 20–24 pages
• while cs.CV and cs.NE average only 14–15 pages

This significant variation demonstrates differences in writing styles, document depth, or research reporting norms across fields. Fields like mathematics and theoretical computer science tend to produce more comprehensive or technically dense documents, whereas applied areas like computer vision may favor more concise communication.

3.2.2 Data Size and Training/Test Division

Expected training time on standard hardware: 30 minutes to 8 hours, depending on method complexity.

Minimum Training Data Requirements:

• Simple methods: 50+ samples per class
• Logistic Regression: 100+ samples per class
• XGBoost: 1,000+ samples for optimal performance
• BERT/Transformer models: 2,000+ samples per class

In all experiments, 30% of the data was reserved as the test set. To evaluate the robustness of the model, several variations of the dataset were used: the original class-distributed data, a balanced dataset based on the minimum class size (~2,505 samples), and additional balanced datasets with fixed sizes of 100, 140, and 1,000 samples per class.

4. Results and Analysis

Our experiments reveal counter-intuitive results about the performance-efficiency trade-offs in long document classification.

Why Traditional ML Outperforms Transformers

Our benchmark demonstrates that traditional machine learning approaches offer several advantages:

1. Computational Efficiency: Process entire documents without token limits
2. Training Speed: 10x faster training times with comparable accuracy
3. Resource Requirements: Function effectively on standard CPU hardware
4. Scalability: Handle large document collections without GPU infrastructure

4.1 Performance Rankings

The comparative evaluation across four datasets—Original, Balanced-2505, Balanced-140, and Balanced-100—demonstrates clear performance hierarchies:

Top Performers by F1-Score:

XGBoost consistently outperforms all other models:

• Original: F1 = 0.86
• Balanced-2505: F1 = 0.85
• Balanced-140: F1 = 0.80
• Balanced-100: F1 = 0.75

Logistic Regression and SVM offer strong alternatives:

• Logistic Regression: F1 ≈ 0.71–0.83
• SVM: F1 ≈ 0.72–0.83

Transformer Models perform well on larger datasets:

• BERT-base: Up to F1 = 0.82
• DistilBERT: F1 ≈ 0.75–0.77

Poor Performers:

• RoBERTa-base: F1 down to 0.57 (surprisingly poor)
• Keyword-based methods: F1 = 0.53–0.62

Key Takeaway: XGBoost demonstrates consistent superiority across all dataset sizes, while traditional ML methods generally outperform transformer models in both accuracy and efficiency metrics.

4.2 Cost-Benefit Analysis of Each Method

Training and Inference Times:

Most Efficient:

• Logistic Regression: Trains in <20 seconds, inference <1 second
• XGBoost: Trains in 22-368 seconds, inference ~0.08 seconds

Resource Intensive:

• SVM: Training >2,480 seconds (~41 minutes), inference >1,322 seconds (22 minutes)
• Transformer models: Training up to 2,700 seconds (~45 minutes), inference ~140 seconds

Inefficient:

• Keyword-based: Fast training (2.6s) but extremely slow inference (up to 335 seconds)

Key Takeaway: Efficiency analysis reveals that simpler methods often provide better practical performance for deployment scenarios, challenging assumptions about the necessity of complex models.

4.3 Complete Model Evaluation Summary

4.4 Model Selection Decision Matrix

4.5 Robustness Analysis

High Confidence Findings:

• XGBoost maintains consistent performance across all dataset sizes
• Logistic Regression provides reliable results with minimal computational requirements
• Traditional ML approaches are surprisingly competitive with modern transformers

Areas Requiring Further Research:

• RoBERTa-base’s poor performance may be task-specific and needs investigation
• Optimal chunking strategies for transformer models require domain-specific tuning

Key Takeaway: Results demonstrate that model complexity does not guarantee superior performance, emphasizing the importance of empirical evaluation over theoretical assumptions.

5. Deployment Scenarios

In this section, we explore deployment scenarios for text classification models, highlighting the best-suited algorithms for different operational constraints—ranging from production systems to rapid prototyping—based on trade-offs between accuracy, efficiency, and resource availability.

Production Systems

• Recommendation: XGBoost
• Rationale: Best balance of accuracy (F1: 0.86) and efficiency (training: 6 minutes, inference: instant)
• Use case: High-volume document processing, real-time classification

Resource-Constrained Environments

• Recommendation: Logistic Regression
• Rationale: Excellent efficiency (training: <20s) with good accuracy (F1: 0.83)
• Use case: Startups, embedded systems, edge computing

Maximum Accuracy Requirements

• Recommendation: BERT-base (with GPU infrastructure)
• Rationale: High accuracy (F1: 0.82) with acceptable computational cost
• Use case: Research applications, high-stakes classification

Rapid Prototyping

• Recommendation: Logistic Regression → XGBoost → BERT pipeline
• Rationale: Progressive complexity allows quick iteration and validation
• Use case: Proof of concepts, A/B testing new categories

Key Takeaway: Selection criteria should prioritize practical deployment requirements over theoretical model sophistication, with XGBoost emerging as the optimal choice for most scenarios.

6. Frequently Asked Questions

What is the best method for long document classification?

XGBoost consistently delivers the highest accuracy (F1: 0.86) with reasonable computational requirements, making it the best overall choice for most applications. It is also relatively robust to data volume.

How does BERT compare to traditional machine learning for document classification?

BERT achieves an F1 score of up to 0.82 but requires ten times more computational resources than XGBoost, which reaches an F1 score of 0.81 with significantly faster training and inference. Moreover, BERT’s resource and time consumption grow exponentially with data size, making it impractical for operational environments. In contrast, XGBoost achieves an even higher F1 score of 0.86 when more data is provided.

What document length is considered “long” in NLP?

Documents with 1,000+ words (2+ pages) are typically considered long documents, presenting unique challenges for traditional classification methods.

How much training data do you need for document classification?

Minimum requirements: 100 samples per class for basic models, 1,000+ for optimal XGBoost performance, 2,000+ for transformer models.

Can you classify documents in real-time?

Yes, Logistic Regression and XGBoost provide sub-second inference times suitable for real-time applications. BERT requires 2+ minutes per document.

What’s the accuracy difference between simple and complex methods?

Simple methods achieve 55-62% F1-score, intermediate methods 75-86%, and complex methods 75-82%. The performance gap is smaller than expected.

7. Conclusion

This comprehensive benchmark study demonstrates that traditional machine learning approaches remain highly competitive for long document classification tasks. The analysis reveals that sophisticated transformer models, while powerful, often do not justify their computational overhead for many practical applications.

Key Findings Summary

1. XGBoost emerges as the most versatile solution, offering the best combination of accuracy (F1: 0.86), efficiency, and robustness across different dataset sizes.
2. Logistic Regression provides exceptional value for resource-constrained environments, delivering competitive accuracy (F1: 0.83) with minimal computational requirements.
3. Traditional ML outperforms transformers in efficiency by a factor of 10x while maintaining comparable accuracy levels.
4. RoBERTa-base significantly underperformed expectations, suggesting that model selection requires careful empirical validation rather than relying on general reputation.
5. Keyword-based methods are inadequate for complex classification tasks, with poor accuracy and surprisingly high inference latency.

Strategic Recommendations

For most organizations, XGBoost should be the default starting point for long document classification projects. Its robust performance, reasonable training times, and minimal infrastructure requirements make it suitable for both prototyping and production deployment.

Logistic Regression remains an excellent choice for scenarios requiring rapid deployment, interpretability, or operating under strict resource constraints. The method’s simplicity and reliability make it particularly valuable for baseline establishment and quick validation.

Transformer-based approaches like BERT-base should be considered only when maximum accuracy is required and sufficient GPU infrastructure is available. The computational investment may be justified for high-stakes applications where marginal accuracy improvements provide significant business value.

This study challenges the assumption that more complex models necessarily deliver better results for long document classification. The counter-intuitive results suggest that careful selection based on specific requirements—rather than defaulting to the most sophisticated available method—leads to more successful deployments.

Future Research Directions

Areas requiring further investigation include optimal chunking strategies for transformer models, domain-specific fine-tuning approaches, and the development of more efficient attention mechanisms for long document processing. The surprising underperformance of RoBERTa-base also warrants deeper analysis to understand the factors affecting its effectiveness in this specific task domain.

All results are reproducible using the provided experimental parameters and publicly available datasets. Complete implementation code and detailed configuration settings are available for validation and extension of this research.

Key Takeaway: This benchmark demonstrates that empirical evaluation trumps theoretical sophistication in model selection, with traditional ML methods proving surprisingly effective for long document classification across diverse practical scenarios.

References

1. 1. Chen, Tianqi, and Carlos Guestrin. “XGBoost: A Scalable Tree Boosting System.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016.
   2. Genkin, Alexander, David D. Lewis, and David Madigan. “Sparse Logistic Regression for Text Categorization.” DIMACS Working Group on Monitoring Message Streams Project Report, 2005.
   3. Liu, Yinhan, et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach.” arXiv preprint arXiv:1907.11692, 2019.
   4. Devlin, Jacob, et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
   5. anh, Victor, et al. “DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter.” arXiv preprint arXiv:1910.01108, 2019.
   6. Campos, Ricardo, et al. “YAKE! Keyword Extraction from Single Documents Using Multiple Local Features.” Information Sciences, vol. 509, 2020, pp. 257–289.

Download Resources and Libraries

• Liqun W. (2021). Long Document Dataset – GitHub Repository
  
• S2ORC (Semantic Scholar Open Research Corpus)
  
• Congressional and California state bills – GitHub Repository
  
• GOVREPORT – long document summarization dataset
  
• CUAD (Contract Understanding Atticus Dataset)
  
• MIMIC-III (Medical Information Mart for Intensive Care)
  
• SEC 10 K/Q Filings
  
• scikit-learn 1.3.0
  
• transformers 4.38.0
  
• torch 2.7.1
  
• xgboost 3.0.2