Evaluate and Fine-Tune a Model for Support Ticket Classification

Project brief

We have approximately 24,000 historical English-language support tickets labelled across 12 routing categories. Our current keyword rules require frequent maintenance and often confuse billing, subscription, and account-access requests.

We need a reproducible comparison of classification approaches and, where justified, a fine-tuned model.

The question to answer

Can a model improve routing quality enough to justify its inference cost and maintenance requirements?

Fine-tuning is one candidate approach. We expect you to compare it with a simpler baseline before recommending a deployment path.

Work packages

Data audit: Inspect label consistency, class imbalance, duplicate tickets, and sensitive fields. Flag ambiguous labels for our support team to adjudicate.

Evaluation design: Create leakage-resistant training, validation, and held-out test splits. Keep related conversations together. Report macro-F1, per-class precision and recall, confusion patterns, and the proportion routed for manual review.

Experiments: Compare the existing rules, a lightweight text-classification baseline, and an agreed model-based approach. Proceed with one fine-tuning run and one refinement cycle only after the baseline review confirms a reasonable case.

Recommendation: Provide the selected model or baseline, inference code, reproducible experiment configuration, and a documented rollout recommendation.

How success will be assessed

We will agree numerical targets after the data audit, before model development. The final assessment must use the locked test set and include a breakdown of failure cases, inference costs, and operational limitations.

A recommendation to retain a simpler baseline is acceptable if supported by evidence. We are not buying a predetermined improvement claim.

Working arrangements

Our team will supply the dataset through an approved environment and resolve up to 200 flagged labelling questions. Training data must not be uploaded to third-party services without approval.

The $8,000 budget covers the full conditional scope. Compute costs are separate. Production platform development and continuous retraining are excluded.

Please describe your approach to label leakage, imbalanced classes, and reproducibility.