TestSense AI is a machine-learning application that classifies Continuous Integration and Continuous Deployment (CI/CD) error messages into predefined technical categories. It combines text classification, model evaluation, out-of-distribution (OOD) analysis, and an interactive Streamlit dashboard to help developers investigate pipeline failures.
Project scope: Error classification, prediction-score analysis, model evaluation, and human-in-the-loop review. TestSense AI does not autonomously execute fixes or repair pipelines.
- CI/CD error classification: Predict a category from a CI/CD error message.
- Top-5 candidate categories: Inspect the five highest-scoring category predictions.
- Manual-review recommendations: Flag predictions whose highest class score falls below a configurable threshold.
- Model comparison: Compare word-level TF-IDF against combined word- and character-level TF-IDF features.
- Model evaluation: View accuracy, macro F1, and weighted F1 on a held-out test set.
- Error analysis: Review misclassified examples and frequently confused categories.
- OOD evaluation: Inspect model performance on examples that differ from the training distribution, including examples with unseen labels.
- Matplotlib visualizations: Explore confidence distributions, category distributions, prediction distributions, and accuracy by confidence band.
- CSV export: Download prediction scores for further analysis.
- Interactive dashboard: Use the model through a Streamlit web interface.
| Area | Technologies |
|---|---|
| Language | Python |
| Machine learning | scikit-learn, Logistic Regression |
| Text features | Word TF-IDF, character TF-IDF, n-grams |
| Data processing | pandas, NumPy |
| Visualization | Matplotlib |
| Dashboard | Streamlit |
| Model persistence | joblib |
| Version control | Git, GitHub |
- Input: The user submits a CI/CD error message.
- Feature extraction: Word- and character-level TF-IDF features represent the message.
- Classification: A Logistic Regression classifier predicts a category learned during training.
- Score review: The dashboard shows the top candidate categories and their model scores.
- Human review: A configurable threshold flags lower-scoring predictions for review.
- Evaluation: Saved test and OOD reports and Matplotlib figures provide insight into model performance and limitations.
The classifier uses error_message as its input and category as its target. Fields such as root_cause and fix_command are not used as prediction inputs, helping prevent target leakage.
Score caution: The highest class score is not necessarily a calibrated probability that the prediction is correct. High scores do not guarantee correctness, and low scores do not necessarily indicate an error.
| Metric | Result |
|---|---|
| Accuracy | 83.29% |
| Macro F1 | 0.8555 |
| Weighted F1 | 0.8375 |
The combined word- and character-level TF-IDF candidate was selected based on validation macro F1.
| Metric | Baseline: word TF-IDF | Selected: word + character TF-IDF |
|---|---|---|
| Accuracy | 83.29% | 84.58% |
| Macro F1 | 0.8472 | 0.8566 |
| Weighted F1 | 0.8381 | 0.8498 |
The held-out test set was used after model selection and achieved 83.29% accuracy. Validation and test results measure performance on the supplied splits, not a guarantee of production performance.
| Metric | Result |
|---|---|
| Total OOD examples | 489 |
| Examples with categories known during training | 434 |
| Examples with labels unseen during training | 55 |
| Accuracy on known-category examples | 52.30% |
| Macro F1 on known-category examples | 0.4356 |
Performance drops substantially on the supplied OOD data. The model is a closed-set classifier: it predicts among categories seen during training and cannot reliably identify a new category just because an input is unfamiliar.
The dashboard uses figures generated by Matplotlib:
| Figure | Purpose |
|---|---|
confidence_distribution.png |
Distribution of maximum prediction scores |
accuracy_by_confidence.png |
Observed accuracy across confidence bands |
ood_category_distribution.png |
Category distribution in the OOD dataset |
ood_prediction_distribution.png |
Model prediction distribution on OOD examples |
Additional reports include confusion matrices, classification reports, model comparisons, and misclassified examples.
- Python installed (Python 3.12 is a reasonable deployment target if it is compatible with the pinned dependencies)
- Git
- The trained model and report/figure files listed under Required files
Clone your TestSense AI GitHub repository, then open a terminal in the repository directory:
git clone https://github.com/Chandrakant-Suman/TestSense.git
cd TestSense
Replace the two URL/name placeholders with your GitHub repository details.
Windows PowerShell:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
For reproducible model loading, keep the scikit-learn version compatible with the version used when testsense_best.joblib was created. Pin dependency versions in requirements.txt before publishing a release.
python -m streamlit run app.py
Open the local URL printed in the terminal, usually http://localhost:8501.
- Push the project to GitHub.
- Confirm that
app.py,requirements.txt, the trained model, the test report, and the required figure files are committed. - Visit Streamlit Community Cloud and sign in with GitHub.
- Create an app using your repository, the
mainbranch (or your chosen branch), andapp.pyas the main file. - Choose a Python version supported by your dependency versions and deploy.
- Review the deployment logs and test classification, report metrics, images, and CSV download on the public URL.
Do not commit API keys, credentials, .env files, or Streamlit secrets to GitHub.
TestSense/
โโโ app.py
โโโ requirements.txt
โโโ README.md
โโโ LICENSE
โโโ .gitignore
โโโ models/
โ โโโ testsense_best.joblib
โโโ reports/
โ โโโ test_report.json
โ โโโ test_predictions.csv
โ โโโ test_confusion_matrix.csv
โ โโโ model_comparison.csv
โ โโโ ood_predictions.csv
โ โโโ ood_confidence_analysis.csv
โ โโโ figures/
โ โโโ accuracy_by_confidence.png
โ โโโ confidence_distribution.png
โ โโโ ood_category_distribution.png
โ โโโ ood_prediction_distribution.png
โโโ src/
โ โโโ inspect_data.py
โ โโโ train_model.py
โ โโโ compare_models.py
โ โโโ evaluate_model.py
โ โโโ evaluate_ood.py
โ โโโ analyze_errors.py
โ โโโ analyze_confidence.py
โโโ tests/
โโโ data/
โโโ raw/
At runtime, the dashboard needs:
models/testsense_best.joblibreports/test_report.json- The four PNG files inside
reports/figures/
Keep those paths and filenames unchanged unless you also update app.py. Dataset CSVs and the other reports are useful for training, evaluation, and reproducibility, but the dashboard's runtime requirements are the files listed above.
- The classifier predicts only categories represented in training.
- OOD performance is materially lower than test-set performance on the provided data.
- Model scores are not established as calibrated probabilities of correctness.
- The manual-review threshold is a heuristic and should be evaluated using validation data.
- The application does not independently establish a root cause or verify that a remediation is safe.
- The current version does not execute fix commands or autonomously repair pipelines.
Always inspect the underlying logs and verify predictions before making deployment or infrastructure changes.
- Calibrate class probabilities and evaluate threshold choices.
- Improve detection of unfamiliar and out-of-distribution inputs.
- Improve performance on frequently confused categories.
- Add automated regression tests and GitHub Actions CI.
- Monitor input distributions and prediction behavior after deployment.
- Explore retrieval-assisted troubleshooting and safe, human-approved remediation recommendations.
TestSense AI uses the CI/CD Pipeline Failures Dataset by Riaan Sendhav.
- Source repository: https://github.com/riaansendhav-cmd/cicd-selfhealing-dataset
- Dataset repository description: A labelled CI/CD failure dataset for classification, root-cause analysis, fix recommendation, and self-healing research.
- License stated in the upstream README: Creative Commons Attribution 4.0 International (CC BY 4.0)
The upstream README describes a main dataset of 7,777 labelled samples and an OOD set of 489 samples. The local project uses the supplied train.csv, val.csv, test.csv, and ood.csv files.
If you redistribute the dataset or derived materials, follow the upstream license terms and retain appropriate attribution. The dataset's license does not automatically determine the license for this project's independently written code. Check the upstream repository for any updates to its licensing and attribution requirements.
@dataset{sendhav2026cicd,
title = {CI/CD Pipeline Failures Dataset: A Benchmark for Self-Healing Pipeline Research},
author = {Sendhav, Riaan},
year = {2026},
publisher = {GitHub},
url = {https://github.com/riaansendhav-cmd/cicd-selfhealing-dataset}
}
The root LICENSE file applies to the original TestSense AI source code authored for this project only. The upstream dataset and other third-party materials remain subject to their own applicable license terms; see Dataset Attribution.
Thanks to Riaan Sendhav for publishing the CI/CD Pipeline Failures Dataset for research and experimentation in CI/CD failure classification and self-healing systems.