Skip to content
Chandrakant-SumanPublic

About

AI-powered tool that reads test logs, predicts failure categories, evaluates its predictions, and generates actionable reports.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

ย 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

TestSense AI ๐Ÿงช

Intelligent CI/CD Error Classification & Failure Analysis

TestSense AI is a machine-learning application that classifies Continuous Integration and Continuous Deployment (CI/CD) error messages into predefined technical categories. It combines text classification, model evaluation, out-of-distribution (OOD) analysis, and an interactive Streamlit dashboard to help developers investigate pipeline failures.

Project scope: Error classification, prediction-score analysis, model evaluation, and human-in-the-loop review. TestSense AI does not autonomously execute fixes or repair pipelines.

โœจ Features

  • CI/CD error classification: Predict a category from a CI/CD error message.
  • Top-5 candidate categories: Inspect the five highest-scoring category predictions.
  • Manual-review recommendations: Flag predictions whose highest class score falls below a configurable threshold.
  • Model comparison: Compare word-level TF-IDF against combined word- and character-level TF-IDF features.
  • Model evaluation: View accuracy, macro F1, and weighted F1 on a held-out test set.
  • Error analysis: Review misclassified examples and frequently confused categories.
  • OOD evaluation: Inspect model performance on examples that differ from the training distribution, including examples with unseen labels.
  • Matplotlib visualizations: Explore confidence distributions, category distributions, prediction distributions, and accuracy by confidence band.
  • CSV export: Download prediction scores for further analysis.
  • Interactive dashboard: Use the model through a Streamlit web interface.

๐Ÿ› ๏ธ Technology Stack

Area Technologies
Language Python
Machine learning scikit-learn, Logistic Regression
Text features Word TF-IDF, character TF-IDF, n-grams
Data processing pandas, NumPy
Visualization Matplotlib
Dashboard Streamlit
Model persistence joblib
Version control Git, GitHub

๐Ÿง  How It Works

  1. Input: The user submits a CI/CD error message.
  2. Feature extraction: Word- and character-level TF-IDF features represent the message.
  3. Classification: A Logistic Regression classifier predicts a category learned during training.
  4. Score review: The dashboard shows the top candidate categories and their model scores.
  5. Human review: A configurable threshold flags lower-scoring predictions for review.
  6. Evaluation: Saved test and OOD reports and Matplotlib figures provide insight into model performance and limitations.

The classifier uses error_message as its input and category as its target. Fields such as root_cause and fix_command are not used as prediction inputs, helping prevent target leakage.

Score caution: The highest class score is not necessarily a calibrated probability that the prediction is correct. High scores do not guarantee correctness, and low scores do not necessarily indicate an error.

๐Ÿ“Š Evaluation Results

Held-out test set

Metric Result
Accuracy 83.29%
Macro F1 0.8555
Weighted F1 0.8375

Model selection on the validation set

The combined word- and character-level TF-IDF candidate was selected based on validation macro F1.

Metric Baseline: word TF-IDF Selected: word + character TF-IDF
Accuracy 83.29% 84.58%
Macro F1 0.8472 0.8566
Weighted F1 0.8381 0.8498

The held-out test set was used after model selection and achieved 83.29% accuracy. Validation and test results measure performance on the supplied splits, not a guarantee of production performance.

Out-of-distribution evaluation

Metric Result
Total OOD examples 489
Examples with categories known during training 434
Examples with labels unseen during training 55
Accuracy on known-category examples 52.30%
Macro F1 on known-category examples 0.4356

Performance drops substantially on the supplied OOD data. The model is a closed-set classifier: it predicts among categories seen during training and cannot reliably identify a new category just because an input is unfamiliar.

๐Ÿ“ˆ Visual Analytics

The dashboard uses figures generated by Matplotlib:

Figure Purpose
confidence_distribution.png Distribution of maximum prediction scores
accuracy_by_confidence.png Observed accuracy across confidence bands
ood_category_distribution.png Category distribution in the OOD dataset
ood_prediction_distribution.png Model prediction distribution on OOD examples

Additional reports include confusion matrices, classification reports, model comparisons, and misclassified examples.

๐Ÿ–ฅ๏ธ Run Locally

Prerequisites

  • Python installed (Python 3.12 is a reasonable deployment target if it is compatible with the pinned dependencies)
  • Git
  • The trained model and report/figure files listed under Required files

1. Get the project

Clone your TestSense AI GitHub repository, then open a terminal in the repository directory:

git clone https://github.com/Chandrakant-Suman/TestSense.git
cd TestSense

Replace the two URL/name placeholders with your GitHub repository details.

2. Create and activate a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

3. Install dependencies

python -m pip install -r requirements.txt

For reproducible model loading, keep the scikit-learn version compatible with the version used when testsense_best.joblib was created. Pin dependency versions in requirements.txt before publishing a release.

4. Start the dashboard

python -m streamlit run app.py

Open the local URL printed in the terminal, usually http://localhost:8501.

โ˜๏ธ Deploy to Streamlit Community Cloud

  1. Push the project to GitHub.
  2. Confirm that app.py, requirements.txt, the trained model, the test report, and the required figure files are committed.
  3. Visit Streamlit Community Cloud and sign in with GitHub.
  4. Create an app using your repository, the main branch (or your chosen branch), and app.py as the main file.
  5. Choose a Python version supported by your dependency versions and deploy.
  6. Review the deployment logs and test classification, report metrics, images, and CSV download on the public URL.

Do not commit API keys, credentials, .env files, or Streamlit secrets to GitHub.

๐Ÿ“ Project Structure

TestSense/
โ”œโ”€โ”€ app.py
โ”œโ”€โ”€ requirements.txt
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ LICENSE
โ”œโ”€โ”€ .gitignore
โ”œโ”€โ”€ models/
โ”‚   โ””โ”€โ”€ testsense_best.joblib
โ”œโ”€โ”€ reports/
โ”‚   โ”œโ”€โ”€ test_report.json
โ”‚   โ”œโ”€โ”€ test_predictions.csv
โ”‚   โ”œโ”€โ”€ test_confusion_matrix.csv
โ”‚   โ”œโ”€โ”€ model_comparison.csv
โ”‚   โ”œโ”€โ”€ ood_predictions.csv
โ”‚   โ”œโ”€โ”€ ood_confidence_analysis.csv
โ”‚   โ””โ”€โ”€ figures/
โ”‚       โ”œโ”€โ”€ accuracy_by_confidence.png
โ”‚       โ”œโ”€โ”€ confidence_distribution.png
โ”‚       โ”œโ”€โ”€ ood_category_distribution.png
โ”‚       โ””โ”€โ”€ ood_prediction_distribution.png
โ”œโ”€โ”€ src/
โ”‚   โ”œโ”€โ”€ inspect_data.py
โ”‚   โ”œโ”€โ”€ train_model.py
โ”‚   โ”œโ”€โ”€ compare_models.py
โ”‚   โ”œโ”€โ”€ evaluate_model.py
โ”‚   โ”œโ”€โ”€ evaluate_ood.py
โ”‚   โ”œโ”€โ”€ analyze_errors.py
โ”‚   โ””โ”€โ”€ analyze_confidence.py
โ”œโ”€โ”€ tests/
โ””โ”€โ”€ data/
    โ””โ”€โ”€ raw/

Required files for the dashboard

At runtime, the dashboard needs:

  • models/testsense_best.joblib
  • reports/test_report.json
  • The four PNG files inside reports/figures/

Keep those paths and filenames unchanged unless you also update app.py. Dataset CSVs and the other reports are useful for training, evaluation, and reproducibility, but the dashboard's runtime requirements are the files listed above.

โš ๏ธ Limitations

  • The classifier predicts only categories represented in training.
  • OOD performance is materially lower than test-set performance on the provided data.
  • Model scores are not established as calibrated probabilities of correctness.
  • The manual-review threshold is a heuristic and should be evaluated using validation data.
  • The application does not independently establish a root cause or verify that a remediation is safe.
  • The current version does not execute fix commands or autonomously repair pipelines.

Always inspect the underlying logs and verify predictions before making deployment or infrastructure changes.

๐Ÿ”ฎ Possible Improvements

  • Calibrate class probabilities and evaluate threshold choices.
  • Improve detection of unfamiliar and out-of-distribution inputs.
  • Improve performance on frequently confused categories.
  • Add automated regression tests and GitHub Actions CI.
  • Monitor input distributions and prediction behavior after deployment.
  • Explore retrieval-assisted troubleshooting and safe, human-approved remediation recommendations.

๐Ÿ“š Dataset Attribution

TestSense AI uses the CI/CD Pipeline Failures Dataset by Riaan Sendhav.

The upstream README describes a main dataset of 7,777 labelled samples and an OOD set of 489 samples. The local project uses the supplied train.csv, val.csv, test.csv, and ood.csv files.

If you redistribute the dataset or derived materials, follow the upstream license terms and retain appropriate attribution. The dataset's license does not automatically determine the license for this project's independently written code. Check the upstream repository for any updates to its licensing and attribution requirements.

Suggested dataset citation

@dataset{sendhav2026cicd,
  title     = {CI/CD Pipeline Failures Dataset: A Benchmark for Self-Healing Pipeline Research},
  author    = {Sendhav, Riaan},
  year      = {2026},
  publisher = {GitHub},
  url       = {https://github.com/riaansendhav-cmd/cicd-selfhealing-dataset}
}

๐Ÿ“„ License

The root LICENSE file applies to the original TestSense AI source code authored for this project only. The upstream dataset and other third-party materials remain subject to their own applicable license terms; see Dataset Attribution.

๐Ÿ™Œ Acknowledgements

Thanks to Riaan Sendhav for publishing the CI/CD Pipeline Failures Dataset for research and experimentation in CI/CD failure classification and self-healing systems.

About

AI-powered tool that reads test logs, predicts failure categories, evaluates its predictions, and generates actionable reports.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages