Skip to content

About

A minimal end-to-end solution: parse HDFS logs, template via simplified Drain, embed with BERT (or fallback), train BiLSTM+Attention, and serve predictions via FastAPI with WebSocket. Frontend dashboard displays live alerts.

Resources

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

Log Anomaly Detection System (HDFS_2k)

A minimal end-to-end solution: parse HDFS logs, template via simplified Drain, embed with BERT (or fallback), train BiLSTM+Attention, and serve predictions via FastAPI with WebSocket. Frontend dashboard displays live alerts.

Structure

project/
 ├── data/
 ├── preprocessing/
 │    └── log_parser.py
 ├── model/
 │    ├── bert_bilstm_train.py
 │    ├── inference.py
 │    └── saved_model/
 ├── backend/
 │    ├── app.py
 │    ├── streaming_api.py
 │    └── utils.py
 ├── frontend/
 │    ├── index.html
 │    ├── anomaly_dashboard.js
 │    └── style.css
 └── README.md

Quick Start

  1. Create environment and install deps:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
  1. Put dataset in project/data/HDFS_2k.log (already present at workspace root data/).

  2. Train model:

cd project
python model/bert_bilstm_train.py
  1. Run backend:
uvicorn backend.app:app --host 0.0.0.0 --port 8000
  1. Open dashboard:
  • Navigate to http://localhost:8000/ui
  • Click Connect WS to receive live feed
  • Upload file or paste a line and Send
  1. CLI streaming example:
python project/backend/streaming_api.py

Notes

  • If transformers cannot be installed, the pipeline uses a deterministic hash-based embedding fallback.
  • Auto-actions can be integrated in backend/utils.py.

About

A minimal end-to-end solution: parse HDFS logs, template via simplified Drain, embed with BERT (or fallback), train BiLSTM+Attention, and serve predictions via FastAPI with WebSocket. Frontend dashboard displays live alerts.

Resources

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages