A minimal end-to-end solution: parse HDFS logs, template via simplified Drain, embed with BERT (or fallback), train BiLSTM+Attention, and serve predictions via FastAPI with WebSocket. Frontend dashboard displays live alerts.
project/
├── data/
├── preprocessing/
│ └── log_parser.py
├── model/
│ ├── bert_bilstm_train.py
│ ├── inference.py
│ └── saved_model/
├── backend/
│ ├── app.py
│ ├── streaming_api.py
│ └── utils.py
├── frontend/
│ ├── index.html
│ ├── anomaly_dashboard.js
│ └── style.css
└── README.md
- Create environment and install deps:
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt-
Put dataset in
project/data/HDFS_2k.log(already present at workspace rootdata/). -
Train model:
cd project
python model/bert_bilstm_train.py- Run backend:
uvicorn backend.app:app --host 0.0.0.0 --port 8000- Open dashboard:
- Navigate to
http://localhost:8000/ui - Click Connect WS to receive live feed
- Upload file or paste a line and Send
- CLI streaming example:
python project/backend/streaming_api.py- If transformers cannot be installed, the pipeline uses a deterministic hash-based embedding fallback.
- Auto-actions can be integrated in
backend/utils.py.