Introduction
Predictive maintenance is a technique that uses data analysis tools and techniques to detect anomalies in your operation and possible defects in equipment and processes so you can fix them before they result in failure.
In this case, we are going to predict the engine failure using the turbofan engine dataset. First, we will be generating a model for predicting the remaining useful time(rul) of the engine. Next, we will be using the kafka streaming service to stream data into the predictor and finally we are setting a RUL value where it should alert the system. For example, in this case if the predicted RUL is below 120. It would post an alert.
Download the .zip files for implementation.
Technologies used,
- Spark and it’s ML.
- Kafka - sql, Spark structured streaming.
Workflow / Architecture
1. Notebook
- EDA on dataset.
- Model, scaler and transformation is done using pyspark ML and saved separately.
- RMSE - 50 (which I got using Azure ML studio)
Attaching the screenshot of a few random engine predictions vs actual RUL.
2. Python
- Simulator - Simulates data from the CSV and posts to stream topic(kafka) by randomly picking engines.
- Predictor - Uses the exported model for predicting the RUL of incoming data on the stream topic and if the defined RUL is met- posts it to Alert topic(kafka).
- Alert - Listens to kafka alert topic and notifies.
3. Shell
For starting the services,
- engine alert monitor (listens to alert topic).
- Simulator.
- Monitor/prediction.
*** We can replace the simulator with the machine logs and send it directly to the kafka topic***
– simulator is just taken for the implementation/demo purpose.
Following - The real time implementation screenshots
In the above fig.
Top left corner - Started the zookeeper(broker) and kafka.
Bottom right corner- Started the engine-alert topic(It would be listening for the alerts).
Above fig.
Top left corner - running the simulator - it randomly posts data from the CSV's to engine-stream topic.
Bottom left - running the monitor/prediction script with a single argument(RUL).
Where,
RUL argument is the value that we wanted the system to alert when it drops below.
Above fig.
Here, the RUL argument has been set to 100 - which means if the predicted RUL falls below 100, it would be sending that data(namely engine id, cycle and predicted RUL) to the alert topic(kafka).
Received alert in the alert window.
Above fig,
Here, the RUL has been set to 120.
Summary
Thus, we finally received alerts for the data which fall below the RUL of 120. This is a technique that is used in many of the OEM.
You can find the full working source code of the project below. If anyone needs guidance in implementing it. Feel free to contact me.
Join the conversation! Your thoughts help the community grow.