¶ RM MONITORING, Version 1.1
¶ What's New
RM Monitoring v.1.1 now includes CPU & GPU usage: temperatures and power consumption.
- New sensors for CPU core temperatures and CPU core powers
- CPU temperature sensor name is ‘cpu_n_t’ where ‘n’ is the number of the CPUs, e.g. if your system is 2-CPU, then you'll have cpu_1_t and cpu_2_t
- CPU power consumption sensor name is ‘cpu_n_pwr’ where ‘n’ is the number of the CPUs, e.g. if your system is 2-CPU, then you'll have cpu_1_pwr and cpu_2_pwr
- New sensors for GPU temperatures and GPU powers
- GPU temperature sensor name is ‘cpu_n_t’ where ‘n’ is the number of the GPUs, e.g. if your system is 4-CPU, then you'll have gpu_1_t, gpu_2_t, gpu_3_t and gpu_4_t,
- GPU power consumption sensor name is ‘cpu_n_pwr’ where ‘n’ is the number of the GPUs, e.g. if your system is 2-CPU, then you'll have gpu_1_pwr, gpu_2_pwr, gpu_2_pwr and gpu_2_pwr,
- Improved sensors organization in a dashboard display
- Controller Health Monitor indicates if any alarms or errors in a liquid cooling system have been discovered.
- Panels for Flow, Pressure and Level sensors (sensors themselves are under production and estimated for an upcoming release)
- Current and Total system uptime panels
For the list of all sensors, please visit Sensors page.
¶ Technical Requirements
- OS: Windows 10 / Ubuntu 20.04
Dependency for Ubuntu: the target system must have nvidia-smi and sensors utilities installed
- Hard disk drive: 292MB
- Controller firmware version 37.1 or newer
¶ Downloads
¶ Installation Package
- SQLite (database engine)
- Grafana (monitoring & analytics application)
¶ Installation Steps for Windows OS
- Double click on the executable to start installation.
- Check the checkbox if you want to use local database
¶ Using Configuration File
Shortcut to Configuration file is available from Start Menu -> Comino -> rm-monitor.yml. Open the file in the notepad application.
SerialPort:
Name: auto (Do not edit, this is controller virtual port)
RequestPeriodSecs: 5 (Controller polling time in seconds)
InternalDB:
Enabled: true (change to FALSE if you do not need local data base)
RetentionPeriodDays: 120 (how long the local data base stores data)
HttpServer:
Address: localhost (Address at which the api will be available)
Port: 20000 (Port at which the api will be available)
InfluxDB:
Enabled: true
BucketName: rm-monitor-bucket
ServerURL: http://localhost:8086
AuthToken: <paste here token you've copied from InfluxDB>
OrganizationName: default-org <paste here organization name from InfluxDB>
¶ Using Internal Data Base and Grafana Interface
To use local Grafana interface, click on RM Monitor shortcut in Start Menu -> Comino -> Open RM Monitor or proceed to http://localhost:3000/d/-GhXmAY7k/rm
¶ Controller Health Monitor
Panel consists of a number of "health" sensors and indicates with a color if anything happens or requires attention. New sensors:
- ALARM, ERROR – indicates a number of issues during a liquid cooling system operation. ALARM is a warning, indicates approaching to the critical threshold and requires attention to the system until it becomes an ERROR.
- FLOW, LEVEL, PRESSURE – sensors are under production and estimated for an upcoming release.
- CURRENT UPTIME – shows the duration of the controller current session.
- TOTAL UPTIME – shows total duration of the controller operation.
¶ Cooling Performance Panel
CPU & GPUs Temperatures as well as Total Energy Consumptions (CPU and GPUs powers) have been added to Air and Coolant inlet & outlet temperatures and calculated coolant system efficiency.
¶ Energy Consumption
Shows the power consumption of the processor and video cards as a real-time data.
¶ Fans & Pumps
Shows number of rotation per minute for each fan & pump.
¶ Suggested Monitoring Alerts
OK – everything is OK, no notifications or alarms
Warning – warning, approaching to the critical threshold
Critical –critical error, the controller turns everything off, the start is blocked until the temperature returns to the green zone
| Sensor Name | Critical | Warning | OK | Warning | Critical | |
|---|---|---|---|---|---|---|
| COOLANT IN | T1 | ..1 |
2..3 |
4..57 |
58..59 |
60.. |
| COOLANT OUT | T2 | ..1 |
2..3 |
4..57 |
58..59 |
60.. |
| AIR IN | T3 | ..1 |
2..3 |
4..34 |
35..37 |
38.. |
| AIR OUT | T4 | ..1 |
2..3 |
4..59 |
60..64 |
65.. |
| VOLTAGE 12V | V | ..10.7 |
10.8..11.3 |
11.4..12.6 |
12.7..13.2 |
13.3.. |
| HUMIDITY | RH | 0..19 |
20..29 |
30..59 |
60..85 |
86..100 |
| STM | T0 | ..-20 |
-19..0 |
1..74 |
75..77 |
78.. |
| VRM | T5 | ..-20 |
-19..0 |
1..89 |
90..104 |
105.. |
| PCB | T6 | ..-20 |
-19..0 |
1..99 |
100..117 |
118.. |
* all temperature degrees in the table are in Celsius
¶ Using REST API
To get sensors data via URL through Rest API proceed to http://localhost:20000/sensors
Check the table with SENSORS for more details
¶ DNAT
Before connecting to the monitoring system make sure that both your RM machine and the monitoring system are in the same network, i.e. if you’re using the monitoring system in your local network, then you can start connecting. Otherwise, you need to translate the destination IP address of the internal server to public IP address (DNAT).
¶ Using Cloud InfluxDB
To connect to cloud InfluxDB paste server url, authentication token and organization name into the configuration file.
¶ Bug Report and Program Improvement
If you notice a bug in a sensor report, or an undefined value, please email to monitoring@comino.com. Your suggestions on monitoring improvements are also appreciated. Thanks!