Rules

besu-health-alerts

2.444s ago

3.496ms

Rule State Error Last Evaluation Evaluation Time
alert: HighMemoryUsage expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.15 for: 2m labels: severity: warning annotations: description: Available memory is less than 15% for 2 minutes. summary: ⚠️ High memory usage on {{ $labels.instance }} ok 2.444s ago 1.1ms
alert: DiskSpaceLow expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.1 for: 2m labels: severity: critical annotations: description: Root filesystem has less than 10% space left. summary: "\U0001F4BE Low disk space on {{ $labels.instance }}" ok 2.443s ago 550us
alert: HighCPUUsage expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 90 for: 2m labels: severity: warning annotations: description: CPU usage over 90% for 2 minutes. summary: "\U0001F525 High CPU usage on {{ $labels.instance }}" ok 2.443s ago 660.3us
alert: NodeExporterDown expr: up{job="node_exporter"} == 0 for: 1m labels: severity: critical annotations: description: Prometheus target `node_exporter` is unreachable. summary: ❌ Node Exporter down on {{ $labels.instance }} ok 2.442s ago 238.3us
alert: RPCDown expr: probe_success{job="blackbox_eth_blocknumber"} == 0 for: 1m labels: severity: critical annotations: description: The endpoint at {{ $labels.instance }} failed probe. summary: ⛔ RPC endpoint is down ok 2.442s ago 133.8us
alert: ValidatorNotSigning expr: time() - max by(instance) (besu_block_timestamp{job=~"validators|besu"}) > 7200 for: 10m labels: severity: critical annotations: description: Validator {{ $labels.instance }} hasn't signed a block in the last 2 hours. summary: ⛔ Validator not signing blocks ok 2.442s ago 203.6us
alert: NoNewBlocks expr: increase(besu_block_number[10m]) == 0 for: 5m labels: severity: critical annotations: description: No block has been produced in the last 10 minutes. Network may be stuck. summary: "\U0001F4C9 No new blocks produced" ok 2.442s ago 108.9us
alert: DataDiskUsageCritical expr: (node_filesystem_size_bytes{mountpoint="/data"} - node_filesystem_avail_bytes{mountpoint="/data"}) / node_filesystem_size_bytes{mountpoint="/data"} > 0.9 for: 2m labels: severity: critical annotations: description: The /data mount is over 90% full on {{ $labels.instance }} summary: "\U0001F4E6 Disk usage > 90% on /data" ok 2.442s ago 220.8us
alert: FaucetTxErrorSpike expr: increase(faucet_tx_errors_total[5m]) > 3 for: 1m labels: severity: warning annotations: description: Faucet had more than 3 transaction errors in 5 minutes on {{ $labels.instance }} summary: "\U0001F6B1 Faucet transaction errors" ok 2.442s ago 87.71us
alert: PrometheusTargetMissing expr: up == 0 for: 2m labels: severity: critical annotations: description: 'Target {{ $labels.instance }} (job: {{ $labels.job }}) is down.' summary: "\U0001F4E1 Prometheus target down: {{ $labels.job }}" ok 2.442s ago 170.9us

besu_block_lag

2.257s ago

536.3us

Rule State Error Last Evaluation Evaluation Time
record: besu_head_block expr: max by(instance) (blockchain_chain_head_block or blockchain_best_block_number) ok 2.257s ago 295.4us
record: besu_validator_max_block expr: max without(instance) (besu_head_block{role="validator"}) ok 2.257s ago 77.28us
record: besu_rpc_block expr: max without(instance) (besu_head_block{role="rpc"}) ok 2.257s ago 53.65us
alert: RPC_Block_Lag expr: (besu_validator_max_block - on() besu_rpc_block) > 5 for: 3m labels: severity: page annotations: description: RPC head lags validators by >5 blocks for 3m. summary: RPC node is {{ $value }} blocks behind ok 2.257s ago 98.37us

rpc_probes

2.369s ago

451us

Rule State Error Last Evaluation Evaluation Time
alert: RPC_Https_Probe_Failing expr: probe_success{job="blackbox_eth_blocknumber"} == 0 for: 2m labels: severity: page annotations: description: POST eth_blockNumber failed >2m. summary: RPC probe failing ({{ $labels.instance }}) ok 2.369s ago 295.5us
alert: RPC_Https_Probe_Slow expr: probe_duration_seconds{job="blackbox_eth_blocknumber"} > 2 for: 5m labels: severity: warn annotations: description: Latency > 2s for 5m. summary: RPC probe slow ({{ $labels.instance }}) ok 2.369s ago 121.5us