Metadata-Version: 2.1
Name: prometheus-monitoring-scripts
Version: 0.0.0
Summary: prometheus-monitoring-scripts
Author-email: Jordan Tardif <jordan@dreamhost.com>
License: Apache Software License 2.0
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: prometheus-client
Requires-Dist: storable
Provides-Extra: test
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: pytest-cov ; extra == 'test'
Requires-Dist: coverage ; extra == 'test'

# Prometheus Monitoring Scripts

A collection of custom exporters for Prometheus monitoring, designed to collect metrics from various systems and services.

## Installation

Clone the repository and set up the development environment using the Makefile:

```bash
git clone https://git.dreamhost.com/dreamhost/infra/prometheus-monitoring-scripts.git
cd prometheus-monitoring-scripts
make setup
source env/bin/activate
```

This will create a virtual environment, install the package in development mode, and install all required dependencies.

> Note: The `make setup` command requires the `uv` tool, a modern Python package manager. If you don't have `uv` installed, you can install it following the instructions at [https://docs.astral.sh/uv/getting-started/installation/](https://docs.astral.sh/uv/getting-started/installation/).

## Development

The project includes several Makefile targets to help with development:

- `make setup` - Set up the development environment
- `make style` - Check code style using Ruff
- `make autopep` - Automatically fix code style issues using Ruff
- `make test` - Run functional tests
- `make test_smoke` - Run smoke tests

To run specific tests, use:

```bash
make test test=tests/path/to/test
```

## Usage

The monitoring scripts are organized as modules that can be called using the `custom_exporter` command:

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter <module_name> [arguments]
```

In a dev environment, the paths are configured such that you can run `custom_exporter` without a full path.

### Available Modules

- `disk.xfs` - XFS quota metrics for user disk usage
- `disk.xfs_errors` - XFS filesystem error events scraped from kernel log
- `disk.io_errors` - Generic host I/O error events (block layer + userspace EIO) scraped from journal
- `disk.podman` - Disk usage metrics for Podman containers
- `disk.zram` - zram compressed-swap device usage: real RAM cost, compression ratio, incompressible pages
- `mailq.generic` - Postfix mail queue metrics
- `mailq.podman` - Postfix mail queue metrics for Podman containers
- `mailq.mailman` - Mailman queue monitor
- `backups.users` - User backup metrics
- `backups.vms` - VM backup metrics
- `service.podman` - Podman service metrics
- `dphactl.core` - Core dp-ha-ctl metrics for monitoring Redis connection and service status
- `dphactl.systemctl` - Systemctl-based checks for Podman socket and HAManager service 
- `dphactl.logging_checks` - Log-related checks for Redis auth failures and timeouts
- `dphactl.containers` - Container difference checks between hosts
- `dphactl.verify` - Missing file verification checks from dp-ha-ctl
- `dpsnapshot.zfs` - ZFS snapshot timestamp and age monitoring

## Module Documentation

### disk.xfs

Collects XFS quota metrics for disk and inode usage by user on a given mount point.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter disk.xfs /home
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `disk_used_bytes` | Gauge | Disk space used | `user` |
| `disk_quota_bytes` | Gauge | Disk quota (hard limit, 0 = no limit) | `user` |
| `disk_used_inodes` | Gauge | Inodes used | `user` |
| `disk_quota_inodes` | Gauge | Inode quota (hard limit, 0 = no limit) | `user` |
| `disk_project_used_bytes` | Gauge | Disk space used by the account's project | `account_id` |
| `disk_project_quota_soft_bytes` | Gauge | Project disk soft limit (0 = no limit) | `account_id` |
| `disk_project_quota_hard_bytes` | Gauge | Project disk hard limit (0 = no limit) | `account_id` |
| `disk_project_used_inodes` | Gauge | Inodes used by the account's project | `account_id` |
| `disk_project_quota_soft_inodes` | Gauge | Project inode soft limit (0 = no limit) | `account_id` |
| `disk_project_quota_hard_inodes` | Gauge | Project inode hard limit (0 = no limit) | `account_id` |

The exporter runs `xfs_quota` against the mount point given as its argument and reports all users with uid >= 10000.

The `disk_project_*` families are emitted only when `projects` is passed as a second argument (`custom_exporter disk.xfs /home projects`). They read the XFS project quota report, skipping project 0 (space outside any marked tree). Enable this only where project ids are account ids (shared hosts marked by quota enforcement); on VPS hosts project ids identify containers, not accounts. Quota enforcement sets the soft limit to the plan size and the hard limit above it as a buffer, so customer-facing displays should compare usage against the soft limit.

### disk.xfs_errors

Counts XFS filesystem error events found in the kernel log (`journalctl -k -b`) since the current boot. XFS does not auto-remount RO on most metadata errors, so `node_filesystem_readonly` does not flip on hosts experiencing corruption. This exporter surfaces those events into Prometheus.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter disk.xfs_errors
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `xfs_metadata_corruption_events_total` | Counter | XFS "Metadata corruption detected" log events since boot | `device` |
| `xfs_metadata_io_error_events_total` | Counter | XFS "metadata I/O error" log events since boot | `device` |
| `xfs_log_io_error_events_total` | Counter | XFS "log I/O error" log events since boot | `device` |
| `xfs_filesystem_shutdown_events_total` | Counter | XFS "Filesystem has been shut down" log events since boot | `device` |
| `xfs_repair_requested_events_total` | Counter | XFS "Unmount and run xfs_repair" log events since boot | `device` |

Counter values reset to 0 at host reboot. Prometheus handles counter resets correctly in `rate()` / `increase()`. The `device` label is the kernel-side device name as it appears in XFS log messages (e.g. `md10`, `sdc1`, `dm-0`).

The user running this exporter must be able to read the kernel journal. On most distros that means membership in `systemd-journal` or `adm`, or running as root.

### disk.io_errors

Counts generic I/O error events found in the full journal (`journalctl -b`, not just kernel) since boot. Complements `disk.xfs_errors` by catching:

- Block-layer errors below the filesystem (`blk_update_request: I/O error`), which point at the underlying device.
- Buffer I/O errors (`Buffer I/O error on dev ...`).
- Userspace processes hitting EIO (e.g. `rm: cannot remove '...': Input/output error`), which represents the user-visible blast radius of an FS or block-level problem.

Pre-filters at the journald level via `journalctl -g` for efficiency.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter disk.io_errors
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `host_block_io_errors_total` | Counter | `blk_update_request: I/O error` events since boot | `device` |
| `host_buffer_io_errors_total` | Counter | `Buffer I/O error on dev ...` events since boot | `device` |
| `host_userspace_io_errors_total` | Counter | Userspace processes reporting `Input/output error` since boot | (none) |

Counter values reset at host reboot. The user running this exporter must be able to read the journal (`systemd-journal` or `adm` group, or root).

### disk.zram

Reads zram (compressed RAM swap) device stats directly from `/sys/block/zram*`. node_exporter reports zram only as swap (logical) via `/proc/meminfo` and has no zram collector, so the compressed real-RAM footprint and the compression ratio are otherwise invisible.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter disk.zram
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `zram_disksize_bytes` | Gauge | Advertised logical swap capacity (not RAM reserved) | `device` |
| `zram_orig_data_size_bytes` | Gauge | Uncompressed size of stored data (RAM relieved) | `device` |
| `zram_compr_data_size_bytes` | Gauge | Compressed size of stored data | `device` |
| `zram_mem_used_total_bytes` | Gauge | Real RAM consumed by the compressed pool incl. overhead | `device` |
| `zram_mem_limit_bytes` | Gauge | Configured RAM cap for the pool (0 = unlimited) | `device` |
| `zram_repeated_pages` | Gauge | Pages filled with a single repeated value (commonly zero pages), stored with no allocation (from mm_stat `same_pages`) | `device` |
| `zram_incompressible_pages` | Gauge | Incompressible pages stored uncompressed (mm_stat `huge_pages`) | `device` |
| `zram_info` | Gauge | Device present (value 1); carries the active compression algorithm | `device`, `algorithm` |

Compression ratio is left to PromQL: `zram_orig_data_size_bytes / zram_compr_data_size_bytes`. Hosts with no configured zram device emit nothing, so the module is safe to scrape fleet-wide. All values come from `/sys/block/zramN/mm_stat`, `disksize`, and `comp_algorithm` (world-readable; no privilege required).

### mailq.generic

Collects Postfix mail queue metrics by counting files in queue directories.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter mailq.generic
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `postfix_queue_size` | Gauge | Number of emails in queue | `queue` |

Monitors all standard Postfix queues: active, bounce, deferred, incoming, and maildrop.

### mailq.podman

Extends mail queue monitoring to Postfix instances running in Podman containers.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter mailq.podman [container_filter]
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `postfix_queue_size` | Gauge | Number of emails in queue | `machine`, `queue` |
| `postfix_queue_errors` | Gauge | Increment for each error accessing queues | `machine` |

The optional container filter parameter allows monitoring specific containers.

### mailq.mailman

Monitors Mailman queue sizes by counting files in queue directories across multiple mailman instances.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter mailq.mailman [base_path]
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `mailman_queue_size` | Gauge | Number of files in queue directories | `service`, `queue` |

This exporter scans mailman instances in the specified base path (default: `/dh/mailman`) and counts files in both "in" and "out" queue directories. Each service (mailman instance directory) is tracked separately with the directory name as the service label.

### backups.users

Collects metrics about user backup status and history.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter backups.users
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `backup_last_successful` | Gauge | Timestamp of last successful backup | `user`, `machine`, `vmhost` |
| `backup_last_rsync_exit_code` | Gauge | Exit code of last rsync operation | `user`, `machine`, `vmhost` |
| `backup_last_attempted_backup` | Gauge | Timestamp of last backup attempt | `user`, `machine`, `vmhost` |
| `backup_last_user_state` | Gauge | State of last backup (1=active state) | `user`, `machine`, `vmhost`, `state` |
| `backup_state_retrive_failed` | Gauge | Indicates backup state retrieval failed | `machine`, `vmhost` |
| `backup_status` | Gauge | Overall backup status | `machine`, `vmhost`, `status` |

Reads backup state from `/usr/local/dh/var/localdata/backup.state` and user information from `/usr/local/dh/etc/localdata/users.json`.

### backups.vms

Extends backup monitoring to virtual machines using the same metrics as user backups.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter backups.vms
```

Uses the same metrics as `backups.users` but applied to VM guests defined in `/usr/local/dh/etc/localdata/guests.json`.

### service.podman

Monitors systemd services running within Podman containers.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter service.podman <services> [container_filter]
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `node_systemd_unit_state` | Gauge | State of systemd units (1=in this state) | `machine`, `name`, `state`, `type` |
| `podman_exec_errors` | Gauge | Increments for each failed exec into container | `machine` |

The first argument is required and should be a comma-separated list of service names to monitor. The optional second argument filters which containers to check.

### dphactl.core

Provides core metrics for the DP-HA-CTL system, focusing on basic functionality like Redis connection and service status.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter dphactl.core
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `dp_hactl_redis_connected` | Gauge | Status of Redis backend connection (1=connected) | |
| `dp_hactl_service_up` | Gauge | Status of dp-ha-ctl service (1=up) | |

The module implements timeouts for all commands and preserves subprocess exit codes for detailed error reporting.

### dphactl.systemctl

Monitors system services related to DP-HA-CTL functionality using systemctl.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter dphactl.systemctl
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `podman_socket_active` | Gauge | Status of podman.socket systemd unit (1=active) | |
| `hamanager_service_active` | Gauge | Status of hamanager.service systemd unit (1=active) | |

This module can be easily removed or replaced when node-exporter checks are enabled on the hosts.

### dphactl.logging_checks

Examines log files to track Redis-related issues.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter dphactl.logging_checks
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `hamanager_redis_auth_failures` | Gauge | Count of Redis authentication failures in logs | |
| `hamanager_redis_timeouts` | Gauge | Count of Redis timeouts in logs | |

Scans the log files in `/var/log/hamanager/` for specific patterns related to Redis errors.

### dphactl.containers

Monitors container differences between hosts.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter dphactl.containers
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `dp_hactl_missing_containers_total` | Gauge | Count of missing containers | `type`, `problem` |

Reports missing standby and primary containers, with specific error labels for timeout or connection issues.

### dphactl.verify

Runs verification checks on the DP-HA-CTL system to ensure proper configuration.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter dphactl.verify
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `dp_hactl_verify_problems_total` | Gauge | Total number of problems per container | `machine` |
| `dp_hactl_verify_problem_details` | Gauge | Detailed breakdown of problem types | `machine`, `problem` |

Runs the `dp-ha-ctl verify` command and categorizes detected problems for detailed monitoring.

All dphactl modules implement proper error handling with specific subprocess exit code preservation for detailed alerting and debugging.

### dpsnapshot.zfs

Monitors ZFS snapshot timestamps and calculates age metrics for alerting on stale snapshots.

```bash
/opt/prometheus-monitoring-scripts/bin/custom_exporter dpsnapshot.zfs
```

#### Metrics

| Metric Name | Type | Description | Labels |
|-------------|------|-------------|--------|
| `zfs_last_sent_snapshot_timestamp` | Gauge | Unix timestamp of last sent snapshot | |
| `zfs_last_sent_snapshot_age_seconds` | Gauge | Age of last sent snapshot in seconds | |

Reads snapshot information from `/home/.zfs_last_sent_snapshot` file, expecting format `LAST_SENT_SNAPSHOT=prefix_YYYY-MM-DD_HH:MM:SS`. Handles missing files and parsing errors gracefully by logging to stderr without failing.
