I have a problem with my zabbix instance. I am trying to monitor my proxmox nodes on…
- User
I have a problem with my zabbix instance. I am trying to monitor my proxmox nodes on zabbix. I am using the template for this, but trying to edit the template so that instead of getting 3 alerts for the same thing, only 1 node (the one running the vm) sends the alert about the vm. How is the best way to do this?
- Assistant
You're encountering a common challenge when monitoring clustered environments like Proxmox with Zabbix. The issue arises because each Zabbix agent on a Proxmox node can query the entire cluster's state via the Proxmox API.
Here's why you get 3 alerts:
- Proxmox Node 1 (Zabbix Host A): Queries Proxmox API, sees VM1, VM2, VM3. Creates items/triggers for all three.
- Proxmox Node 2 (Zabbix Host B): Queries Proxmox API, sees VM1, VM2, VM3. Creates items/triggers for all three.
- Proxmox Node 3 (Zabbix Host C): Queries Proxmox API, sees VM1, VM2, VM3. Creates items/triggers for all three.
If VM1 goes down, all three Zabbix hosts will detect it and raise an alert.
The best way to solve this is to filter the Low-Level Discovery (LLD) process so that each Proxmox node only discovers and monitors VMs that are currently running on that specific node.
Let's break down how to implement this:
Best Way: Filter Low-Level Discovery (LLD)
This method is superior because it prevents unnecessary items and triggers from being created on hosts that aren't responsible for a given VM, saving resources and preventing duplicate alerts at the source.
Prerequisites:
- Zabbix Agent: Ensure the Zabbix agent is running on each Proxmox node.
- Proxmox API Token: Create a read-only API token for a Zabbix-specific user in Proxmox.
- Go to Datacenter -> Permissions -> API Tokens.
- Add a new token. Assign it to a user (e.g.,
zabbix@pve). - Grant the user
PVEAuditorrole (or custom role withVM.AuditandSys.Auditpermissions) at the/path. - Store the Token Secret securely! (e.g., in a file like
/etc/zabbix/proxmox_api_token.confthat only thezabbixuser can read).
Steps:
1. Create a Custom LLD Script
This script will query Proxmox and only output VMs running on the current node.
Create a file, e.g.,
/etc/zabbix/scripts/proxmox_vm_discovery_local.py:#!/usr/bin/env python3 import json import subprocess import os import sys import socket def get_proxmox_api_token(token_file): try: with open(token_file, 'r') as f: lines = f.readlines() for line in lines: if line.startswith('PVE_TOKEN_ID='): token_id = line.split('=')[1].strip() elif line.startswith('PVE_TOKEN_SECRET='): token_secret = line.split('=')[1].strip() return token_id, token_secret except FileNotFoundError: print(f"Error: Token file not found at {token_file}", file=sys.stderr) sys.exit(1) except Exception as e: print(f"Error reading token file: {e}", file=sys.stderr) sys.exit(1) def main(): token_file = "/etc/zabbix/proxmox_api_token.conf" # Adjust path as needed token_id, token_secret = get_proxmox_api_token(token_file) # Set environment variables for pvesh os.environ['PVE_TOKEN_ID'] = token_id os.environ['PVE_TOKEN_SECRET'] = token_secret current_node_hostname = socket.gethostname() # Get the hostname of the current Zabbix host try: # Get all resources (VMs and containers) from the cluster cmd = ["pvesh", "get", "/cluster/resources", "--type", "vm", "--output-format", "json"] result = subprocess.run(cmd, capture_output=True, text=True, check=True) resources = json.loads(result.stdout) except subprocess.CalledProcessError as e: print(f"Error executing pvesh: {e}", file=sys.stderr) print(f"Stderr: {e.stderr}", file=sys.stderr) sys.exit(1) except json.JSONDecodeError as e: print(f"Error decoding JSON from pvesh: {e}", file=sys.stderr) sys.exit(1) lld_data = [] for res in resources: if res.get('type') == 'vm': # Check if the VM's assigned node matches the current Zabbix host's hostname # 'node' property indicates the node the VM is configured to run on (or last ran on) # For running VMs, it should reflect the active node. if res.get('node') == current_node_hostname: lld_data.append({ "{#VMID}": res.get('vmid'), "{#VMNAME}": res.get('name', f"VM_{res.get('vmid')}"), "{#VMSTATUS}": res.get('status', 'unknown') }) print(json.dumps({"data": lld_data}, indent=4)) if __name__ == "__main__": main()Explanation of the script:
- It uses
pvesh(Proxmox shell command) to query the Proxmox API. - It reads the
PVE_TOKEN_IDandPVE_TOKEN_SECRETfrom a secure file. - It gets the
current_node_hostnameof the machine where the script is run. - It iterates through all discovered VMs (
/cluster/resources --type vm). - Crucially, it checks
if res.get('node') == current_node_hostname:to ensure only VMs residing on the current node are included in the LLD output.
Securely store your Proxmox API token:
Create
/etc/zabbix/proxmox_api_token.conf(or similar):PVE_TOKEN_ID=your_user@pve!your_token_id PVE_TOKEN_SECRET=your_secret_stringSet permissions:
sudo chmod 700 /etc/zabbix/scripts/ sudo chmod 755 /etc/zabbix/scripts/proxmox_vm_discovery_local.py sudo chmod 600 /etc/zabbix/proxmox_api_token.conf sudo chown -R zabbix:zabbix /etc/zabbix/scripts/ sudo chown zabbix:zabbix /etc/zabbix/proxmox_api_token.conf(Make sure
zabbixis the user running the Zabbix agent process).2. Configure Zabbix Agent UserParameter
Add a
UserParameterto your Zabbix agent configuration file (/etc/zabbix/zabbix_agentd.confor a file inzabbix_agentd.d/):UserParameter=proxmox.vm.discovery_local,/etc/zabbix/scripts/proxmox_vm_discovery_local.pyAfter adding this, restart the Zabbix agent service on all your Proxmox nodes:
sudo systemctl restart zabbix-agentTest the UserParameter:
You can test it from the Zabbix server:
zabbix_get -s <your_proxmox_node_ip> -k proxmox.vm.discovery_localYou should see JSON output containing only the VMs running on that specific Proxmox node.
3. Modify Your Zabbix Proxmox Template
Now, you need to update the template you're using in Zabbix.
-
Navigate to Templates: In Zabbix UI, go to Configuration > Templates.
-
Find your Proxmox Template: Select the template you're using (e.g., "Proxmox Virtual Environment by Zabbix agent" or a custom one).
-
Go to Discovery Rules: Click on "Discovery rules".
-
Edit the VM Discovery Rule:
- Find the discovery rule related to Proxmox VMs (it might be named something like "Discover Proxmox VMs" or similar).
- Change its Key from whatever it currently is (e.g.,
proxmox.vm.discovery) to your new UserParameter key:proxmox.vm.discovery_local. - Important: Ensure the "Liveness check" type (if any) or any other part of the rule does not conflict. The
Type of informationshould remain "JSON".
-
Update Item Prototypes (if necessary):
- Under the edited Discovery Rule, go to "Item prototypes".
- Verify that the keys for items like
proxmox.vm.status[{#VMID}]still use the{#VMID}and{#VMNAME}macros that your script outputs. Your script uses these exact macros, so they should be compatible. - Ensure the actual item keys the template uses (e.g.,
proxmox.vm.status[{#VMID}]) still align with how your Zabbix Agent is configured to get that data (e.g.,UserParameter=proxmox.vm.status[*],/path/to/script proxmox.vm.status $1). If the original template usespveshdirectly via a script, make sure that script is also API token aware.
-
Update Trigger Prototypes:
- Under the Discovery Rule, go to "Trigger prototypes".
- No changes should be needed here, as the trigger logic typically refers to the item prototypes (e.g.,
last(/Proxmox Template/proxmox.vm.status[{#VMID}])=0). Since the underlying item will now only be discovered on the correct host, the trigger will only fire on that host.
4. Force Discovery and Verify
-
Wait for automatic discovery (usually an hour by default), or manually force it:
- Go to Configuration > Hosts.
- Select your Proxmox hosts.
- From the "Mass update" dropdown, choose "Check now" for "Host availability" and "Discovery" if available (sometimes it just means waiting for agent data).
- Alternatively, you can go into each host, click "Triggers" or "Items" and then "Apply" the template again.
-
Verify:
- Check Configuration > Hosts > Items for each Proxmox node. You should now only see VM-related items for the VMs running on that specific node.
- Check Configuration > Hosts > Triggers similarly.
- Check Monitoring > Latest data for a specific Proxmox host. Filter by VM-related items to confirm only local VMs are monitored.
- Most importantly, simulate a VM going down (e.g., stop one not running on the Zabbix server's own Proxmox node if you're testing from there). You should only receive one alert from the correct Proxmox host.
Alternative (Less Recommended): Trigger-Level Filtering
While the LLD method is best, you could also modify the trigger conditions. This is less efficient because all hosts still discover and collect data for all VMs, but triggers simply don't fire.
How it works:
You would need an additional item prototype that tells you which Proxmox node a VM is currently running on. Let's say you have an item prototype
proxmox.vm.current_node[{#VMID}]that returns the hostname of the node the VM is running on.Then, you would modify your trigger prototype to include a condition like this:
({TEMPLATE_NAME:proxmox.vm.status[{#VMID}].last()}=0) and ({TEMPLATE_NAME:proxmox.vm.current_node[{#VMID}].last()} = "{HOST.HOST}"){HOST.HOST}: This Zabbix macro resolves to the hostname of the current Zabbix agent.=0: Assuming 0 means the VM is down/problematic. Adjust according to your item's value map.
Downsides of Trigger-Level Filtering:
- Resource Consumption: All Proxmox nodes are still collecting data for all VMs in the cluster, even if they don't host them. This creates unnecessary items, history, and evaluation load.
- Complexity: Requires an additional item prototype and more complex trigger expressions.
Conclusion:
The LLD filtering method is the cleanest and most efficient way to achieve your goal of single, accurate alerts for Proxmox VMs. It ensures that Zabbix only monitors what's relevant to each host, reducing noise and resource usage.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.