how can I run a python script to check message within a azure service bus topic as…
- User
how can I run a python script to check message within a azure service bus topic as receiver on an linux machine for permanent check on the topic
Assistant
Okay, let's set up a Python script to permanently monitor an Azure Service Bus Topic subscription on a Linux machine.
Prerequisites:
- Python 3.7+: Ensure Python 3.7 or newer is installed on your Linux machine.
- pip: Python's package installer.
- Azure Service Bus Namespace, Topic, and Subscription:
- You need an existing Service Bus Namespace.
- Inside the namespace, a Topic must be created.
- A Subscription to that Topic must exist. This is what your receiver will listen to. If you don't have one, create it in the Azure portal. Each subscriber needs its own subscription to get a copy of all messages sent to the topic.
- Connection String: Get the connection string for your Service Bus Namespace.
- Go to your Service Bus Namespace in the Azure portal.
- Under "Settings," click "Shared access policies."
- Choose a policy with "Listen" permissions (and "Manage" if your script needs to create subscriptions, though for this receiver, "Listen" is enough for an existing subscription).
RootManageSharedAccessKeyhas all permissions. - Copy the "Primary Connection String" or "Secondary Connection String."
Steps:
1. Install the Azure Service Bus SDK
Open your Linux terminal and run:
pip install azure-servicebusIt's highly recommended to do this within a Python virtual environment:
python3 -m venv my-servicebus-app source my-servicebus-app/bin/activate pip install azure-servicebus # When done: deactivate2. Create the Python Script
Let's call the script
topic_listener.py.import os import asyncio import logging import time from azure.servicebus.aio import ServiceBusClient from azure.servicebus.exceptions import ServiceBusError, OperationTimeoutError, ServiceBusConnectionError # --- Configuration --- # Best practice: Use environment variables for sensitive data CONNECTION_STR = os.environ.get("AZURE_SERVICE_BUS_CONNECTION_STRING") TOPIC_NAME = os.environ.get("AZURE_SERVICE_BUS_TOPIC_NAME") SUBSCRIPTION_NAME = os.environ.get("AZURE_SERVICE_BUS_SUBSCRIPTION_NAME") # --- Logging Setup --- logging.basicConfig( level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s", handlers=[ logging.StreamHandler(), # Output to console # logging.FileHandler("servicebus_listener.log") # Optional: Output to file ] ) logger = logging.getLogger(__name__) async def process_message(msg): """ This is where you'd put your logic to handle the message. For now, we'll just log it. """ try: message_body = str(msg) logger.info(f"Received message: {message_body}") logger.info(f" Sequence Number: {msg.sequence_number}") logger.info(f" Message ID: {msg.message_id}") logger.info(f" Delivery Count: {msg.delivery_count}") # Add your custom processing logic here # e.g., save to database, call another API, etc. # If processing is successful, complete the message. # This removes it from the subscription. await msg.complete() logger.info(f"Message {msg.message_id or msg.sequence_number} completed.") except Exception as e: logger.error(f"Error processing message: {e}") # Decide what to do: # - Abandon: msg.abandon() - Puts it back in the queue for another try (increases delivery_count) # - Dead-letter: msg.dead_letter("ProcessingError", f"Reason: {e}") - Moves to DLQ # For this example, we'll let it be re-delivered if an error occurs during processing, # until it hits the max delivery count and goes to the DLQ. # Or, you could explicitly abandon it here if you want to control retries more finely. # await msg.abandon() # Example async def receive_messages_continuously(): if not all([CONNECTION_STR, TOPIC_NAME, SUBSCRIPTION_NAME]): logger.error("Missing one or more environment variables: AZURE_SERVICE_BUS_CONNECTION_STRING, AZURE_SERVICE_BUS_TOPIC_NAME, AZURE_SERVICE_BUS_SUBSCRIPTION_NAME") return logger.info(f"Starting listener for Topic: {TOPIC_NAME}, Subscription: {SUBSCRIPTION_NAME}") while True: # Keep running indefinitely try: async with ServiceBusClient.from_connection_string(conn_str=CONNECTION_STR) as client: # max_wait_time specifies how long the receiver will wait for messages before timing out # and allowing the loop to check for shutdown or other conditions. # For continuous listening, a longer wait time reduces chattiness. async with client.get_subscription_receiver( topic_name=TOPIC_NAME, subscription_name=SUBSCRIPTION_NAME, max_wait_time=30 # Wait up to 30 seconds for a message ) as receiver: logger.info("Receiver created. Waiting for messages...") async for message in receiver: await process_message(message) # Add a small delay if needed, e.g., if processing is very fast # await asyncio.sleep(0.1) except OperationTimeoutError: # This is expected if max_wait_time is set and no messages arrive. # The loop will simply continue and try to receive again. logger.debug("No messages received in the last wait cycle. Continuing to listen...") pass except ServiceBusConnectionError as e: logger.error(f"Connection error: {e}. Retrying in 60 seconds...") await asyncio.sleep(60) except ServiceBusError as e: logger.error(f"Service Bus Error: {e}. Retrying in 30 seconds...") await asyncio.sleep(30) except KeyboardInterrupt: logger.info("Receiver shutting down due to KeyboardInterrupt...") break except Exception as e: logger.error(f"An unexpected error occurred: {e}. Retrying in 60 seconds...") await asyncio.sleep(60) if __name__ == "__main__": try: asyncio.run(receive_messages_continuously()) except KeyboardInterrupt: logger.info("Application shutting down.") finally: logger.info("Listener stopped.")3. Set Environment Variables
Before running the script, set the required environment variables in your Linux terminal:
export AZURE_SERVICE_BUS_CONNECTION_STRING="your_connection_string_here" export AZURE_SERVICE_BUS_TOPIC_NAME="your_topic_name" export AZURE_SERVICE_BUS_SUBSCRIPTION_NAME="your_subscription_name"Replace the placeholder values with your actual Service Bus details.
4. Run the Script
python3 topic_listener.pyYou should see log messages indicating it's waiting for messages. When a message is sent to the topic, this script (listening on its subscription) should pick it up.
5. Making it Run Permanently (Daemonizing)
To run this script continuously and reliably on a Linux machine, you should use a process manager like
systemdor a simpler tool likenohuporscreen/tmux.systemdis the recommended way for production services.Option A: Using
nohup(Simpler, less robust)nohupallows a command to keep running even after you log out.nohup python3 /path/to/your/topic_listener.py > /path/to/your/topic_listener.log 2>&1 &nohup: Prevents the process from being killed when you log out.python3 /path/to/your/topic_listener.py: Your script.> /path/to/your/topic_listener.log: Redirects standard output to a log file.2>&1: Redirects standard error to the same log file.&: Runs the command in the background.
To stop it, you'll need to find its process ID (
ps aux | grep topic_listener.py) and thenkill <PID>.Option B: Using
systemd(Recommended for robustness)This is the standard way to manage services on modern Linux distributions.
a. Create a systemd service file: Create a file named
servicebus-listener.servicein/etc/systemd/system/:bash sudo nano /etc/systemd/system/servicebus-listener.servicePaste the following content, adjusting paths and user/group as necessary:```ini [Unit] Description=Azure Service Bus Topic Listener After=network.target [Service] User=your_linux_user # The user the script should run as Group=your_linux_group # The group for the user WorkingDirectory=/path/to/your/script_directory/ ExecStart=/path/to/your/my-servicebus-app/bin/python3 /path/to/your/script_directory/topic_listener.py # Environment variables for the script Environment="AZURE_SERVICE_BUS_CONNECTION_STRING=your_connection_string_here" Environment="AZURE_SERVICE_BUS_TOPIC_NAME=your_topic_name" Environment="AZURE_SERVICE_BUS_SUBSCRIPTION_NAME=your_subscription_name" Restart=always # Restart the service if it crashes RestartSec=10 # Wait 10 seconds before restarting StandardOutput=journal # Send stdout to journald StandardError=journal # Send stderr to journald SyslogIdentifier=servicebus-listener [Install] WantedBy=multi-user.target ``` **Important:** * Replace `your_linux_user`, `your_linux_group`, `/path/to/your/script_directory/`, and `/path/to/your/my-servicebus-app/bin/python3` (if using a venv) with your actual values. * If you are NOT using a virtual environment, `ExecStart` would just be `/usr/bin/python3 /path/to/your/script_directory/topic_listener.py` (or wherever your `python3` executable is). * You can also store environment variables in a separate file and use `EnvironmentFile=/path/to/env_file` in the service unit. This is more secure. Example `env_file`: ``` AZURE_SERVICE_BUS_CONNECTION_STRING="your_connection_string_here" AZURE_SERVICE_BUS_TOPIC_NAME="your_topic_name" AZURE_SERVICE_BUS_SUBSCRIPTION_NAME="your_subscription_name" ```b. Reload systemd daemon:
bash sudo systemctl daemon-reloadc. Enable the service (to start on boot):
bash sudo systemctl enable servicebus-listener.serviced. Start the service:
bash sudo systemctl start servicebus-listener.servicee. Check status and logs:
bash sudo systemctl status servicebus-listener.service sudo journalctl -u servicebus-listener.service -f # To follow logsf. To stop the service:
bash sudo systemctl stop servicebus-listener.serviceKey Considerations for a Permanent Check:
- Error Handling & Retries: The script includes basic retry logic for connection issues and other Service Bus errors. You might want to make this more sophisticated (e.g., exponential backoff).
- Message Completion: Crucially,
await msg.complete()is called after successful processing. If your script crashes before completing, the message will reappear on the subscription after its lock duration expires, allowing for reprocessing. Ensure yourprocess_messagelogic is idempotent (i.e., processing the same message multiple times doesn't cause adverse effects). - Dead-Letter Queue (DLQ): If a message repeatedly fails processing (e.g., it's malformed), it will eventually be moved to the subscription's Dead-Letter Queue after exceeding the
Max Delivery Countconfigured on the subscription. You should monitor the DLQ. - Logging: Robust logging (as implemented with the
loggingmodule) is essential for diagnosing issues. - Resource Management: The
async withstatements ensure that the Service Bus client and receiver are properly closed, even if errors occur. - Scalability: If you need to process a high volume of messages, you might run multiple instances of this script (ensure they connect to different subscriptions if you want each instance to get all messages, or they can connect to the same subscription if you want them to compete for messages from that one subscription, effectively load balancing). For topic/subscription, usually each distinct consumer has its own subscription.
- Security: Using environment variables for the connection string is better than hardcoding. For Azure-hosted services (like VMs, App Service, Functions), Managed Identities are the most secure way to authenticate, eliminating the need for connection strings in code. If your Linux machine is an Azure VM, consider using Managed Identity.
This setup provides a robust way to continuously listen for messages on an Azure Service Bus topic from your Linux machine.
- User
do I create a systemd service in addition or instead of a background job
Assistant
You create a
systemdservice instead of relying on simpler background job methods likenohup your_script.py &or running it in ascreen/tmuxsession for a truly permanent and robust check.Here's why:
Simple Background Jobs (e.g.,
command &,nohup command &)command &: Runs the command in the background. If you close the terminal session that started it, the process will usually be terminated (receive a SIGHUP signal).nohup command &:nohup(No Hang Up) makes the command immune to the SIGHUP signal, so it will continue running even if you close the terminal. Output is typically redirected tonohup.out.screen/tmux: These are terminal multiplexers. You can start your script in ascreenortmuxsession, detach from the session, and it will keep running. You can reattach later.
Limitations of Simple Background Jobs for Permanent Services:
- No Automatic Restart: If your Python script crashes due to an unhandled exception or if the machine reboots, it will not automatically restart. You'd have to manually start it again.
- No Monitoring: There's no built-in mechanism to easily check if the process is still running or to get its status in a standardized way (other than
psandgrep). - No Boot-Time Startup: They don't automatically start when the Linux machine boots up. You'd need to add them to
rc.localor a cron@rebootjob, which are less robust thansystemd. - Logging: While
nohupredirects output,systemdoffers more integrated and manageable logging viajournald. - Dependency Management:
systemdcan ensure your script only starts after necessary services (like networking) are up. - Resource Control:
systemdcan manage resources (CPU, memory) for your service using cgroups. - Standardization:
systemdprovides a standard interface (systemctl start/stop/status/enable/disable) for managing all services on the system.
Systemd Service
systemdis an init system and service manager for Linux. It's designed to:- Start services at boot time.
- Manage service dependencies.
- Automatically restart services if they fail.
- Provide standardized commands for managing services.
- Integrate with
journaldfor centralized logging. - Control service execution environment (user, group, environment variables).
Conclusion:
For a script that needs to run permanently, reliably, and survive reboots or crashes, you should use
systemd.- You write your Python script.
- You create a
systemdservice unit file that tellssystemdhow to run and manage your Python script. systemdthen takes over the responsibility of starting, stopping, and restarting your script.
You do not do this:
# INCORRECT for a permanent service managed by systemd sudo systemctl start my-service # and then also try to nohup the script inside the ExecStartor
# INCORRECT: Don't manually nohup and then try to make systemd manage that nohup'd process nohup python3 my_script.py & # ...and then try to write a systemd unit for this already running process.Instead, the
ExecStart=line in yoursystemdservice file will directly invoke your Python script (e.g.,/usr/bin/python3 /path/to/your/script.py).systemditself handles running it in the "background" (as a daemon) and managing its lifecycle.In short:
systemdis the robust, standard way to run daemons/services permanently on modern Linux systems. You use it instead of manual backgrounding techniques for this purpose.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.