In Elixir, is there a library (standard lib or hex package) that allows me to: - Start a…
- User
In Elixir, is there a library (standard lib or hex package) that allows me to:
- Start a given executable with commandline args and modified ENV variables (e.g. "PORT=1138")
- Stop the executable again if it is still running (as if pressing CTRL+C in a terminal)
- Read STDOUT and STDERR of the executable while it is running
Assistant
Okay, you're describing a common pattern in Elixir/Erlang for interacting with external programs that run for a potentially long time and produce streaming output.
The core mechanism in the Elixir/Erlang standard library for this is the
Portmodule.Portallows the BEAM VM to communicate with external processes (written in any language) by opening a communication channel. The BEAM process that opens the port becomes the "controlling process" and receives messages from the external process's standard output and standard error, and can send messages to its standard input.While
Portis the low-level tool, it's typically wrapped in an OTP process (like aGenServer) to manage its lifecycle, handle incoming data asynchronously, and provide a clean API.There isn't one single "library" function like
System.run_and_stream, but usingPortwithin aGenServeris the standard Elixir pattern.Here's how you'd typically implement this using the standard library (
PortandGenServer):Conceptual Implementation using
PortandGenServerYou'd create a
GenServermodule that does the following:- Starts the Process: In its
initcallback, it usesPort.open/2with the:spawn_executableoption, providing the command, arguments, and the:envoption for environment variables. - Receives Output: It implements
handle_info/2to listen for messages from the port. Data from STDOUT and STDERR (which are often merged by default unless configured otherwise) arrive as{:data, {port_identifier, binary_data}}messages. - Handles Termination: It also listens for
{:exit, {port_identifier, reason}}messages, which indicate the external process has terminated (either normally or due to an error/signal). - Provides a Stop Mechanism: It exposes a client function (e.g.,
stop/1) which callsGenServer.call/2to send a message to the GenServer. The GenServer's corresponding callback (e.g.,handle_call/3) then callsPort.close/1on the port identifier.Port.close/1attempts to shut down the external process gracefully (often equivalent to sending SIGINT or closing pipes). - Manages Lifecycle: Sets
Process.flag(:trap_exit, true)ininitso that if the controlling process crashes, the GenServer'sterminate/2callback is called, allowing it to close the port cleanly.
Example GenServer Implementation
defmodule ExternalProcessAgent do use GenServer # Client API @doc """ Starts a supervised external process agent. `command`: The executable path or name (if in PATH). `args`: List of strings for command-line arguments. `env`: List of `{"KEY", "value"}` tuples for environment variables. `opts`: GenServer options and `:on_output` callback. `:on_output` should be an MFA (Module, Function, Args) or a function that accepts one argument (the output binary). Defaults to printing to IO. """ def start_link(command, args \\ [], env \\ [], opts \\ []) when is_list(args) and is_list(env) do GenServer.start_link(__MODULE__, {command, args, env, opts}, opts) end @doc """ Stops the external process managed by the agent. """ def stop(pid) when is_pid(pid) do GenServer.call(pid, :stop, :infinity) # Use a reasonable timeout in production end # GenServer Callbacks @impl true def init({command, args, env, opts}) do # Options for Port.open # :binary ensures output is received as binaries # :args provides command line arguments # :env provides environment variables specific to this process # :stderr_to_stdout (default) merges stderr into the same output stream as stdout port_options = [:binary, {:args, args}, {:env, env}] # Start the external process via the port case Port.open({:spawn_executable, command}, port_options) do {:ok, port} -> IO.puts("Started external process: #{command} #{Enum.join(args, " ")}") # Trap exits so terminate/2 is called if our linked parent dies Process.flag(:trap_exit, true) on_output_mfa = Keyword.get(opts, :on_output, {IO, :write, []}) {:ok, %{port: port, command: command, args: args, on_output: on_output_mfa}} {:error, reason} -> IO.error("Failed to start external process '#{command}': #{inspect(reason)}") {:stop, {:error, reason}} # Stop the GenServer if port creation fails end end @impl true def handle_info({:data, {port, data}}, state = %{port: port, on_output: on_output_mfa}) do # Received data from the external process's STDOUT/STDERR try do case on_output_mfa do {mod, fun, fun_args} -> apply(mod, fun, [data | fun_args]) func when is_function(func, 1) -> func.(data) _ -> IO.write(data) # Default end rescue e -> IO.error("Error in on_output callback: #{inspect(e)}") end {:noreply, state} end @impl true def handle_info({:exit, {port, reason}}, state = %{port: port, command: command}) do # The external process exited IO.puts("External process '#{command}' (#{inspect(port)}) exited with reason: #{inspect(reason)}") # Now that the external process is done, stop the GenServer {:stop, :normal, state} end # Handle unexpected messages from the port or elsewhere @impl true def handle_info(msg, state) do IO.warn("ExternalProcessAgent received unexpected message: #{inspect(msg)}") {:noreply, state} end @impl true def handle_call(:stop, _from, state = %{port: port, command: command}) do IO.puts("Received stop signal for '#{command}'. Closing port.") # Request the port (and thus the external process) to close. # This will typically cause the process to exit, triggering the {:exit, ...} handle_info. Port.close(port) # Reply immediately, the {:exit, ...} message will handle the actual GenServer termination. {:reply, :ok, state} end @impl true def handle_call(request, _from, state) do IO.warn("ExternalProcessAgent received unexpected call: #{inspect(request)}") {:reply, :error, state} # Or {:noreply, state} depending on desired behavior end @impl true def terminate(reason, %{port: port, command: command}) do # This is called when the GenServer is shutting down (e.g., due to stop call, exit message) IO.puts("Terminating agent for '#{command}' (#{inspect(port)}) with reason: #{inspect(reason)}. Ensuring port is closed.") # Ensure the port is closed. Port.close is idempotent. Port.close(port) :ok end endHow to Use It:
# Example 1: Run a simple command that exits {:ok, pid1} = ExternalProcessAgent.start_link("echo", ["Hello, world!"], [], name: :my_echo_agent) # Output will be printed via the default IO.write callback # Wait a bit for it to finish (it's very fast) Process.sleep(100) # Check if the process is still alive IO.puts("Echo process alive? #{Process.alive?(pid1)}") # Example 2: Run a process that stays alive and produces output # (Requires `grep`, adjust command for Windows like `findstr`) {:ok, pid2} = ExternalProcessAgent.start_link("grep", ["--line-buffered", "keyword"], [], # Custom callback to process output - maybe log it on_output: fn data -> Logger.info("Grep output: #{inspect(data)}") end, name: :my_grep_agent ) # Send some data to grep's STDIN (optional - this agent doesn't handle STDIN by default) # To send to STDIN, you'd add a handle_cast or handle_call to the GenServer # that uses Port.command(port, {:command, data}). # Let it run for a while Process.sleep(5000) # Stop the process IO.puts("Stopping grep process...") :ok = ExternalProcessAgent.stop(pid2) # Wait for it to terminate Process.sleep(100) IO.puts("Grep process alive? #{Process.alive?(pid2)}") # Example 3: Trying to start a non-existent command IO.puts("Trying to start invalid command...") {:error, reason} = ExternalProcessAgent.start_link("non_existent_command_12345", [], []) IO.puts("Start failed as expected: #{inspect(reason)}")Explanation of Key Features:
Port.open({:spawn_executable, command}, options): This is the core function to start the external process. Using{:spawn_executable, command}is generally safer than{:spawn, "command args"}as it avoids shell injection issues.:argsand:envoptions: Directly support passing command-line arguments and environment variables.:binaryoption: Ensures output data is received as Elixir binaries, which is usually what you want.handle_info({:data, {port, data}}, state): This callback receives the output data asynchronously as the external process writes to its standard output or standard error. You can define a custom function (on_outputoption) to process this data (e.g., parse it, log it, send it to another process).handle_info({:exit, {port, reason}}, state): This callback is triggered when the external process terminates. Thereasonindicates how it exited (e.g.,:normal, exit code). The GenServer then stops itself.Port.close(port): This is the standard way to signal the external process to shut down. For many programs, closing their input/output pipes or receiving a specific signal (like SIGINT on Unix) causes them to exit cleanly, similar to hitting CTRL+C.GenServer: Provides the necessary structure to hold the port state, manage the asynchronous messages, and handle the lifecycle gracefully (start, stop, termination on crash).
Hex Packages?
While there isn't a single standard Hex package specifically for this streaming, non-waiting pattern that completely replaces using
PortandGenServer, many libraries that interact with external processes use Ports internally.System.cmd/3: This is part of the standard library, but it's primarily for running a command synchronously and waiting for it to finish, collecting all output at the end. It does not stream output while running.porcelain: A popular library for running shell commands. It can be used for asynchronous execution and capturing output, but it might focus more on waiting and capturing all output rather than streaming chunk-by-chunk data while the process is running without waiting. You would still likely need to manage its lifecycle if the process is long-running.
For the specific requirements of streaming output while running and graceful stopping without necessarily waiting for the process to finish, the
Portmodule wrapped in aGenServeras shown above is the most direct and idiomatic Elixir/Erlang solution. It's the building block that any higher-level library for this purpose would likely use anyway.So, yes, the capability exists within the standard library (
Port,GenServer), and the example above provides a robust way to use it.- Starts the Process: In its
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.