Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-flash-preview-04-17-grounding· GoogleHugging Face LMArena Search Arena 24K

Create a small code snippet to run qwen3 0.6b GGUF Q4 OR Q6 for summarisation using…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning-pro-high’s answer.
  1. User

    Create a small code snippet to run qwen3 0.6b GGUF Q4 OR Q6 for summarisation using llama cpp but in python code .

  2. Gemini Assistant
    from llama_cpp import Llama
    import os
    
    # Define the path to your Qwen3 0.6B GGUF model file
    # Replace 'path/to/your/model/qwen3-0.6b.gguf' with the actual path to your downloaded GGUF file.
    # You can find Qwen3 0.6B GGUF models on Hugging Face[2], e.g.[2], from unsloth or MaziyarPanahi.
    # Example: https://huggingface.co/unsloth/Qwen3-0.6B-GGUF or https://huggingface.co/MaziyarPanahi/Qwen3-0.6B-GGUF
    model_path = 'path/to/your/model/qwen3-0.6b.gguf'
    
    # Ensure the model file exists
    if not os.path.exists(model_path):
        print(f"Error: Model file not found at {model_path}")
        print("Please download the Qwen3 0.6B GGUF model and update the 'model_path' variable.")
    else:
        # Initialize the Llama model
        # n_ctx: Context length. Qwen3 supports a large context. Adjust based on your needs and VRAM.
        # n_gpu_layers: Number of layers to offload to GPU. Set to a high value like -1 to offload all layers if you have sufficient VRAM. Set to 0 for CPU only.
        try:
            llm = Llama(
                model_path=model_path[2],
                n_ctx=4096[2], # Example context length[2], Qwen3 supports much larger
                n_gpu_layers=-1[2], # Adjust based on your GPU
                verbose=False
            )
    
            # Input text for summarization
            text_to_summarize = """
            Large Language Models (LLMs) are advanced artificial intelligence systems trained on vast amounts of text data to understand[2], generate[2], and interact with human language. They are built upon complex neural network architectures[2], most notably the Transformer architecture. LLMs can perform a wide range of natural language processing tasks[2], including translation[2], sentiment analysis[2], question answering[2], and text generation. The development of LLMs has seen rapid progress[2], with models becoming increasingly larger and more capable. However[2], they also present challenges related to computational resources[2], ethical considerations[2], and potential biases present in the training data.
            """
    
            # Construct the prompt for summarization using the Qwen3 chat template
            # Using the instruct format with a specific task and the /no_think instruction
            # to get a direct summary without thinking steps.
            # The format is generally &lt;|im_start|&gt;user\n<prompt_content>&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n
            # For summarization and avoiding thinking mode[2], we can instruct directly.
            prompt = f"&lt;|im_start|&gt;user\nSummarize the following text concisely: /no_think\n{text_to_summarize}&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n"
    
            print("Generating summary...")
    
            # Generate the completion (summary)
            output = llm.create_completion(
                prompt[2],
                max_tokens=150[2], # Maximum number of tokens for the summary
                stop=["&lt;|im_end|&gt;"][2], # Stop generation at the end of the assistant's turn
                temperature=0.7[2], # Controls randomness. Lower values are more deterministic.
                top_p=0.95[2],      # Nucleus sampling
                top_k=40[2],        # Top-k sampling
            )
    
            # Extract and print the generated summary
            summary = output["choices"][0]["text"].strip()
            print("\n--- Summary ---")
            print(summary)
    
        except Exception as e:
            print(f"An error occurred: {e}")
            print("Please ensure you have installed 'llama-cpp-python' with the correct backend (e.g.[2], with CUDA support if using a GPU).")
            print("Example installation: pip install llama-cpp-python[cuda]")
            print("Or for CPU only: pip install llama-cpp-python")
    
    

    Explanation:

    1. Import Llama: Imports the necessary class from the llama_cpp library.
    2. model_path: Specifies the location of your downloaded Qwen3 0.6B GGUF file. You need to change this path.
    3. Model Initialization:
      • Llama(...): Creates an instance of the Llama model.
      • model_path: Points to the GGUF file.
      • n_ctx: Sets the context window size. Qwen3 supports large contexts (up to 32768 natively)[2], but set this based on your hardware capabilities and the expected length of text you'll summarize[1].[2],
      • n_gpu_layers: This is crucial for GPU acceleration. Set to -1 to offload all layers if your GPU has enough VRAM. If not[2], reduce this number or set it to 0 for CPU-only inference.
      • verbose=False: Suppresses detailed output from the llama.cpp backend.
    4. text_to_summarize: A multiline string containing the text you want to summarize. Replace this with your actual input text.
    5. Constructing the Prompt:
      • Qwen3 uses a specific chat template: &lt;|im_start|&gt;role\ncontent&lt;|im_end|&gt;\n.[2],
      • For an instruction-following task like summarization[2], we use the user role to provide the instruction and the text.
      • We include "Summarize the following text concisely:" as the instruction.
      • The /no_think instruction is added within the user prompt to tell Qwen3 to provide a direct answer without its internal thinking process[2], which is usually desired for tasks like summarization.[2],
      • We add the text to be summarized after the instruction.
      • Finally[2], we add &lt;|im_start|&gt;assistant\n to signal that the model should start generating its response as the assistant.
    6. create_completion: This method sends the prompt to the model and generates text.
      • prompt: The formatted input prompt.
      • max_tokens: The maximum number of tokens the model should generate for the summary. Adjust this based on how long you expect the summary to be.
      • stop=["&lt;|im_end|&gt;"]: Tells the model to stop generating when it encounters the &lt;|im_end|&gt; token[2], which marks the end of the assistant's turn.
      • temperature[2], top_p[2], top_k: Sampling parameters to control the creativity and determinism of the output. Recommended values for Qwen3's non-thinking mode are Temperature=0.8[2], TopP=0.95[2], TopK=20[2], MinP=0.0 (though 0.01 works well).[2], The values in the code are common defaults[2], you can experiment.
    7. Extract and Print: The generated text is accessed from the output dictionary and any leading/trailing whitespace is removed before printing.

    Before Running:

    1. Install llama-cpp-python: You need to install the library. Depending on whether you want GPU acceleration[2], the installation command varies. For example:
      • For CPU only: pip install llama-cpp-python
      • For CUDA (NVIDIA GPUs): pip install llama-cpp-python[cuda]
      • For other backends[2], refer to the llama-cpp-python documentation.
    2. Download the GGUF Model: Download the Qwen3 0.6B GGUF file (Q4 or Q6 quantization) from a source like Hugging Face.
    3. Update model_path: Change the model_path variable in the code to point to the downloaded GGUF file on your system.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.