Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

User I'm trying to use async to get search results. It returns…

  1. User

    User I'm trying to use async to get search results. It returns "#object[clojure.core.async.impl.channels.ManyToManyChannel 0x2d731c6 "clojure.core.async.impl.channels.ManyToManyChannel@2d731c6"]" instead of the URL list. Here is my code.

    (ns search.08 (:require [clojure.string :as str]) (:require [clojure.core.async :as a :refer [>! <! >!! <!! go chan buffer close! thread alts! alts!! timeout sliding-buffer]]))

    ;; Revisit the problem of returning the first hits from multiple search engines, ;; this time using core.async constructs instead of futures and promises. (You’ll need to use the alts! macro). ;; Compare and contrast the two approaches.

    (def search-engine-urls {:bing {:name "Bing" :url "http://www.bing.com/search?q="} :google {:name "Google" :url "https://www.google.com/search?client=firefox-b-d&"} :yandex {:name "Yandex" :url "https://yandex.com/search/?text="} :brave {:name "Brave" :url "https://search.brave.com/search?q="} :yahoo {:name "Yahoo" :url "https://search.yahoo.com/search?p="} :mojeek {:name "Mojeek" :url "https://www.mojeek.com/search?q="}})

    (defn get-search-string-base "Converts the keyword of any of the search engines in the map to the base component of their search URL." [engine] (get-in search-engine-urls [engine :url]))

    (defn fetch-html-from-search-url "Grabs the entire HTML contents of a specified search engine page searching for a specified query." [engine query] (let [engine-url (get-search-string-base engine) url (str engine-url query)] (slurp url)))

    (def url-regex #"\b(?:https?|ftp)://[-A-Za-z0-9+&@#/=%?\-.:;]*[-A-Za-z0-9+&@#/=%\-\?]") ; Regex to get non-malformed URLs.

    ;; We want our search results to get useful content, so these are words I don't want to see in the URLs. (def unwanted-url-keywords #{"yahoo" "google" "yandex" "w3.org" "&quot" "translate.ru" "yastatic.net" "bing.net" "yimg.com" "brave.com" "bing.com" "mojeek.com"})

    (defn extract-urls "Returns a lazy sequence of all the URLs extracted from an input chunk of text." [text] (re-seq url-regex text))

    (defn fetch-and-extract-all-urls "Uses extract-url on a chunk of HTML retrieved from a search engine." [engine query] (extract-urls (fetch-html-from-search-url engine query)))

    (defn string-good-words-only-checker "Checks a string against a list of words. If any of the words on the list are found in the string, returns false." [list string] (not-any? true? (map #(str/includes? string %) list)))

    (defn filter-out-strings-containing-these "Filters a list of strings against another list of words, and removes any that are found." [filter-list coll] (filter #(string-good-words-only-checker filter-list %) coll))

    (def search-buffer (chan (sliding-buffer 5)))

    (defn search "Searches the Internet for the designated query. Engines supported: \n:bing :google :yandex :brave :yahoo :mojeek \nExample search: search :bing \"eggs\"" [engine query] (>!! search-buffer (fetch-and-extract-all-urls engine query)) (go (->> (<! search-buffer) (filter-out-strings-containing-these unwanted-url-keywords) (set) (vec) (sort))))

    (>!! search-buffer (fetch-and-extract-all-urls :bing "eggs")) (search :bing "eggs")

  2. Assistant

    It looks like your code is trying to use both the synchronous and asynchronous capabilities of core.async in Clojure, but there are a few issues present that need to be addressed.

    Let's go through the main problems in your search function:

    1. You are placing a search result into the search-buffer, which is good, but you are not returning anything to the caller that can be used to obtain the result.

    2. The search function sends a job into the channel and then immediately starts an asynchronous go-block, but search will return immediately with the go-block's channel instead of waiting for the go-block to complete. This is where you're seeing the ManyToManyChannel output.

    3. The search-buffer channel is never closed, which is not a problem per se, but in a real application, you would want to manage the lifecycle of resources such as channels.

    To make your code return the actual results, you need to restructure it so that the caller of the search function can receive the processed results. Here’s how this can be done:

    (defn search
      "Returns a channel that will contain the search results."
      [engine query]
      (let [out (chan)] ;; Create a new channel to return the search results.
        (go
          (let [html (<! (fetch-html-from-search-url engine query))]
            (->> html
                 (extract-urls)
                 (filter-out-strings-containing-these unwanted-url-keywords)
                 (set)
                 (vec)
                 (sort)
                 (>! out)))) ;; Put the result onto the out channel.
        out)) ;; Return the out channel.
    

    Now, this search function returns a channel that will eventually contain the search results. To get the results, you would use <!! or <! depending on whether you're in a synchronous or asynchronous context:

    (let [result-chan (search :bing "eggs")]
      (println (<!! result-chan))) ;; For synchronous retrieval outside of go-block.
    
    ;; or
    
    (go
      (let [results (<! (search :bing "eggs"))] ;; For asynchronous retrieval within a go-block.
        (println results)))
    

    Keep in mind that fetching URLs with slurp is a blocking operation and should be done within a thread block if you don't want to block the main thread or the core.async go-block threads.

    The fetch-html-from-search-url function should be modified to use thread:

    (defn fetch-html-from-search-url
      "Returns a channel with the HTML content from a search engine given a query."
      [engine query]
      (let [engine-url (get-search-string-base engine)
            url (str engine-url query)]
        (a/thread (slurp url))))
    

    This way, the fetching of the HTML occurs in a separate thread, and the result is placed onto a channel which is then returned. The <! operator can be used to take the result off this channel within a go block.

    Remember that when using channels, you typically don't get the results directly, but you communicate through channels, which can represent pending or completed asynchronous operations. This is why the original search function was returning a channel description when printed, since the channel itself is an object representing an asynchronous conduit for data, not the data itself.

  3. User

    I don't want the resulting URLs to be printed, I want them to be returned as a vector. Can you give the output of the code with this in mind?

  4. Assistant

    Certainly! If you want the search function to return a vector of URLs, you will need to wait for the vector to be fully constructed within the go block before returning it. In a synchronous environment, you can use <!! to block until the results are available. In an asynchronous context like inside a go block, you use <! to wait without blocking.

    Here's an example of how you might structure your search function to return a vector of URLs:

    (defn search
      "Returns a channel that will contain the search results as a vector."
      [engine query]
      (let [result-chan (chan)] ;; Create a new channel to return the search results.
        (go
          (let [urls (fetch-and-extract-all-urls engine query)
                filtered-urls (filter-out-strings-containing-these unwanted-url-keywords urls)
                url-vector (vec (sort (set filtered-urls)))] ;; Process the URL into a sorted vector.
            (>! result-chan url-vector))) ;; Pass the vector to the result channel.
        result-chan)) ;; Return the channel where the result will be available.
    

    In your main code, you can then retrieve the result vector from the search function as follows:

    (defn get-search-results [engine query]
      (let [result-chan (search engine query)]
        (<!! result-chan))) ;; Use <!! to block and wait for the channel to return the result.
    
    ;; Example synchronous usage outside of a go-block
    (let [urls (get-search-results :bing "eggs")]
      (println urls)) ;; This should print out the vector of URLs.
    

    Please remember that filter-out-strings-containing-these requires a sequence of URLs to work from, which means that fetch-and-extract-all-urls should provide a sequence.

    The function get-search-results uses <!! to retrieve the result from the channel synchronously, blocking until the search go-block has finished processing.

    This code assumes that you do not need to use core.async's sliding buffer, and that fetch-and-extract-all-urls should perform its duty synchronously or in a separate thread. If this assumption is incorrect and you want to perform the URL fetching concurrently for multiple search engines and then use alts! to get the first result, further adjustments will be necessary.

    Your fetch-html-from-search-url function should be in a go block or use core.async/thread in order to prevent blocking the main thread during HTTP requests. Your current implementation of fetch-html-from-search-url is synchronous and will block, which could cause issues with responsiveness and performance in your application.