how do I extract all "http[s]://+/" from files in a tree, and print only those urls,…
- User
how do I extract all "http[s]://+/" from files in a tree, and print only those urls, from cli
Assistant
To extract all
http://andhttps://URLs from files in a directory tree and print only those URLs from the command line, you can use a combination ofgrep,find, and other Unix tools. Here's a step-by-step solution:Method 1: Using
grepwithfindfind /path/to/directory -type f -exec grep -Eo 'https?://[^[:space:]]+' {} + | sort | uniqExplanation:
-
find /path/to/directory -type f- Recursively finds all files (
-type f) in the specified directory.
- Recursively finds all files (
-
-exec grep -Eo 'https?://[^[:space:]]+' {} +grep -Eouses extended regex (-E) and prints only the matching part (-o).- The regex
https?://[^[:space:]]+matches:httpfollowed by an optionals(https?).://(the protocol separator).[^[:space:]]+(any sequence of non-whitespace characters after://).
-
sort | uniq- Sorts the URLs and removes duplicates (optional).
Method 2: Using
ripgrep(faster alternative)If you have
ripgrepinstalled (rg), you can use:rg -o 'https?://[^[:space:]]+' /path/to/directory | sort | uniqExplanation:
rg -osearches recursively and prints only the matching part.- The regex is the same as above.
Method 3: Using
ack(another alternative)If you have
ackinstalled:ack -o 'https?://[^[:space:]]+' /path/to/directory | sort | uniqNotes:
- Replace
/path/to/directorywith your target directory (e.g.,.for the current directory). - If you want to include more URL patterns (e.g.,
ftp://), modify the regex:grep -Eo '(https?|ftp)://[^[:space:]]+' - If you want to exclude certain file types (e.g., binary files), add
-Itogrep:find /path/to/directory -type f -exec grep -I -Eo 'https?://[^[:space:]]+' {} + | sort | uniq
Example Output:
http://example.com https://example.org/path https://another-site.comThis will extract all HTTP/HTTPS URLs from files in the directory tree and print them.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.