Extract a pattern from the output of curl

Posted by allentown on Stack Overflow See other posts from Stack Overflow or by allentown
Published on 2010-04-25T04:14:29Z Indexed on 2010/04/25 4:23 UTC
Read the original article Hit count: 436

Filed under:

regex

|

curl

I would like to use curl, on the command line, to grab a url, pipe it to a pattern, and return a list of urls that match that pattern.

I am running into problems with greedy aspects of the pattern, and can not seem to get past it. Any help on this would be apprecaited.

curl http://www.reddit.com/r/pics/ | grep -ioE "http://imgur\.com/.+(jpg|jpeg|gif|png)"

So, grab the data from the url, which returns a mess of html, which may need some linebreaks somehow replaced in, onless the regex can return more than one pattern in a single line. The patter is pretty simple, any string that matches...

starts with http://imgur.com/
has A-Z a-z 0-9 (maybe some others) and is so far, 5 chars long, 8 should cover it forever if I wanted to limit that aspect of the patter, which I don't
ends in a .grraphic_file_format_extention (jpg, jpeg, gif, png)

Thats about it, at that url, with default settings, I should generally get back a good set of images. I would not be objectionable to using the RSS feel url for the same page, it may be easier to parse actually.

Thanks everyone!

© Stack Overflow or respective owner

Related posts about regex

Find multiple regex in each line and skip result if one of the regex doesn't match

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a list of variables: variables = ['VariableA', 'VariableB','VariableC'] which I'm going to search for, line by line ifile = open("temp.txt",'r') d = {} match = zeros(len(variables)) for line in ifile: emptyCells=0 for i in range(len(variables)): regex = r'('+variables[i]+r')[:|=|\(](-… >>> More
OWASP Regex Repository: Is this regex correct?

as seen on Stack Overflow - Search for 'Stack Overflow'
I was looking at the regular expression for validating various data types from the (OWASP Regex Repository). One of the regular expressions in there is called safetext and looks like: ^[a-zA-Z0-9\s.\-]+$ My first question is: Is this regular expression correct? complementary question If this… >>> More
Make a Perl-style regex interpreter behave like a basic or extended regex interpreter

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a tool to help students learn regular expressions. I will probably be writing it in Java. The idea is this: the student types in a regular expression and the tool shows which parts of a text will get matched by the regex. Simple enough. But I want to support several different regex… >>> More
JS regex isn't matching, even thought it works with a regex tester

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm writing a piece of client-side javascript code that takes a function and finds the derivative of it, however, the regex that's supposed to match with the power rule fails to work in the context of the javascript program, even though it sucessfully matches when it's used with an independent regex… >>> More
c# RegEx with "|"

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to be able to check for a pattern with | in them. For example an expression like d*|*t should return true for a string like "dtest|test". I'm no regex hero so I just tried a couple of things, like: Regex Pattern = new Regex("s*\|*d"); //unable to build because of single backslash Regex Pattern… >>> More

Related posts about curl

iPhone Curl Left and Curl Right transitions

as seen on Stack Overflow - Search for 'Stack Overflow'
I am looking for a way to do a UIViewAnimationTransitionCurlUp or UIViewAnimationTransitionCurlDown transition on the iPhone but instead of top to bottom, do it from the left to right (or top/bottom in landscape mode). I've seen this asked aroud the internet a few times but none sems to get an answer… >>> More
PHP Curl and Curl

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi , I am able to send a get request using PHP Curl . But the same thing when i try from command line in Linux (/usr/bin/curl ) I am unable to do so. Please find below my PHP curl that is working $url = "http://172.20.22.26"; $headers = array("Host: 172.20.22.26", "User-Agent:… >>> More
php, curl , php curl , multipart/form-data , upload picture redirect

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to upload some pictures using php cURL on a classified ad website .I think that I set all the parameters properly but I see that there is a kind of redirect after I post the picture . The issue is that the url where I'm getting redirected gives 404 error instead to return the html that… >>> More
Allow Incoming Responses from Curl On Ubuntu 11.10 - Curl

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. My server is running behind a Cisco Linksys WRTN310N Router with the DD-wrt firmware Installed. In… >>> More
cURL works but PHP cURL fails to internet [migrated]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Trying to diagnose an issue using PHP to cURL to an Internet location on a RedHat Linux server. cURL is installed and working, and: <?php var_dump(curl_version()); ?> shows all the correct information in the output. The issue is I can use PHP to cURL to localhost on the box itself, but… >>> More