{"id":468,"date":"2008-05-19T13:25:00","date_gmt":"2008-05-19T18:25:00","guid":{"rendered":"http:\/\/www.juixe.com\/techknow\/index.php\/2008\/05\/19\/using-hpricot\/"},"modified":"2008-05-19T13:25:00","modified_gmt":"2008-05-19T18:25:00","slug":"using-hpricot","status":"publish","type":"post","link":"http:\/\/juixe.com\/techknow\/index.php\/2008\/05\/19\/using-hpricot\/","title":{"rendered":"Using Hpricot"},"content":{"rendered":"

Hpricot<\/a> is a HTML parser for the Ruby programming language. With Hpricot you can scan and scape a HTML document. To illustrate how to use Hpricot i’ll write a list the code of a short script I recently wrote. The script grabs all the links for the past week from A Rubyist Railstastic Adventure<\/a>, a tumblelog.<\/p>\n

The general structure of the HTML used by the web page that I will be scraping is something like the following.<\/p>\n

[source:html]
\n<div class=”post”>
\n <div class=”date”>
\n Sun
\n <em>
\n May
\n <big>18<\/bi>
\n <\/em>
\n <\/div><\/p>\n

<div class=”link”>
\n <a href=”http:\/\/www.juixe.com” class=”link”>Juixe TechKnow<\/a>
\n <\/div>
\n<\/div><\/p>\n

[\/source]<\/p>\n

One thing to note about the HTML produced by the site we will scape is that the date is optional in the post. The date is only displayed once for a day, so some posts don’t have a given date. Also, there are several other types of posts such as quotes, images, etc. We are only interested in posts with links. Again, the Ruby\/Hpricot script will only gather the links for the past week.<\/p>\n

[source:ruby]
\nrequire ‘rubygems’
\nrequire ‘hpricot’
\nrequire ‘open-uri’
\nrequire ‘parsedate’<\/p>\n

# Convert days to number to seconds
\ndef days_to_sec(days)
\n secs = days.to_i
\n secs *= 24
\n secs *= 60
\n secs *= 60
\n secs
\nend<\/p>\n

# pretty print the link in a list
\ndef print_link(link)
\n print ” <li>”,
\n “<a href=’#{link.attributes[‘href’]}’>”,
\n “#{link.inner_html.strip}”,
\n “<\/a>”,
\n “<\/li>\\n”
\nend<\/p>\n

def get_links(doc)
\n curr_date = Time.new
\n (doc\/”div.post”).each do |post|
\n post_date_elem = (post\/”div.date\/em”)
\n date = post_date_elem.inner_html.strip<\/p>\n

# Parse the date of the post
\n if date != “”
\n date_day = (post_date_elem\/”big”).text
\n date_mon = nil
\n date.each_line do |line|
\n date_mon = line.strip if date_mon.nil?
\n break if date_mon.nil?
\n \t end
\n date_str = “#{date_mon} #{date_day}, #{$week_ago.year}”
\n \t data = ParseDate.parsedate date_str
\n \t curr_date = Time.local data[0], data[1], data[2]
\n end<\/p>\n

# Stop if already looking past one week
\n break if curr_date < $week_ago<\/p>\n

# Handle all links in post
\n (post\/”a.link”).each do |link|
\n print_link link
\n end
\n end<\/p>\n

if curr_date > $week_ago
\n next_page = Hpricot(open($rubyist_next))
\n get_links next_page
\n end
\nend<\/p>\n

# The Rubyist home page to be scraped
\n$rubyist_home = “http:\/\/rubyist.tumblr.com\/”
\n$rubyist_next = “http:\/\/rubyist.tumblr.com\/page\/2”
\n# Scrape one weeks worth of links
\n$week_ago = Time.new – days_to_sec(7)<\/p>\n

# Run the script
\ndoc = Hpricot(open($rubyist_home))
\nget_links doc
\n[\/source]<\/p>\n

Technorati Tags: ruby<\/a>, rubyist<\/a>, hpricot<\/a>, html<\/a>, scape<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Hpricot is a HTML parser for the Ruby programming language. With Hpricot you can scan and scape a HTML document. To illustrate how to use Hpricot i’ll write a list the code of a short script I recently wrote. The script grabs all the links for the past week from A Rubyist Railstastic Adventure, a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[13,22,3],"tags":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p902K-7y","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/posts\/468"}],"collection":[{"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/comments?post=468"}],"version-history":[{"count":0,"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/posts\/468\/revisions"}],"wp:attachment":[{"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/media?parent=468"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/categories?post=468"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/juixe.com\/techknow\/index.php\/wp-json\/wp\/v2\/tags?post=468"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}