Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Scraping data from National weather services

Hi I want to get the latest weather data from the link below, meaning the very first row of data, I am trying to scrape the data but have not been successful in doing so. Can anyone please help, or if you know any better way point me to that direction thanks!

this is what i have,

$data = file_get_contents('http://w1.weather.gov/obhistory/KIKV.html');

$regex = '/<td>(.+?) </td>/';

preg_match($regex,$data,$match);

var_dump($match);

echo $match[1];

i get an error on '/(.+?) /' this line that "Unknown modifier 't'" ? Any suggestion

1 Answer

Although parsing html with a regex is usually a very bad idea, the error you are receiving, is caused by the / character. You are using that character as the delimiter for your pattern, so you need to escape it or use another character:

$regex = '/<td>(.+?) <\/td>/';

or

$regex = '#<td>(.+?) </td>#';

Source:http://stackoverflow.com/questions/11943420/scraping-data-from-national-weather-services

Advantages of Medical Records Scanning

Medical records contain important and sensitive information regarding patients, such as birth and family records, illness, and other personal matters. Most hospitals have a certain section where doctors keep these files. They process many papers every day so they have to organize everything to avoid mistakes. There are instances, however, when calamities and accidents strike that lead to the loss of files. Papers are hard to recover once they get wet or burned. This is why many hospitals and medical facilities look for alternatives in keeping medical records.

Technology lets people make digital copies, which is useful for keeping different types of files. This adds to the flexibility of medical records as you can save photos and videos unlike the traditional word texts. Many hospitals use Electronic Medical Records or EMR to convert old documents to a digital format. This reduces the clutter in record sections as you only need a computer to keep different files. It also improves the efficiency of their staff as they do not have to spend significant time looking for a specific record.

Here are some of the advantages of medical records scanning:

Saves Space

Most hospitals need an entire room to keep all the records of their patients. Digital records, on the other hand, only require a corner of the room. Digital copies remove all the bulk in an area and create additional space. It is like shrinking an entire room to a hard drive by using medical records scanning. This makes the area look bigger and spacious. People can even use the new area for a more beneficial purpose.

Security

A disadvantage of keeping physical records is the risk of theft. The hospital is a busy place and many people go in and out every day. You cannot monitor everyone so it is hard to find the real culprit. This might cause information leak, exposing the patient's confidential information. The benefit of having digital copies is it has a system that monitors people who access it. It records what documents you take, edit, and delete. This increases the security and offers a peace of mind to your patients. You do not have to worry about losing the files either because they have a backup, making file recovery easy.

Lower Labor Costs

Many devices can perform tasks that require many people. This reduces the need of employing and paying more people to do the same job. It also saves time because you do not have to screen all the applicants. You can allot the money you save from labor costs for another thing.

Fewer Mistakes

Digital systems perform a task according to what you input so there is no room for mistakes. It ensures that all files are processed correctly. It saves you from problems due to paperwork mistakes. The hospital is a delicate environment and one mistake can risk the life of a patient. medical records scanning is a great way to improve the working process in hospitals and medical facilities. It reduces paperwork, increases security, and adds space. Try it now to see its benefits.

Source: http://ezinearticles.com/?Advantages-of-Medical-Records-Scanning&id=7466788

Web Data Extraction Services and Data Collection Form Website Pages

For any business market research and surveys plays crucial role in strategic decision making. Web scrapping and data extraction techniques help you find relevant information and data for your business or personal use. Most of the time professionals manually copy-paste data from web pages or download a whole website resulting in waste of time and efforts.

Instead, consider using web scraping techniques that crawls through thousands of website pages to extract specific information and simultaneously save this information into a database, CSV file, XML file or any other custom format for future reference.

Examples of web data extraction process include:

• Spider a government portal, extracting names of citizens for a survey
• Crawl competitor websites for product pricing and feature data
• Use web scraping to download images from a stock photography site for website design

Automated Data Collection

Web scraping also allows you to monitor website data changes over stipulated period and collect these data on a scheduled basis automatically. Automated data collection helps you discover market trends, determine user behavior and predict how data will change in near future.

Examples of automated data collection include:

• Monitor price information for select stocks on hourly basis
• Collect mortgage rates from various financial firms on daily basis
• Check whether reports on constant basis as and when required

Using web data extraction services you can mine any data related to your business objective, download them into a spreadsheet so that they can be analyzed and compared with ease.

In this way you get accurate and quicker results saving hundreds of man-hours and money!

With web data extraction services you can easily fetch product pricing information, sales leads, mailing database, competitors data, profile data and many more on a consistent basis.

Source:http://ezinearticles.com/?Web-Data-Extraction-Services-and-Data-Collection-Form-Website-Pages&id=4860417

How to Make Your WordPress Web development Successful

There are various CMS programs to choose from. Among all types of CMS programs, WordPress has got a special recognition. The CMS has been using for years. Earlier, it was using for blogging and these days, it is confidently using to create commercial, educational, trust and various types of websites. It will be right to say that the CMS has turned to be advanced and one of the most trustable CMS to manage your website content.

To make your WordPress development successful, all you need to hire a well-experienced and expert WordPress designing company. Gone are days when there were hardly any WordPress developers. These days, you can easily come across WordPress designers creating attractive and user friendly websites. A WordPress designing team promise to develop unique, high-end, user friendly and search engine friendly websites.

There are thousands of advanced plugins and themes are available in WordPress. In fact, every day to day a new and highly advanced plugin or theme has developed. The CMS is no more for creating plain websites, it can be used for the development of e-commerce sites. You can trust it to operate business websites. Plugins are available that can be customized easily. Even themes are available that can merge to create a unique and impressive theme. It is easy not only development but also changes in web portals. A few simple steps are required to follow to give WordPress websites a new look.

If you wish to have a website under budget, you can consider WordPress. Hire a WordPress Development Company in order to develop the desired website on WordPress. Explain your thoughts in front of the WordPress developing firm so that it will work on your project to give it into reals shape. The development firm will develop your website after examine your competitor websites. Analysing of plugins is possible in order to find well suitable plugins.

These readymade plugins are easy to download and doesn’t take much time to operate. A few simple steps are required to follow and to operate WordPress plugins. The development is mainly based on selection of right plugins and creating a well-suitable theme. It is important to hire a well-experienced WordPress developing company that has knowledge broadly about plugins and themes. These designing companies are able to apply plugins in the way that make organization and data management easy. At the same time, they are ready to provide website maintenance services.

Source:http://blogs.siliconindia.com/websitemaintenancecompany/Business/How-to-Make-Your-WordPress-Web-development-Successful-bid-UYxw00E991585700.html

Law of Cards: Beckett Sues Two More for Scraping Data

The recent trend in trading card cases focuses more on protecting trading card data than on trading cards themselves. Beckett has filed two more lawsuits over alleged data scraping.

Collectors Universe (CU) started this trend with lawsuits against MWP Software/Matthew Perry and SaintSoft LLC/John Kountz. In both of these suits, CU alleged the defendants copied CU data available only to paying CU subscribers, and then unlawfully (and in violation of CU's terms of service) used that data on the defendants' websites or in their own apps.

Pokémon then followed by suing the Pokellector (a website which attempts to provide a comprehensive list of all Pokémon cards, along with images of those cards) for allegedly reproducing copyrighted images of its cards without permission, trademark infringement and unlawfully rebranding the images of Pokémon cards with a Pokellector trademark.

Beckett continued the trend by bringing a trade secret lawsuit against Check Out My Cards (COMC) which, among other things, made a big deal out of COMC's "scraping" of checklists and pricing data (although COMC alleges such scraping may have been authorized).

On April 24, Beckett filed two more lawsuits focusing on pricing and checklist data. This time against Custom Plush Productions Inc's The Card Collector and Derek Miller of Miller Cards.

These two new complaints cannibalize a lot of the COMC complaint and are nearly mirror images of each other as well.

It also looks like Beckett might have let an intern from the marketing department edit these complaints. Paragraph six of both state, "Beckett is the preeminent company and most trusted source operating within the collectibles industry, and it has been since it formed in 1984."

Sounds more like a PR piece than a lawsuit, right?

For The Card Collector case, Beckett alleges The Card Collector accessed Beckett's pricing guides through the Beckett website and unlawfully "scraped" Beckett's copyrightable checklist and pricing data. Allegedly, The Card Collector now offers Beckett's card checklists, grading and pricing data on its own website and through a CD-ROM it sells to others. For that Beckett brings claims for copyright infringement, common law misappropriation, "accounting" (bring out the accountants!), unjust enrichment, a violation of Computer Fraud and Abuse Act, trespass and a Harmful Access by Computer Under Texas Penal Code.

A little bit of everything, right?

For the second new suit, Beckett alleges Miller Cards operates several websites (such as millercards.net and freebaseballcardspriceguide.com) that provide checklists and pricing guides for card collectors. Beckett contends that Miller accessed Beckett's price guides through his Beckett membership, violated the membership agreement by "scraping" this data and then unlawfully reproduced the data on his websites. It brings the identical claims against Miller Cards that it brought against the Card Collector.

Giving all of this scraping going on, I guess, the next planking or Tebowing-type meme should be the "scraping" Beckett meme. I'm looking forward to seeing your pictures on what that should look like.

Now, we've only seen Beckett's side of the story, but it looks bad for both defendants. Sure, there are issues about how much copyright protection should be afforded checklists, but these cases are not just about publicly available checklist data. The checklist and pricing information from these cases was allegedly scraped from behind a pay wall at and then distributed to others in violation of Beckett's terms of service.

Allegedly stealing something from behind a pay wall to give or sell to others just doesn’t look good.

Given this, in all likelihood, these cases will disappear (settle) quickly before we get into any real issues.

These two cases (and the COMC case) have another implication for the industry. Beckett has a template complaint that it looks like it's going to recycle again and again against scrapers. And it's policing the Internet looking for infringers. I predict then we'll see this complaint filed again this year, perhaps multiple times.

If you'd like to review a copy of these complaints, click here for the Card Collector complaint and here for the Miller Cards complaint.

The information provided in Paul Lesko's "Law of Cards" column is not intended to be legal advice, but merely conveys general information related to legal issues commonly encountered in the sports industry. This information is not intended to create any legal relationship between Paul Lesko, the Simmons Browder Gianaris Angelides & Barnerd LLC or any attorney and the user. Neither the transmission nor receipt of these website materials will create an attorney-client relationship between the author and the readers.

The views expressed in the "Law of Cards" column are solely those of the author and are not affiliated with the Simmons Law Firm. You should not act or rely on any information in the "Law of Cards" column without seeking the advice of an attorney. The determination of whether you need legal services and your choice of a lawyer are very important matters that should not be based on websites or advertisements.

Source: http://www.cardboardconnection.com/news/law-of-cards-beckett-sues-two-more-for-scraping-data