How to download complete XML records from PubMed and extract data
My first PubMed script (An R Script to Automatically download PubMed
Citation Counts By Year of Publication) extracted yearly counts for
any number of search strings, by using PubMed’s E-utilities.
Specifically, it’s using the
esearch-function, which will report the
number of hits for your search and/or the articles PMIDs. This method is
very reliable and fast if you’re only interested in the number of hits
for a given query. However, PubMed’s E-utilities have a lot more
features than that, some of which I will use in this article to download
complete article records in XML.
What’s cool about
esearch is that you can tell it to save a history of
the articles found by your query, and then use another function called
efetch to download that history. This is done by adding
&usehistory=y to your search, which will generate this XML (in
addition to some other XML-tags):
Once we have extracted the WebEnv string, we just tell PubMed’s efetch
to send us the articles saved in WebEnv. There’s one complication,
though. PubMed “only” allows us to fetch 10 000 articles in one go,
therefore my code includes a loop that will batch download the data, and
paste it together in order to create valid XML-code. The XML cutting and
pasting is done with
gsub, since the unparsed XML-data is just a long
string. It’s not the most beautiful solution, but it seems to work.
Now that all XML-data is saved in one object, we just need to parse it an extract whatever PubMed field(s) we’re interested in. I’ve included a function that will parse the XML-code and extract journal counts, although you could use the same method to extract any field.
These two graphs were created by using the following 3 queries (notice
that I use single-quotes inside my query). This script does not have the
functionality to download different queries automatically for you, so I
ran my three searches individually. The R code for
extractJournal() are at the end of this article.
I needed to reshape my data a bit, and combine it into one object,
before I used
ggplot2 to make the graphs. I did it like this:
Now that I have all my top 20 data in one object in the long format, the
ggplot2 code is pretty simple.
To check the reliability of my method I compared the number of extracted journals to the total number of hits. These are the numbers:
2010: 1487 / 1488 = 0.999328
2011: 1488 / 1493 = 0.996651
All years: 14345 / 14354 = 0.999373
Since the error is so low, I didn’t bother to check why some journals were left out. My guess is, that they were missing in the original data as well.
Update 2013-08-17: Moved the script to github and fixed the broken batch procedure. It should be more stable now.
Published April 27, 2012 (View on GitHub)