Updated FDA Search add-on for Firefox (WOHOOO!)

Just updated the FDA Search add-on from version 0.3.0 to 0.6.0 (I just make these numbers up as I have no idea what they're supposed to mean).

It's a pretty significant upgrade, so I doubled the version #.

What can it do now?

you can highlight a word or phrase and search

• the FDA's website
• the CDC's website
• drug adverse events
• vaccine adverse events
• drug safety analysis
• clinical trials
• FDA warning letters

Also has some nifty icons added, which took me 4ever to figure out how to add. Mozilla's xul documentation is horrendous.

The FDA has an Area 51 on its web-site??

Here's something you won't see every day...it's the FDA's robots.txt file.

For the uninitiated, robots.txt is a small file placed on a web-site to indicate which pages on your website can be crawled by indexing robots (Googlebot, Yahoo! Slurp, etc.). It basically says "Hey Googlebot, you can index these pages, but stay away from those over there."

Here's the FDA's robots.txt file it is in its entirety--I've colored the parts that intrigue me.

#robots.txt file for http://www.fda.gov

#Added for Bristol-Myers on Sept 2005
User-agent: vspider  Disallow: / 

#For all other crawlers
User-agent: *
Disallow: /scripts/
Disallow: /data/
Disallow: /binn/
Disallow: /cder/test/
Disallow: /opacom/area51/ 
Disallow: /oashi/aids/listserv/
Disallow: /cdrh/ftparea/cdrh/MDR/coll/mdr/mdrcoll/
Disallow: /foi/warning_letters/d1371b.pdf
Disallow: /foi/warning_letters/archive/
Hit-rate: 30 # wait 30 seconds before starting a new URL request default=30
Visiting-hours: 23:00EDT-05:00EDT #index this site between 11PM - 5AM EDT
Concurrent-hits: 2 # limit concurrent active URLS to 2 for each index server

1. What's the deal with Bristol Myers' request to ban vspider? And why did the FDA comply with the request? From what I can tell, vspider is a personal indexing robot that can be used by anyone to index a site. Curious in CT.

2. What's going on in area51 and why can't it be indexed? I tried to look at the contents and got a "denied" error...so perhaps it holds the medical records for the little green men in Nevada.

3. Why block indexing of one specific warning letter (d1371b.pdf)? If you try to go to fda.gov/foi/warning_letters/d1371b.pdf you get a 404 (not found) error, but I have a copy from my own search engine. It's a pretty vanilla warning letter from 1998 sent to Trinity Chemical Corporation. Again, I'd love to hear the rationale behind this decision.

4. Why block indexing of the archived warning letters?

Thanks to Pharmamkting blog

Tip of the hat to John Mack @ pharmamkting blog for mentioning my site!

FDA Search Firefox Add-on

Just tried to publish my Firefox add-on, titled FDA Search 0.1.0. What I didn't realize is that Firefox keeps your add-on in purgatory until some FF regulars actually review it, and only then can you nominate your add-on for inclusion in the public repository. Thanks to James Cook and his add-on, which I used as a template for mine.

VAERS Update...this time with feeling

The FDA (or is it the CDC? Or is it Health & Human Services? Can't really tell....) just updated its Vaccine Adverse Event Reporting System (VAERS) dataset (April 13, 2009).

What they haven't bothered to note is that they've also changed the structure of their dataset.

VAERS data used to come packaged as 2 CSV files--one called VAERSData.csv and one called VAERSVaccine.csv.

VAERSData.csv used to contain 20 slots per row for symptoms associated with each vaccine adverse event.

In the database world, one says that there is a 1-to-many relationship between a VAERS report and the Symptoms associated with that report.


And day 1 of any remedial Database management course will inform you that because of the 1-to-many relationship, you should separate the VAERS record from the Symptoms records and link each symptom back to its report via a foreign key.

The reason for doing this is that 20 slots for symptoms *may* seem like more than enough slots for any case that you'll ever come across, but at some point, some hypochondriac is going to slip in 21 symptoms and totally screw with your file.

Well, 19 years later, the VAERS folks have finally given the Symptom data its own csv file, appropriately titled VAERSSymptoms.csv. They didn't bother to tell anyone that they did this...and I had to tinker with my parsing algorithm last night to adjust for the changes, but the discerning pharmacovigilantes among us were able to figure it out when we asked ourselves

"Why are there 3 csv files per year now when there used to be 2? And why haven't they updated their file explaining the structure of the VAERS files?" As of this writing, the most recent revision of the VAERS explanation is June 2007.

Ah yes...lest I forget...as of this writing, the Zip file containing all of 2008's VAERS data is contaminated with 2009 data. I think that's just to keep everyone on their toes.