Analyzing the Harvard Q Guide
Harvard Course Ratings
A few weeks ago, I scraped the Harvard Q Guide for fun and (academic) profit. I was wondering how the average Computer Science course compared to other Harvard courses, but neither the Q Guide nor CS50’s alternative offered an averaging function. Naturally, the next best thing to do would be to scrape the entire dataset and sort it myself.
The scraper is written entirely in Python, utilizing knowledge from CS 109 (Data Science) for data collection and STAT 110 (Introduction to Probability) for rudimentary analysis. As a disclaimer, I’m no statistics major so there may be some flaws. There is likely self-selection bias in classes or departments that hasn’t been accounted for, so I would be hesitant to draw broader conclusions on the entire Harvard curriculum.
Code is available here: https://github.com/rogergzou/Q-Guide-Scraper
DATA COLLECTION
Full evaluation information for all Harvard courses since Fall 2006 is available via the Q Guide’s website, but is only accessible to Harvard students and faculty. It includes ratings on the professors and teaching fellows, the coursework (quality, time to complete), the likelihood students would recommend it to others, and the quality of the course as a whole, as well as anonymous comments from students. Prior to 2014 it included difficulty ratings, but that’s an entirely different story. Not only is it a rich dataset, but students who submit (or opt out of submitting) feedback are also allowed to view their grades earlier than others. Though it has its flaws, it is a great help for selecting classes during shopping week.
I played around with the Python Requests library for a while, but couldn’t figure out how to make a working POST request to the login form, so ended up using Selenium Webdriver to enter my credentials in whenever prompted.
Students find courses via links for a specific semester (e.g. Spring 2008). The URL for a particular course for a specific term looks like https://webapps.fas.harvard.edu/course_evaluation_reports/fas/course_summary.html?course_id=XXXXX. Through trial and error and for loops, I discovered the lowest course ID is 10307, and the highest course ID is currently 42559. I iterated and downloaded each page’s HTML source, and then used BeautifulSoup and Pandas to save info for each year’s courses into a CSV. This janky approach led to problems with courses like Freshman Seminar 40G: 9/11 — The Event and the 9/11 Commission Report, where Python would interpret the slashes as subfolders and save it as such. To fix this, I wrote a script that would convert these foldered files into a single file.
About 14,000 of the IDs between 10307 and 42559 led to restricted pages (probably faculty-only evaluation) or the course-not-found page. Those I excluded by checking for the presence of the HTML tag containing the course name. Some courses that should be viewable (e.g. EXPOS 40) currently are not, so were not included in this analysis. There are also some questionable duplicates (two CS 50’s for Fall 2014, perhaps letter grade and SAT/UNSAT versions), so the integrity of this dataset may not be entirely accurate.
With pauses in between page requests to avoid spamming, the scraping took roughly a day to complete.
FINDINGS
AVERAGES AND CATEGORY AVERAGES
The average Harvard course has 24.7 students and comes with 6.1 hours of work per week. Overall, it has a 4.18 rating out of 5, with a slightly higher 4.24 recommend rating. In fact, every major category (defined as 200+ courses since 2006) except for MATH and PHIL had average recommend ratings higher than overall ratings. SLAVIC, VES, and MUSIC overall ratings average 0.2 above the mean, while CHEM courses average a startling 0.4 below the mean.
STUDENT’S EYE VIEW
Readers who have taken STAT 110 may note these figures represent the “dean’s-eye-view” (the simple average of the class sizes/ratings). When weighted for the “student’s-eye-view” (the average class size/rating experienced by students, as it would be reflected by surveying students), the picture looks somewhat different. Larger classes impact more students. However, this may misrepresent the experience of concentrators in a category with large, popular intro classes like COMPSCI or ECON. Most students in Introduction to Computer Science (COMPSCI 50) have no intention of taking another COMPSCI course, but their presence has significant impact on the COMPSCI category.
The average Harvard student’s course has 119.2 students and comes with 5.73 hours of work per week. Overall, it has a 3.95 rating out of 5, with a higher 4.04 recommend rating. PHIL students still hesitate to give higher recommend ratings than overall ones, but MATH and the rest of the major categories do. CHEM courses are now only average 0.17 below the mean, while PHYSICS and ECON courses average 0.3 below the mean.
The reason the number of students skyrockets is because of large categories like ECON (224 students avg), COMPSCI (209 students avg), and GEN-EDS (166 students avg), which in turn are so high because of massive intro classes like Principles of Economics (ECON 10A) and Introduction to Computer Science I (COMPSCI 50) or low workload classes like Classical Chinese Ethical and Political Theory (ETHRSON 18).
ENROLLMENT
Since 2006, the three courses with the highest Harvard enrollment were Justice (MOR-REAS 22), Principles of Economics (ECON 10A), and Classical Chinese Ethical and Political Theory (ETHRSON 18). Justice tops the list at 1072 students in Fall 2007.
These three courses take up the top 12 spots of the last decade (each spot referring to enrollment in a unique year). If one includes Introduction to Computer Science I (CS 50), it becomes the top 17.
On a more recent note, the top 3 courses of the 2014–2015 school year are also ECON 10A (711 students), CS 50 (696 students), and ER 18 (694 students). Interestingly, all these courses are in the Fall.
For those who took STAT 110 with Blitzstein, STAT 110 was the 5th largest course of Fall 2014. Given how ER 18 was on sabbatical in Fall 2015, it likely moved up to 4th place for Fall 2015.
A note pertaining for ETHRSON: There were 7 Fall courses and 4 Spring courses taken by 1566 students in the 2014–2015 school year, 214 in the Spring and and 1352 in the Fall. If you asked a Fall student what ER they were taking, 51% would say ER 18. If you asked a Spring student, 54% would say ER 38 (The Meaning of Life). It will be very interesting seeing what Ethical Reasoning courses Fall 2015 students will take given ER 18 was not an option.
NUMBER OF COURSES
Weighing each course equally, it’s no surprise that there are more Freshman Seminars (FRSEMR) than any other course.
Equally unsurprising is the dominance of ECON in Enrollment, which took 35,234 course slots over the course of 8 and a half years. To put this in perspective, that’s more course slots taken than the first 136 categories combined (34,800). There are 198 categories in total.
WORKLOAD
A special note: Workload for Fall 2006 to Spring 2014 was on a 1 to 5 scale, while courses afterwards had an integer entry of X hours per week. I standardized the 1 to 5 scale to a 1.5 hours to 16.5 hours scale (with a 3 equating to 8.5 hours), but this is an imperfect measure. The 1 to 5 scale fails to accurately capture the workload towards the ends of the scale (courses with 1.0 or 5.0). Pre-change, 41 courses (including duplicates) over 15 semesters received a 5.0 Workload rating. Post-change, 24 courses over 2 semesters have received workloads greater than 16.5 hours per week.
The course most grossly misrepresented was COMPSCI 161, which on the 5.0 scale is 16.5 hours per week but on the new system rates a hefty 30.3 hours per week.
Most classes in the top 10 appear to be grad classes with low enrollment rates. Filtered with a floor of 30+ enrollment, we begin to see classes the average student may actually recognize, like MATH 25A.
Too much COMPSCI in this table…maybe let’s up it to a floor of 250+ enrollment.
…
I don’t know what I expected.
Being COMPSCI is suffering.
In fact, any minimum enrollment level between 59 and 259 generates a pure COMPSCI top 10 list. Fun fact: CS 50 has drained 57017.8 hours of work since 2006, more than any other class and 5x more than ECON 10A.
CONCLUSION
From this cursory look into the Q Guide, I found out Harvard students are either quite generous in rating their courses or Harvard just has very good courses in general. Over 17 semesters, 1466 courses (including same courses but different years) have received 5.0 Overall ratings, while only 6 courses received a 1.0 (all of which had only 1 evaluation).
As a Computer Science concentrator, my courses rate below average on their overall evaluations. From a student’s perspective, COMPSCI courses have the most workload (10.7 hours/week yay overloading) out of any of the categories, 5 hours more than the mean. It’s a bit depressing thinking of all the additional time spent psetting, but most the STEM categories (COMPSCI, PHYSICS, MATH, CHEM, etc.) have higher workloads than the mean. COMPSCI also has the second biggest average class size. None of this was all that surprising.
The most curious thing to me was that students constantly gave higher recommend ratings than overall ratings. Why a student might recommend a bad course eludes my grasp. Perhaps misery loves company, or perhaps a particular course is a better way to fulfill the concentration requirement than an even worse one. It may reflect optimism that the future course will be better than the one they took, whose score is reflected in the overall rating. That the CHEM and PHIL categories gave lower recommend ratings could either be a fluke or indicative of a third confounding factor.
One of the best things about scraping the Q guide was being able to impose additional sorting parameters to discover hidden gems, like a GOV Data Science course, that would usually slip by unknown. Sorting by workload instead of tediously clicking through each COMPSCI course saves a lot of time and effort; I’m not sure why CS50 courses doesn’t have that option. If any Harvard College student wants the dataset, please email me with your @college account.
FURTHER RESEARCH AND CAVEATS
I opted to only scrape the default page, which has total enrollment and overall, workload, and recommend ratings. While it would have been interesting to run some sentiment analysis on student comments or figure out how professors are rated differently across courses, that information was not on the default tab and I got lazy. The default page also has only 1 decimal digit of precision, but opening up a new details page for each rating in order get the better 2 digits of precision seemed like overkill.
I’m sure there are many more insights to be gleaned from this data (anyone want to run linear regression, or see how section scores correlate with overall ratings?), but a statistics student I am not. If any Harvard student wants to analyze the data further, I’d be happy to help.
Thanks to Cherie Hu for her amazing editing skills, as well as Kevin Yang, Melody Guan, and Jerry Anunrojwong (CS 109 group) for giving me the tools to run this analysis.
