Identifying Affect in Students’ Tweets

Affect is present in both students’ blog posts and their tweets, but the Twitter dataset offers a better opportunity to study variations in students’ responses throughout the semester. Students consistently completed Primary Source Tweets (PST), Entry Tweets, and Exit Tweets for each class and their responses to each class highlight differences in students’ reactions to course content. Students’ reactions varied widely, but their responses were frequently shaped by the topics, readings, and activities students encountered in class. Their affective responses were a result of their own choices as well as my decisions as the course instructor.

In this section, I use the sentiment analysis package, sentimentr, to show the presence and proportions of various types of affect in the All Tweets Dataset. Analysis of the average sentiment of each hashtag points to the influence of different topics and readings on students’ responses. The average sentiment attached to each tweet type likewise suggests that the form and purpose of an activity impacted affective response. Additionally, a comparison of the PST and Exit Tweets demonstrates that access to context information sometimes altered students’ reactions to content.

Presence & Proportions of Sentiment Analysis in the Tweets

Students exhibited a wide variety of affective responses throughout the semester. As noted in Table 8, sentimentr identified a total of 6,777 tweets (of 11,454) with a sentiment score greater than 0. Fewer than half as many tweets (2,924) returned a score of less than 0. Another 1,650 displayed neutral scores equal to zero and 103 tweets (“NA”) did not contain any words recognized by the lexicons sentimentr uses for comparison. Altogether, sentimentr scored ~59.21% of tweets as positive and only ~25.63% as negative. The average sentiment for the entire All Tweets set was slightly positive, too. The average overall was 0.12323, with a maximum positive value of 1.5718 and a minimum negative value of -1.7888.


Table 8. Count/percentage of tweets with sentiment scores greater, less than, or equal to zero. “NA” represent tweets with no sentiment score returned by sentimentr.

Sentimentr ScoreCountPercentage
Greater than 0678259.2107%
0164314.3443%
Less than 0293625.6330%
NA1030.8992%
Total11454100%

Students’ positive and negative responses to course material depended on the perspectives both the students and I, the instructor, brought to the class. The content of the readings, lecture content, and activities I chose set the stage for students’ affective responses. Personal values and sociocultural contexts, in turn, framed their reactions to content and activities. Average sentiment scores for each hashtag serve as the best evidence for students’ collective responses to specific topics in the class. Delving into the average sentiments of each tweet type within a hashtag similarly illuminates the differences in affective response depending on the form, purpose, and timing of a given Twitter activity.

Impact of Topics and Activities on Affective Response

The average sentiment scores for each hashtag (represented in Figure 3) indicate the topics students responded to most positively or negatively. Classes 03, 13, and 06 had the highest average scores while Classes 18, 16, and 23 ranked lowest. The topics for the most positive classes were Ancient Egypt, Ancient China, and early monotheism in Ancient Israel. Fall of Rome, the Roman Republic, and African Kingdoms/Empires were the respective foci of the classes with the least positive averages.


Figure 3. Average sentiment and count of all tweets calculated by hashtag, sorted by average sentiment score

Bar graph representing total number of tweets and average sentiment by hashtag. Class 18 has the lowest average; class 03 has the highest.

Possible Reasons for Positive Responses

The most positive classes make sense given students’ values and backgrounds. Ancient Egypt (Class 03) likely appealed to students because they have some familiarity with the topic thanks to films like The Mummy or Gods of Egypt and video games, most notably Assassin’s Creed: Origins. The civilization carries a sense of mystery, too; students are often fascinated by mummies, pyramids, and unfamiliar religious rites.

My choice of primary source reading for this topic contributed to the positive response too. The translation of “The Birth of Hatshepsut” details the divine birth of a powerful female king of Egypt. Students read the text as evidence of gender equity in Egypt, a characteristic that tied into their own values.

Positive responses to Ancient China and the Analects of Confucius are likewise the result of values and background. The majority of students in Singapore and in the UB-SIM program are of Chinese descent and many students, regardless of racial identity, had passages from the Analects written on the walls of their primary or secondary school classrooms. Finally, the discussion of selections from Genesis and my lecture on early monotheism likely tapped into students’ religious and spiritual values. About 77% of Singaporeans between the age of 15 and 24 identify with a religious tradition and many students resonated with what they perceived as Abraham’s obedience and faithfulness in the primary source reading.

Possible Reasons for Negative Responses

The most positively received topics were typically those in which my choice of lecture material or primary source reading evoked a sense of familiarity or resonated with personal values. By contrast, the classes with the least positive averages centered on topics students found unfamiliar or disagreeable. Leaving aside Class 18, whose average is mostly the result of the Entry Tweets (as discussed below), the next two most negative classes involved geographically distant civilizations and readings that students found especially unsettling. African kingdoms and empires like Mali, Nubia, and Aksum (Class 23) were potentially the least familiar civilizations for students. At the start of class, I asked them to tell me what first came to mind when they heard “Africa.” Their answers were fairly homogenous: poverty, famine, and war; animals and stunning landscapes; “Africa” by Toto. Unlike Ancient Egypt, which evoked familiar images of pharaohs, pyramids, and mummification, students had virtually no familiar reference points for the polities discussed in Class 23.

The readings for African kingdoms and empires (Class 23) and the Roman Republic (Class 16) were likewise unfamiliar and unsettling to students. The selections I assigned from Ibn Battuta’s travelogue, Travels in Asia and Africa, and the Roman historian Livy’s Histories both contained stories in which students identified mistreatment of women. Ibn Battuta observed and criticized the practice of women going about naked in the Malian emperor’s court while Livy recounts the early Romans’ abduction and coercion of the Sabine women. In contrast to their celebration of Hatshepsut’s power in Ancient Egypt, students roundly condemned perceived inequalities in Ibn Battuta and Livy’s texts. Singapore has one of the lowest gender gaps in the world and students prize the ideal of gender equality. As a result, the readings ran against many of the class participants’ core values.

The Impact of Activity Form, Timing, and Purpose on Affective Response

The average sentiment of each hashtag implies that resonance or conflict between the topics I presented in class and students’ values or identities impacted positive or negative responses. Examining the average sentiments of each form of Twitter activity also illuminates the influence of an activity’s timing, form, or purpose on affective response. Figure 4 demonstrates that the most varied activities (Entry Tweets and General Tweets) produced the most varied responses while repetitive activities (PST and Exit Tweets) showed a smaller range of averages.


Figure 4. Average sentiment of all tweets calculated by hashtag and type


The Entry Tweets showed the largest range between the minimum (Class 18) and maximum (Class 04) average sentiment values. This was likely because I provided a different writing prompt for the Entry Tweet in each class. The question I asked heavily influenced the nature of students’ sentiments. For Class 18, for example, I asked students to describe the words or images they associated with “barbarians.” They used words like cruel, inhumane, uncivilized, primitive, and brutish — all highly negative descriptions. The negative nature of the Entry Tweets decreased the overall average for the Class 18, making it the least positive hashtag of the semester. For Class 04, by contrast, the Entry Tweet read: “What makes a person a hero?” Positive words like inspiration, courage, trustworthy, well-loved, and selfless populated the tweets.

The Exit Tweets had the smallest range between the minimum (Class 03) and maximum (Class 17) averages (shown in Table 9). At the end of each class, students responded to the same question: “What did you think was most significant in today’s class?” The repetitive nature of the question likely contributed to the smaller degree of variation between the most positive and least positive average sentiments. This conclusion is further supported by the unexpectedly small variation in the PST as well. Because students responded differently depending on topic, I anticipated a high degree of difference between the most negative and most positive averages for the PST. Instead, the PST’s variation is the second lowest of the tweet types, which implies that the routine nature of this activity also decreased the variation. For each PST assignment, students completed two tweets containing open-ended comments/questions about a source; the consistency of the activity appears to have narrowed the range of affective response.

This conclusion is further supported by the unexpectedly small variation in the PST as well. Because students responded differently depending on topic, I anticipated a high degree of difference between the most negative and most positive averages for the PST. Instead, the PST’s variation is the second lowest of the tweet types, which implies that the routine nature of this activity also decreased the variation. For each PST assignment, students completed two tweets containing open-ended comments/questions about a source; the consistency of the activity appears to have narrowed the range of affective response.


Table 9. Minimum, maximum, and range of sentiment scores calculated by tweet type

Tweet TypeMinimumMaximumRange
Entry-0.2531042970.3919986540.645102951
General-0.0522699740.3186016310.370871605
PST-0.0649000270.1777692310.242669258
Exit0.0875936410.2229839770.135390335

Context Information and Affective Response

A comparison of the Primary Source Tweets (PST) and the Exit Tweets, contained in Table 10, suggests access to context information also impacted the average sentiment of each tweet type and hashtag. In every class, students responded more positively to a civilization after gaining the context information I included in the lecture. This suggests a change of perspective between students’ initial reactions in the PST and their reflections at the end of the class, which supports the conclusions of historical empathy scholars who argue that guided study of primary sources can encourage greater nuance in students’ affective responses.


Table 10. Change in average sentiment score from PST to Exit Tweets for each hashtag. Hashtags 5, 12, 15, 20, 25 are missing as these were workshops or lecture-only days.

HashtagPSTExitChange
3NA0.222983977NA
40.0867040640.1259502370.039246173
60.1611709700.2171610900.055990120
70.0629692840.1614843430.098515059
80.0309320570.1236790830.092747025
90.1175391650.2088833390.091344174
10NA0.103831639NA
110.0918962150.1219450230.030048807
130.1680879360.1982245860.030136650
140.0884239360.1514119920.062988056
16-0.0134296580.0982569990.111686657
170.0386376740.0875936410.048955968
180.0567333820.1676238380.110890456
190.0918857480.1986649430.106779195
210.0980243590.1337012280.035676868
22-0.0649000270.1758952610.240795288
230.1252680370.1284587760.003190740
240.1777692310.1973087860.019539555

Class 22, which focused on the Mongols, showed the highest rate of change and serves as a useful example of changes in students’ perspectives. There are a number of commonalities between the most common words in the PST and the Exit Tweets (detailed in Table 11). Mongols expectedly tops the list. The prominence of “alathir” and “ibn” reflect the name of the author of one of the readings (“On the Tartars,” by Ibn Al-Athir). Bias and women show up in both lists as do violent words; “death,” “slavery,” “violent,” and “conquer” appear in the PST and “brutal” makes the list for the Exit Tweets.


Table 11. Count and comparison of 15 most common words in PST and Exit Tweets about the Mongols

RankMost Common PSTPST CountMost Common ExitExit Count
1mongols24mongols44
2alathir8khan16
3ibn6women14
4people6religious12
5tartars5alathir7
6world5religion7
7author4tolerance6
8biased4egalitarian6
9compared4genghis6
10death4ibn6
11slavery4pretty6
12violent4wives6
13women4bias5
14conquer3brutal5
15fathers3khubilai5

However, the Exit Tweets list has distinct names and terms in the mix, most notably Genghis Khan and Khubilai Khan and “religious/religion,” “tolerance,” and “egalitarian.” The combination of common terms between the PST and the Exit Tweets plus the introduction of new people and terms implies both continuity and change in students’ perceptions of the Mongols. Their first reactions to the primary source readings mainly focused on the violent nature of the Mongols. The Exit Tweets acknowledged this violence, but also called attention to the aspects of Mongol culture included in the lecture, including a relatively egalitarian social structure and principled religious tolerance across the Mongol empire. Students’ deeper knowledge of the Mongols’ history and conquests resulted in a more nuanced and positive response.

Students’ more positive responses in the Exit Tweets, however, did not always extend to new insights about the primary source reading. In the Class 08 tweets, for instance, the only common words between the top 15 words in the PST and the Exit Tweets were “persians” (the class focused on the Persian Empire) and “herodotus” (who authored the primary source reading, “On the Customs of the Persians”). The lack of common terms suggests there was no change in perspective regarding the primary source. Students instead simply shifted their attention from the things they found difficult about the primary source (“drunk,” “people,” “customs,” “culture”) to new ideas from the lecture (“zoroastrianism,” “tomyris,” “bias,” “cyrus”). The Exit Tweet responses were still positive but did not represent a more nuanced affective response to the primary source from students. Based on the Class 22 and Class 08 tweets, it appears increased context knowledge can moderate affective response, but does not always do so.



Footnotes

  1. As noted in Methodology: Data Analysis, sentimentr codes the sentiment of full sentences rather than individual words. The package creates an aggregate sentiment score based on words included the sentimentr lexicons and valence shifters (words that negate or amplify a phrase, such as “no” in “no good”). Sentimentr therefore offers insight into the average sentiments of each tweet that cannot be captured in the word-by-word analysis of the lexicon methods (AFINN, Bing, and NRC).

  2. See “All Tweets Dataset” in Downloads & Datasets.

  3. Each hashtag represents one class meeting.

  4. Source: Tweets from All Tweets Dataset in Downloads & Datasets. Sentimentr scores calculated in RStudio. See “Clean, Tokenize, Gather All Tweets” to “Count 0” in the R Script in Downloads & Datasets.

  5. Source: Tweets from All Tweets Dataset in Downloads & Datasets. Sentimentr scores calculated in RStudio. See “All Tweets Average Sentiment by Hashtag” in the R Script in Downloads & Datasets.

  6. During an in-class dicussion of the Analects, a few students mentioned they recognized portions of the Analects from quotes painted on the walls of their primary/secondary school classrooms.

  7. Melody Zaccheus, Xue Qiang Pang, and Keng Gene Ng, “Youth in Singapore Shunning Religion,” The Straits Times, Mar 21, 2016, http://web.archive.org/web/20190415021139/https://www.straitstimes.com/singapore/youth-in-singapore-shunning-religion.

  8. The United Nations ranks a nation’s gender inequality based on number of deaths per 100,000 births, adolescent birth rate, percentage of population with some secondary education, representation in government, and labor force participation rate. In 2017, Singapore ranked 12th out of 189 ranked nations. This placed Singapore’s gender equality rank well above other Asian nations including Japan (#22), China (#36), India (#127), and Malaysia (#62). It also far outstrips the United States (#41). That being said, gender justice advocates at the Singapore, non-profit organization AWARE continue to call attention to ongoing gender inequalities, including lack of support for single mothers, widespread abuse of female domestic workers, and inadequate response to reports of sexual harrassment and assault.

  9. See Methodology: Data Collection for a description of each type of Twitter activity.

  10. Source: Tweets from All Tweets Dataset in Downloads & Datasets. Sentimentr scores calculated in RStudio. See “Average Sentiment by Hashtag & Tweet Type” in the R Script in Downloads & Datasets.

  11. Removing the Entry Tweets from the average increases the Class 18 average from 0.00788 to 0.11507, comparable to more middling averages like those of Classes 21, 07, and 10.

  12. Source: Tweets from All Tweets Dataset in Downloads & Datasets. Sentimentr scores calculated in RStudio. See “Min/Max/Range by Tweet Type” in the R Script in Downloads & Datasets.

  13. Foster, Yeager, and Maley found especially compelling evidence of the usefulness of diverse evidence in their study of students’ interaction with narratives related to Truman’s decision to drop the atomic bomb. See Elizabeth Anne Yeager, Stuart J. Foster, and Sean D. Maley, “Why People in the Past Acted as They Did: An Exploratory Study in Historical Empathy,” International Journal of Social Education 13, no. 1 (April 15, 1998): 8–24.

  14. Source: Tweets from All Tweets Dataset in Downloads & Datasets. Sentimentr scores calculated in RStudio. See “PST & Exit Average Sentiment Comparison” in the R Script in Downloads & Datasets.

  15. Source: Tweets from All Tweets Dataset in Downloads & Datasets. Sentimentr scores calculated in RStudio. See “PST & Exit Compared (Class 22 & 08)” in the R Script in Downloads & Datasets.