A couple months ago, Pivotal and DataKind announced the launch of the first Pivotal For Good (P4G) project, a collaboration between Crisis Text Line and yours truly. P4G provides philanthropic organizations with free, dedicated engagements with Pivotal’s industry-leading data scientists for a three-month period. Crisis Text Line, whose trained specialists assist hundreds of at-risk teens every day, is the beneficiary of my current work. In this project, we will be leveraging their data to understand and predict teens’ crisis needs. These insights will ultimately help foster data-driven improvements on Crisis Text Line’s current platform (e.g. their ‘switchboard’) and training/recruiting of their crisis specialists.
I’m excited about the opportunity to collaborate with Bob Filbin, Chief Data Scientist at Crisis Text Line, and help further their cause to support teens in crisis. It has been a humbling experience to see the seriousness and range of issues of the teens who use this service, and the profound impact that this service is making on their lives. We’ll be sharing some of the interesting tidbits we’ve discovered along the way on this blog.
Data & Tools At A Glance
Since the August 2013 launch of Crisis Text Line, the organization has collected over:
- 4,800,000 text messages
- 100,000 conversations
- 36,000 unique texters
- 500 crisis specialists
In this engagement, we’ll be analyzing this data with a variety of tools, which include data scientist favorites such as Python, R and MADlib.
Assessing Conversation Quality
When a person begins a conversation with Crisis Text Line, they are typically feeling some strong emotions, which may include anxiety, sadness, or frustration. In other words, they are experiencing what Crisis Text Line terms as a “hot” moment. The goal for each of these conversations is to move a person from hot to cool, so they no longer feel they are in a state of crisis.
After a specialist closes a conversation, the texter is sent a simple question:
“How are you feeling now? Better, same, or worse?”
The responses to this question are the main way that Crisis Text Line quantifies whether texters feel their crisis situation was alleviated, and how they felt the conversation went. Subsequent questions can also be sent to the texter, asking them to elaborate on their initial feedback with questions specifying what they found helpful during the call.
However, we realized that satisfaction can be measured by more than just a survey response. Taking this into account, our question became, ‘How can we measure the value that the texters are getting?” One such way is detecting signs of gratitude as an indicator that the texter found the service helpful.
Giving Thanks
We explored the presence of gratitude by identifying texters’ use of these wordstrings: {‘thank’, ‘thx’, ‘thnx’,’tks’}. We found that these words occurred at least once in 55.2% of texters’ conversations. This is consistent among all groups of users, including those who did not respond to the survey.
So, when do users say ‘thanks’? To determine this, we normalized all the conversation times, with 0 being the start and 1 being the end of the conversation. (To be specific, x = 0.75 in the chart below indicates that the texter sent a message of ‘thanks’ 3/4ths of the way through the duration of the conversation).
We observe a small spike at the beginning of the conversations, which can be accounted to responses to the automated messages, such as:
“Give us a moment, we’re grabbing a counselor for you.”
The presence of thanks remains flat during the first half of conversations, and begins to steadily increase after the halfway point. At this time, specialists may be shifting the conversations from crisis mitigation towards helping the texter identify ways to cope with his/her situation.
As specialists are wrapping up the conversations, we see these indicators of gratitude dramatically increase. This could potentially be attributed to the advice and resources given by the specialist at that time, or texters’ satisfaction with the conversation as a whole.
This investigation opens up a lot of interesting questions for Crisis Text Line:
- Can we improve studies identifying emotions specific to being in a crisis?
- confusion, depression, anger, anxiety, helplessness, overwhelmedness, hopelessness
- How do people express these emotions over text?
- e.g. saying ‘Thank you’ vs. ‘I feel gratitude’
- Can we detect these emotions changing over time?
- Can we detect changes in satisfaction/perception/gratitude of the texter?
- Can we model what causes these emotions to reduce over time?
This quick study on gratitude verified for Crisis Text Line that there are several ways they can use their data to create new emotion-specific measures for Crisis Text Line’s value to the texter. This would greatly help Crisis Text Line’s understanding of the texters using the service, and how they can best accommodate their various needs.
What’s Next?
As this engagement progresses, we’re getting our hands dirty with the data, and investigating various conversation characteristics. These characteristics will ultimately be used as features in our models and live in Crisis Text Line’s analytics framework for future reuse. Stay tuned for our next blog post, where we dig into some of these features.
Have any ideas (algorithms, methods, etc.) that you’d like to share? Please sound off in the comments!
About the Author
