4. Refine and evaluative research

GOAL OF PHASE 4: Building on the concept developed in stage 3, you will revise and improve the first iteration of the final product concept. “Expert” interviews, paper prototypes and other forms of evaluating the product/service will be conducted with your concepts and real users. The deliverables from stage 4 are the refinements to your design concept and documentation of the evaluative research you conducted.

Week 9

We were able to set up ride-alongs with members of Medic 7, a unit of the Pittsburgh Emergency Medical Services system based in the neighborhood of Greenfield. On Saturday 3/5, Kate and Catherine each shadowed a medic team on a typical 8 hour shift. We plan to get our other two members out on ride-alongs in week 10 or 11, depending on availability.

Shift #1 (7am-3pm)

Kate met the Medic Unit on campus at 7 am and was introduced to its shift members, Mark T. and Tom P., both members of the department for almost 20 years. Tom is currently a paramedic and Mark is a crew chief, although about to be promoted to a supervisory position.  Kate used participant observation methods as well as touchstone tours (of the Medic 7 Facebook page and station lounge) to gain a more empathetic understanding of the daily life of a paramedic. They went on four calls in total; all of which were low priority or not life-threatening.

 Shift #2 (3pm-11pm)

At 3pm Kate and medics returned to campus so that Catherine could begin her ride-along shift.

Key Insights:

  • Paramedics have high level of emotional intelligence and are able to conduct human behavioral analysis and profiling almost unconsciously.
  • Paramedics value their family and create a fraternity amongst themselves that provides strength and stability in a chaotic job environment
  • The job is very stressful and paramedics often need ways to deal with the stress; however, they are a insular sub-culture that will not seek help outside of their community.
  • There is a lot of waiting involved in between calls in which time medics can unwind a little and focus on completing other tasks that are important to them, such as communicating with family and friends via phone or social media.
  • A lot of paramedics die fairly young and memorializing these fallen friends is extremely important, whether through murals and displays in the station house, tattoos, or posts on social media remembering those who are no longer with them.

Week 10

We started the second mini by watching a clip from 2001: A Space Odessey – the moment when things start going wrong and HAL proves to be more ‘intelligent’ than they thought it was.

If this was taken as the worst case scenario for AI, what would the worst and best case scenarios for our systems be?


We found that each of our points on either side of the table fit into one of our three areas of intervention.


Mapping this on a Kano model (this is a useful article on making it practical for designers to use – https://www.interaction-design.org/literature/article/the-kano-model-a-tool-to-prioritize-the-users-wants-and-desires) helped us create a model for how we should think about the features or interventions.


We moved on to re-creating a stakeholder map

IMG_20160314_165740724And revised our diagram of the stakeholder map

3/16: Team meeting & Studio work session

We met at 12pm to catch up on and complete the process documentation for the past few weeks.

In studio, we learn more about speed dating techniques for concept generation and then broke into our teams for a work session. We began by unpacking the comments and feedback we received from our third presentation (generative research) and wrote on some key takeaways:

  • Evolution over time
  • Images are helpful for understanding context
  • Finish research and debrief earlier
  • Noise is a part of the experience but not a distraction
  • How heavily should EMS rely on it?
  • Main role(s)/objectives of CUI (functions & features)
    • change in form = role must be stable
    • change in role = form must be stable
  • Stress relief – indirect rather than direct (reducing load rather than playing music)
  • Technicalities of CUI could think ahead


We started exploring where the CUI would function:IMG_8277

Next, we debriefed on the ride-alongs Kate and Catherine went on over break:


Some major findings:


Some implications:IMG_8290

Meeting with Peter and Bruce:

  • In-between times….What about the time between work and home?
  • Focus on key moments for CUI and HMS (show visually), other components of system can be conveyed verbally
  • Where is the symbiosis? Pick out points where symbiosis, not just assistance, can be offered for best effect.



Fri. 3/18: Brainstorming Speed Dating Scenarios

Went through implications to see if still valid, came up with 12 scenario concepts and assigned


Helpful: Being methodical about going between steps and thinking in frameworks.



Deliverable: This document of scenarios: https://docs.google.com/document/d/1fX93kxxygNQdPI90w_C_vVjChII2qdJ89M2lYHwuvNE/edit?usp=sharing

Conceptual Development: This idea of where the symbiosis and the CUI?


Mon. 3/21: Little Bits Workshop

In class on Monday, we learned to prototype with Little Bits.


Tues. 3/22: Review Scenarios as Team

Our team sent out a template and we split up the scenarios so each person drew 3-4 scenarios.

[Insert Scenarios]

Wed. 3/23: Speed Dating Scenarios Workshop with Pittsburgh EMS


This issue with AI…

“jobs that are growing in developed countries demand complex interaction skills that require deep knowledge, judgement and experience — more than routine transaction or production skills. For example, in the European Union 16 million jobs employing highly qualified people are expected to be added between 2010 and 2020, while jobs employing people with few or no formal qualifications are expected to decline by around 12 million.75 Many of the jobs being created are in nontradeable services such as health care, education and public services, which are also areas that are fundamentally human development enhancing.” –Human Development Report 2015

Reviewing the brief: http://research.microsoft.com/en-us/events/designexpo2016/


Also, can we prototype with various chatbots??

Are there questions we can ask Irina and Kevin now?

Thu 3/24 & Fri, 3/25: Speed Dating Workshop Debrief


We first arranged our scenarios into a matrix of forms (on/with/around) x themes (communication/transport/stress relief). We proceeded to translate the comments we received during our workshop into YESes MAYBE-IFs and NOs to clearly see patterns in what resonated with and what was rejected across our three participants.

The first surprising  revelation came regarding stress relief. It appeard from our previous conversations with the EMS professionals, and observations made during the ride-alongs, that stress is major pain point and something they wished to be addressed. However, we received negative reactions across the board to scenarios involving stress relief, despite the variety of ways and degrees to which the stress was handled.

“stress is hereditary to the job”

“we know what we’re getting into”
“it’s a human problem best tackled by humans”

Learning:Perhaps the symbiosis exists where the CUI knows when to take a step back and let the human conversations take place without barriers. Maybe it creates opportunities for these conversations to take place — maybe by simply taking the users’ minds off of things that can be better handled by the computer?

Aside from this, we also saw that the “around” form was much better received by all three participants. Any sort of wearables or anything they had to carry were rejected because they are “uncomfortable, and tend to catch on things”. Scenarios involving an independently mobile agent, though initially met with interest, were ultimately rejected because they didn’t want to take responsibility for the extra equipment.

Moving forward: We have identified some key features that should be part of our design, and are in the process of constructing new scenarios. Over the next week we will start prototyping some possible designs and testing them with users.



Mon 3/28: Meeting with Peter & Bruce

From our meeting we were able to synthesize our findings further and have developed three moments that best show the symbiosis of EMS professionals and the CUI.

  1. Symbiosis as co-action – experience gap and emergency driving mode, creates opportunity for more experienced partner to step in and offer advice

  2. Technology enabling stress – reducing action, individual specific suggestions

  3. CUI as partner in conversation – think aloud navigation (road-closure information, event traffic, etc.)

Wed 3/30: Starting to Prototype for Conversations

We did role playing where based on a scenario, we decided to act out the scenario with a CUI. In the scene, Kate was an experienced paramedic, Shruti was the less-experienced paramedic, Saumya was the CUI, and Catherine was the Dispatcher/note-taker.



  • Knowing when to interject
    • How? Mode?
      • if it happens to interrupt a human, it stops…”oh, sorry…you go on…” (auditory)
      • visual vs. auditory cues
  • full recognition
  • passive listening

Reflections from Austin’s Workshop

  • You’re learning to speak to the system…the onus of communication is on the person communicating.
  • But what about the onus of the communication being on the system to listen and understand?
  • How you design it sets expectations…
  • Dealing with ambiguity, help completing task, recovering gracefully
  • Awareness, invocation, input, acknowledgement, understanding, dialog and disambiguation, actions and answers, task completed or not complete


Fri 4/1: Team Meeting

We met in the studio from 10:30 to 1 pm to further discuss the conversational elements.

We broke down the scenario to make it more granular

This helped us figure how the whole system would work and what specific aspects we want to deep dive into through out prototype.

Principles to prototype and explore:

  • Prototype: Visual v. Auditory cues
    • Problem: Humans don’t want to interrupt. It should have a more human conversational flow with “lulls”
    • Form?
    • Gaze of user and gaze of CUI form
    • Signal Detection Theory
  • Prototype: Personality
    • Name
    • Gender?
    • Relationship – Mentor/Mentee
    • Does it evolve over time? Considering there are two medics, does it speak to each of them differently? To what degree does it change?
      • Activation-password (visual, auditory, tactile?), how does it recognize the medic?
    • How do we communicate with machine-ness?
  • Prototype: Acknowledgement
    • that its going to do something and what its going to do
  • Prototype: Status of CUI
    • off->not listening->passively listening->actively listening->having or supporting a conversation->proactive (initiates conversation, interjects)
    • ability of humans to override setting – if not in the mood for a proactive CUI, can turn off at any point during shift
  • Prototype: Humor
    • setting up a joke for a human to complete (i.e. teasing one teammate using the jokes created earlier by other teammate)
      • memory – remembers jokes previously used, sets up referential joke based on previously successful jokes that were constructed by human counterparts.
    • voicing support
  • Prototype: Change over between shifts
    • How does it disconnect from one human or team and reconnect to another? Auditory and visual cues (like Alexa powering down and powering up)?

We created scales for various variables as well in order to better understand where each of us felt the CUI should be situated.

File_000 (7).jpeg

On scale of machine to human, where do each of us place our CUI?

On a scale of machine intelligence from current state to ideal state, where should CUIs, generally, be placed?

  • It should never ambition. Its goals and values should not change over time, even if it is evolving/learning?
  • Should it have goals at all?


Mon 4/4: Team Meeting


Rough cut of storyboarded video

Read “someone lied to a laptop”? How to recover from error / manage perception of recovering from error

Gender bias?

Wed 4/6: Team Meeting

We ran through our script in a car to see how the different modes of interaction will unfold in the car. We’re revising the scripted conversation, thinking about how to present it for the workshop, how the GUI is going to work in synchronization with the CUI and how the LED works as a feedback mechanism for the CUI.


Fri 4/8: Testing Prototypes

This slideshow requires JavaScript.

Ran a 1-hr workshop on Friday test CUI concept prototypes with paramedics. Got some good responses. Turned out using CUI to manage work/life balance not a hit despite what we were told during speed dating. Paramedics much more interested in log book idea and V2V concept. They were also very focused on visuals of GUI during the route navigation part.

Presentation Day 4/13

Link to our presentation for phase IV: