1
00:00:00,060 --> 00:00:02,500
The following content is
provided under a Creative

2
00:00:02,500 --> 00:00:04,019
Commons license.

3
00:00:04,019 --> 00:00:06,360
Your support will help
MIT OpenCourseWare

4
00:00:06,360 --> 00:00:10,730
continue to offer high quality
educational resources for free.

5
00:00:10,730 --> 00:00:13,330
To make a donation or
view additional materials

6
00:00:13,330 --> 00:00:17,215
from 100 of MIT courses,
visit MIT OpenCourseWare

7
00:00:17,215 --> 00:00:17,840
at ocw.mit.edu.

8
00:00:21,360 --> 00:00:22,594
RICHARD EBERHARDT: Hi, Pablo.

9
00:00:22,594 --> 00:00:23,350
PABLO SUAREZ: Good to be back.

10
00:00:23,350 --> 00:00:25,016
RICHARD EBERHARDT:
Nice having you here.

11
00:00:25,016 --> 00:00:28,880
So we wanted to
talk about a number

12
00:00:28,880 --> 00:00:32,940
of high-level postmortemy
things for how the class went.

13
00:00:32,940 --> 00:00:36,110
So we've got some high-level
things we wanted to talk about.

14
00:00:36,110 --> 00:00:39,160
We want to talk about the
collaboration between MIT

15
00:00:39,160 --> 00:00:42,790
and the Red Cross/Red Crescent
group and your folks--

16
00:00:42,790 --> 00:00:45,960
how student preparedness
went, in particular, comparing

17
00:00:45,960 --> 00:00:49,750
those teams that got a lot of
support, like client support,

18
00:00:49,750 --> 00:00:50,687
like Snap, of course.

19
00:00:50,687 --> 00:00:53,020
But also, some of the other
teams got a little bit more,

20
00:00:53,020 --> 00:00:54,690
and then the
unsupported teams, which

21
00:00:54,690 --> 00:00:57,250
aren't exactly unsupported,
but they got less,

22
00:00:57,250 --> 00:01:00,250
so that's largely
the Heatwave Team,

23
00:01:00,250 --> 00:01:03,350
I think was the main one that
we want to talk about there.

24
00:01:03,350 --> 00:01:07,770
So collaboration what do
you all think about how

25
00:01:07,770 --> 00:01:08,934
we talked to each other?

26
00:01:08,934 --> 00:01:10,850
How we met with each
other-- things like that?

27
00:01:13,755 --> 00:01:15,130
PHILIP TAN: I like
the way how we

28
00:01:15,130 --> 00:01:17,150
started in the beginning
of the semester.

29
00:01:17,150 --> 00:01:20,280
We had to talk-- and we
had some Skype issues.

30
00:01:20,280 --> 00:01:23,470
But I think the idea of
kicking off the entire class

31
00:01:23,470 --> 00:01:27,210
with the initial
presentation of this

32
00:01:27,210 --> 00:01:28,960
is what your final
project is going to be.

33
00:01:28,960 --> 00:01:30,751
It's going to take us
a while to get there,

34
00:01:30,751 --> 00:01:32,620
but you come into
class understanding

35
00:01:32,620 --> 00:01:33,870
this is what it's going to be.

36
00:01:33,870 --> 00:01:38,074
And Pablo being the
person to describe

37
00:01:38,074 --> 00:01:39,490
why this is going
to be important,

38
00:01:39,490 --> 00:01:42,690
I think, set lot of
expectations well.

39
00:01:42,690 --> 00:01:51,310
So I think later on, when we
actually got to that project,

40
00:01:51,310 --> 00:01:53,590
the degree to which
each team actually

41
00:01:53,590 --> 00:01:55,770
could have been
supported actually

42
00:01:55,770 --> 00:01:58,256
became more of an issue.

43
00:01:58,256 --> 00:02:00,130
The teams that you could
directly talk with--

44
00:02:00,130 --> 00:02:04,460
and you were going to be the
primary user of the game--

45
00:02:04,460 --> 00:02:07,456
clearly benefited.

46
00:02:07,456 --> 00:02:09,289
But you were right there
from the beginning.

47
00:02:09,289 --> 00:02:13,050
These are the teams that,
maybe, they did receive support,

48
00:02:13,050 --> 00:02:15,570
or some of them didn't
receive support,

49
00:02:15,570 --> 00:02:17,420
but weren't talking
with people who

50
00:02:17,420 --> 00:02:23,366
had been there right from
the start, I think maybe

51
00:02:23,366 --> 00:02:24,990
had felt a little
bit more disconnected

52
00:02:24,990 --> 00:02:28,480
from the main
thrust of the class.

53
00:02:28,480 --> 00:02:31,190
At least when they got
to that final assignment.

54
00:02:31,190 --> 00:02:35,190
So something I like to think
about for future classes

55
00:02:35,190 --> 00:02:38,080
is the extent to
which-- if we're

56
00:02:38,080 --> 00:02:41,030
going to repeat this format--
where the primary client comes

57
00:02:41,030 --> 00:02:44,870
in right at the beginning to set
the tone for the entire class.

58
00:02:44,870 --> 00:02:49,120
How do we represent more of
the variety of different things

59
00:02:49,120 --> 00:02:50,745
that you can do right
at the beginning?

60
00:02:53,770 --> 00:02:55,760
So that when students
think about what they're

61
00:02:55,760 --> 00:02:59,590
going to be doing throughout
the entire semester,

62
00:02:59,590 --> 00:03:03,350
they are actually keeping full
scope of the project in mind,

63
00:03:03,350 --> 00:03:06,400
not just the one example that
might be presented to them

64
00:03:06,400 --> 00:03:07,544
on the first day of class.

65
00:03:07,544 --> 00:03:08,960
PABLO SUAREZ: And
we also may want

66
00:03:08,960 --> 00:03:12,170
to have something like
an engagement meter.

67
00:03:12,170 --> 00:03:14,330
Because I wasn't
aware of the fact

68
00:03:14,330 --> 00:03:17,420
that some teams were
feeling disengaged.

69
00:03:17,420 --> 00:03:19,430
I would show up, some
people have questions,

70
00:03:19,430 --> 00:03:21,440
some other have less questions.

71
00:03:21,440 --> 00:03:25,550
And from my perspective, those
who have questions I spent time

72
00:03:25,550 --> 00:03:29,820
and it was about the same
chronometric time per group.

73
00:03:29,820 --> 00:03:31,540
I could see that
some were more lost,

74
00:03:31,540 --> 00:03:33,262
but I wasn't aware
that what was needed

75
00:03:33,262 --> 00:03:34,470
was more fire in their belly.

76
00:03:34,470 --> 00:03:35,240
So if we could--

77
00:03:35,240 --> 00:03:35,960
RICHARD EBERHARDT: I
couldn't see that either.

78
00:03:35,960 --> 00:03:36,350
PABLO SUAREZ: Right.

79
00:03:36,350 --> 00:03:38,683
RICHARD EBERHARDT: The students
never reported it to us.

80
00:03:38,683 --> 00:03:40,480
SARA VERRILLI: Yeah,
I was going to say,

81
00:03:40,480 --> 00:03:42,230
when we asked the students
to do their postmortems,

82
00:03:42,230 --> 00:03:45,070
they were really clear about how
this had been a big motivation,

83
00:03:45,070 --> 00:03:47,140
and interest had been
a big problem for them.

84
00:03:47,140 --> 00:03:50,170
But they never mentioned it
to us during check-in meetings

85
00:03:50,170 --> 00:03:53,150
when we were doing play testing,
when we were working with them.

86
00:03:53,150 --> 00:03:55,420
So that's the sort
of thing that we

87
00:03:55,420 --> 00:03:57,420
need to-- probably
one of the things

88
00:03:57,420 --> 00:04:00,920
you do is probably to
warn them about that.

89
00:04:00,920 --> 00:04:03,570
These are hard topics, and
they can become boring topics

90
00:04:03,570 --> 00:04:05,850
very quickly.

91
00:04:05,850 --> 00:04:10,010
And they need to step up and
let us know when they think

92
00:04:10,010 --> 00:04:12,020
they're facing a brick wall.

93
00:04:12,020 --> 00:04:14,960
RICHARD EBERHARDT: I think
the clues were there.

94
00:04:14,960 --> 00:04:18,649
If we had means and a
way to remind ourselves,

95
00:04:18,649 --> 00:04:21,160
like a checklist or something
simple like that, just to say,

96
00:04:21,160 --> 00:04:22,210
are you engaged?

97
00:04:22,210 --> 00:04:23,604
What's your motivation?

98
00:04:23,604 --> 00:04:25,270
And, again, that's
assuming that we even

99
00:04:25,270 --> 00:04:27,070
want that to be our job.

100
00:04:27,070 --> 00:04:28,540
I think for anything
client based,

101
00:04:28,540 --> 00:04:30,037
it has be our,
probably, our job.

102
00:04:30,037 --> 00:04:31,620
SARA VERRILLI: I
think that when we're

103
00:04:31,620 --> 00:04:35,250
asking them to work on projects
where they have relatively

104
00:04:35,250 --> 00:04:38,587
little control over the topic,
I think we actually do need to.

105
00:04:38,587 --> 00:04:40,170
Because I think part
of it was I don't

106
00:04:40,170 --> 00:04:42,050
think the teams knew
how much freedom they

107
00:04:42,050 --> 00:04:46,622
had to deviate from the
topic they had chosen

108
00:04:46,622 --> 00:04:47,330
and the specific.

109
00:04:47,330 --> 00:04:48,996
They thought they had
chosen this thing,

110
00:04:48,996 --> 00:04:50,900
and now they were stuck with it.

111
00:04:50,900 --> 00:04:53,180
Even though one
team was like, we

112
00:04:53,180 --> 00:04:55,060
chose a topic that just
wasn't deep enough.

113
00:04:55,060 --> 00:04:57,490
There wasn't enough there.

114
00:04:57,490 --> 00:04:59,320
And if they had come
and talked to us,

115
00:04:59,320 --> 00:05:00,820
we could have given
them the freedom

116
00:05:00,820 --> 00:05:03,110
to well, go change to a new
topic, or how can we go--

117
00:05:03,110 --> 00:05:03,330
RICHARD EBERHARDT: Go deeper.

118
00:05:03,330 --> 00:05:04,600
SARA VERRILLI: --how
can we go deeper in it?

119
00:05:04,600 --> 00:05:05,340
Right.

120
00:05:05,340 --> 00:05:08,010
But we were not
sufficiently warned.

121
00:05:08,010 --> 00:05:09,840
PHILIP TAN: But
that's where rapport

122
00:05:09,840 --> 00:05:13,450
with someone who knows a lot
about the domain can help.

123
00:05:13,450 --> 00:05:16,720
And if they don't feel like they
have that rapport or someone

124
00:05:16,720 --> 00:05:20,140
who they can readily contact
to ask even seemingly

125
00:05:20,140 --> 00:05:22,522
silly questions,
then we're not going

126
00:05:22,522 --> 00:05:24,105
to be able to get
down to actually ask

127
00:05:24,105 --> 00:05:25,270
the important questions.

128
00:05:25,270 --> 00:05:26,686
Because sometimes
they can't tell.

129
00:05:26,686 --> 00:05:30,190
They can't necessarily tell what
looks like flagging motivation

130
00:05:30,190 --> 00:05:32,600
on a part of the
team or what just

131
00:05:32,600 --> 00:05:35,911
seems like end-of-semester
pressures on the students,

132
00:05:35,911 --> 00:05:36,410
right?

133
00:05:39,480 --> 00:05:41,460
They haven't been working
on enough projects

134
00:05:41,460 --> 00:05:43,690
to be able to tell
the difference.

135
00:05:43,690 --> 00:05:46,490
And so we do need to help
them through that, I think.

136
00:05:46,490 --> 00:05:47,870
SARA VERRILLI: By that you
mean the students, then?

137
00:05:47,870 --> 00:05:48,575
PHILIP TAN: The students.

138
00:05:48,575 --> 00:05:48,920
I mean the students.

139
00:05:48,920 --> 00:05:51,420
PABLO SUAREZ: And one of the
things you may want to consider

140
00:05:51,420 --> 00:05:53,950
is we know that both in the
real life-- in the future,

141
00:05:53,950 --> 00:05:56,800
in their future, as well
as in future courses--

142
00:05:56,800 --> 00:05:58,950
there will be limited
ability of those

143
00:05:58,950 --> 00:06:01,600
who are experts to interact
with the game design team.

144
00:06:01,600 --> 00:06:03,781
The game design team,
you took the effort--

145
00:06:03,781 --> 00:06:06,030
and I thought it was very
well-balanced-- to have some

146
00:06:06,030 --> 00:06:08,640
who know programming, some
who know narrative, some who

147
00:06:08,640 --> 00:06:11,340
know user interface, and so on.

148
00:06:11,340 --> 00:06:13,480
They will, by default,
not have expertise

149
00:06:13,480 --> 00:06:17,780
on heatwaves or cholera
or etc or climate science.

150
00:06:17,780 --> 00:06:21,740
They cannot assume that
all they need to know--

151
00:06:21,740 --> 00:06:23,650
including the appetite
for engagement--

152
00:06:23,650 --> 00:06:26,730
will be only in the client.

153
00:06:26,730 --> 00:06:32,530
MIT is a bubbling powerhouse
of engaging stuff.

154
00:06:32,530 --> 00:06:35,825
And if they could, as part
of what they are told-- they

155
00:06:35,825 --> 00:06:37,450
will have to do in
the future, and what

156
00:06:37,450 --> 00:06:39,620
they have to do in this
course, to reach out

157
00:06:39,620 --> 00:06:44,350
to engaging people, fellow
students, and junior faculty

158
00:06:44,350 --> 00:06:48,430
or whoever may give them some
time-- to just go and say,

159
00:06:48,430 --> 00:06:49,910
what is exciting about this?

160
00:06:49,910 --> 00:06:51,500
What are the cutting-edge areas?

161
00:06:51,500 --> 00:06:53,120
What are the
institutional challenges.

162
00:06:53,120 --> 00:06:56,355
So that they also have to
fuel their own appetite.

163
00:06:56,355 --> 00:06:57,980
RICHARD EBERHARDT:
Having an assignment

164
00:06:57,980 --> 00:07:01,000
or having part of that final
project being like here's

165
00:07:01,000 --> 00:07:03,650
the research component
that we expect you to do,

166
00:07:03,650 --> 00:07:06,076
but we didn't put an
underline over it.

167
00:07:06,076 --> 00:07:08,700
SARA VERRILLI: We didn't make it
explicit that they needed to--

168
00:07:08,700 --> 00:07:10,741
RICHARD EBERHARDT: We did
when it was in trouble.

169
00:07:10,741 --> 00:07:14,080
Like for Saving Gora Gora,
they were in big trouble.

170
00:07:14,080 --> 00:07:16,750
We told them a number-- you
told them, we told them.

171
00:07:16,750 --> 00:07:19,220
I think you'd be surprised
what they came up with.

172
00:07:19,220 --> 00:07:21,672
Because they actually, I
think, they took the baton

173
00:07:21,672 --> 00:07:22,380
and went with it.

174
00:07:22,380 --> 00:07:23,190
SARA VERRILLI: They
did that [INAUDIBLE].

175
00:07:23,190 --> 00:07:23,490
PHILIP TAN: Nice.

176
00:07:23,490 --> 00:07:25,609
RICHARD EBERHARDT: The
other cholera team did not,

177
00:07:25,609 --> 00:07:27,150
and they were given
the same-- almost

178
00:07:27,150 --> 00:07:28,580
the exact same-- feedback.

179
00:07:28,580 --> 00:07:29,330
That is the thing.

180
00:07:29,330 --> 00:07:31,970
PHILIP TAN: I think there
is a huge social barrier

181
00:07:31,970 --> 00:07:36,360
between the undergraduates
and the rest of the institute.

182
00:07:36,360 --> 00:07:39,620
The noble laureates, and
even the post-docs here,

183
00:07:39,620 --> 00:07:44,070
that's a lot of social
pressure keeping students away

184
00:07:44,070 --> 00:07:45,650
from that kind of interaction.

185
00:07:45,650 --> 00:07:46,660
You're all right.

186
00:07:46,660 --> 00:07:50,086
All of these things do exist
here at MIT, but some of this

187
00:07:50,086 --> 00:07:51,970
is reinforced by
professors themselves--

188
00:07:51,970 --> 00:07:53,553
professors who keep
their doors closed

189
00:07:53,553 --> 00:07:56,120
when they in their
office, things like that.

190
00:07:56,120 --> 00:08:01,680
And so I do think we have
to do give them the license,

191
00:08:01,680 --> 00:08:04,014
and sometimes that's pointing
them to specific people.

192
00:08:04,014 --> 00:08:06,430
RICHARD EBERHARDT: Yeah, giving
them resources like here's

193
00:08:06,430 --> 00:08:07,220
a list.

194
00:08:07,220 --> 00:08:08,710
One of these might
pan out for you.

195
00:08:08,710 --> 00:08:09,585
The others might not.

196
00:08:09,585 --> 00:08:13,950
And that's OK, just do what you
can, kind of prime them for it.

197
00:08:13,950 --> 00:08:17,724
PHILIP TAN: It's hard to be
a teenager in this space.

198
00:08:17,724 --> 00:08:19,390
RICHARD EBERHARDT:
Actually, since we're

199
00:08:19,390 --> 00:08:20,973
talking about the
student preparedness

200
00:08:20,973 --> 00:08:24,880
in that research like what we
saw between what we consider

201
00:08:24,880 --> 00:08:28,152
the supported teams versus
the unsupported teams, which

202
00:08:28,152 --> 00:08:30,110
may or may not be fair--
could be the motivated

203
00:08:30,110 --> 00:08:32,150
teams versus the
unmotivated teams.

204
00:08:32,150 --> 00:08:33,826
Snap, in particular, really--

205
00:08:33,826 --> 00:08:35,409
PABLO SUAREZ:
[INAUDIBLE] correlation.

206
00:08:35,409 --> 00:08:37,809
RICHARD EBERHARDT:
--really well-supported.

207
00:08:37,809 --> 00:08:41,977
They actually reached out to
you all for help with the game.

208
00:08:41,977 --> 00:08:44,310
You actually even did testing
for them that was actually

209
00:08:44,310 --> 00:08:46,780
coming from you and
Jana, yeah, because you

210
00:08:46,780 --> 00:08:47,800
were running workshops.

211
00:08:47,800 --> 00:08:48,633
PABLO SUAREZ: Right.

212
00:08:48,633 --> 00:08:51,140
RICHARD EBERHARDT: How did those
workshops run-- how did you

213
00:08:51,140 --> 00:08:52,980
interact with the team?

214
00:08:52,980 --> 00:08:54,570
PABLO SUAREZ: So,
first of all, I

215
00:08:54,570 --> 00:08:58,070
think it's important to
distinguish not only what you

216
00:08:58,070 --> 00:08:59,990
are calling supportive
versus unsupportive,

217
00:08:59,990 --> 00:09:01,950
but also Snap versus
everything else.

218
00:09:01,950 --> 00:09:04,660
Because Snap was a
game concept that

219
00:09:04,660 --> 00:09:07,750
preexisted the course
that needed refinement

220
00:09:07,750 --> 00:09:09,815
and digital interface.

221
00:09:09,815 --> 00:09:11,690
So from that perspective,
it's not surprising

222
00:09:11,690 --> 00:09:13,150
that they have something
clearly defined that they

223
00:09:13,150 --> 00:09:14,710
could run faster and deeper.

224
00:09:14,710 --> 00:09:17,040
And the group was a group
that engaged and went

225
00:09:17,040 --> 00:09:19,890
deep and went fast.

226
00:09:19,890 --> 00:09:21,970
The interaction
we had with them,

227
00:09:21,970 --> 00:09:25,450
also, very importantly,
we have events

228
00:09:25,450 --> 00:09:27,250
that we could use the product.

229
00:09:27,250 --> 00:09:28,930
So it was much easier
to give feedback

230
00:09:28,930 --> 00:09:30,445
saying when we do
this, we expect

231
00:09:30,445 --> 00:09:33,070
people that would get bored, why
don't we try this other thing.

232
00:09:33,070 --> 00:09:34,528
RICHARD EBERHARDT:
It's milestones,

233
00:09:34,528 --> 00:09:35,950
which we give them milestones.

234
00:09:35,950 --> 00:09:37,630
But there is no
weight behind them.

235
00:09:37,630 --> 00:09:40,620
These ones have an actual,
if it doesn't work by then,

236
00:09:40,620 --> 00:09:42,670
someone's going to have
egg on their face, right?

237
00:09:42,670 --> 00:09:45,000
PABLO SUAREZ: And from that
perspective, we know this,

238
00:09:45,000 --> 00:09:49,820
but we can try to do
more of giving students

239
00:09:49,820 --> 00:09:53,170
more proportion of work that
we know-- that we, the clients

240
00:09:53,170 --> 00:09:54,800
know-- that we have milestones.

241
00:09:54,800 --> 00:09:57,570
And that is limited by
how much of the real world

242
00:09:57,570 --> 00:09:59,630
gives us milestones to abide by.

243
00:09:59,630 --> 00:10:01,310
But we can try that.

244
00:10:01,310 --> 00:10:05,360
I do not feel comfortable with
the label supportive versus

245
00:10:05,360 --> 00:10:10,150
unsupportive because it sounds
like some got time and support

246
00:10:10,150 --> 00:10:11,660
and some others didn't.

247
00:10:11,660 --> 00:10:15,520
I think it's more about the
form of the relationship

248
00:10:15,520 --> 00:10:18,640
and how the time was used or
the opportunity to use the time

249
00:10:18,640 --> 00:10:19,510
was used.

250
00:10:19,510 --> 00:10:22,990
Because I have no doubt that
you supported the team when

251
00:10:22,990 --> 00:10:25,110
you knew support was needed.

252
00:10:25,110 --> 00:10:28,280
And as far as I know, the
same was true on our part.

253
00:10:28,280 --> 00:10:30,520
I think it's more
about having sensors

254
00:10:30,520 --> 00:10:33,340
about when support is needed
that is not being delivered.

255
00:10:33,340 --> 00:10:35,673
RICHARD EBERHARDT: I think
when we're talking about them

256
00:10:35,673 --> 00:10:38,150
in this way, I'm
using it based on how

257
00:10:38,150 --> 00:10:40,050
they describe their
own relationship

258
00:10:40,050 --> 00:10:41,480
and their own experience.

259
00:10:41,480 --> 00:10:43,515
So that the teams who we
are calling unsupported

260
00:10:43,515 --> 00:10:45,140
are the ones who, in
their postmortems,

261
00:10:45,140 --> 00:10:46,598
called out and
said, we didn't feel

262
00:10:46,598 --> 00:10:48,240
like we got client support.

263
00:10:48,240 --> 00:10:49,660
They got the same amount.

264
00:10:49,660 --> 00:10:52,000
It was a different
way, different quality,

265
00:10:52,000 --> 00:10:53,110
different quantity.

266
00:10:53,110 --> 00:10:55,693
SARA VERRILLI: The way to put
it is the same amount of support

267
00:10:55,693 --> 00:10:58,199
was there for them.

268
00:10:58,199 --> 00:10:59,990
One other way to put
it is they didn't know

269
00:10:59,990 --> 00:11:02,590
how to reach out and get it.

270
00:11:02,590 --> 00:11:04,170
They didn't use
the resources that

271
00:11:04,170 --> 00:11:06,810
were there because,
at some level,

272
00:11:06,810 --> 00:11:08,470
we weren't pushing them at them.

273
00:11:08,470 --> 00:11:11,950
And a lot of MIT courses tend
to encourage students to solve

274
00:11:11,950 --> 00:11:13,454
all their problems themselves.

275
00:11:13,454 --> 00:11:14,870
They don't want
them reaching out.

276
00:11:14,870 --> 00:11:17,430
They want them--
figure this out.

277
00:11:17,430 --> 00:11:20,315
And so it may come
as more of a second--

278
00:11:20,315 --> 00:11:22,440
it may not come to them as
first nature to go, hey,

279
00:11:22,440 --> 00:11:25,065
we need to go and reach out
for more help and grab that--

280
00:11:25,065 --> 00:11:27,440
RICHARD EBERHARDT: And so
that's actually the same thing.

281
00:11:27,440 --> 00:11:29,020
So what we saw with
the Snap team is

282
00:11:29,020 --> 00:11:31,360
we wanted-- the feedback
they were getting from us

283
00:11:31,360 --> 00:11:33,615
was look at your front
end, look at your UI.

284
00:11:33,615 --> 00:11:37,370
We wanted to see a lot
of changes on the UI.

285
00:11:37,370 --> 00:11:39,365
Because they were working
on these milestones,

286
00:11:39,365 --> 00:11:41,250
they were really worried
about the back end

287
00:11:41,250 --> 00:11:43,291
so they were doing a lot
of work in the back end.

288
00:11:46,272 --> 00:11:47,980
They were getting two
different messages,

289
00:11:47,980 --> 00:11:50,970
and they chose client message
rather than instructor message.

290
00:11:50,970 --> 00:11:52,275
Totally OK and totally viable.

291
00:11:54,997 --> 00:11:56,080
We took that into account.

292
00:11:56,080 --> 00:11:58,260
But I think the
main issues we had

293
00:11:58,260 --> 00:12:00,150
with them is when
they got our message,

294
00:12:00,150 --> 00:12:02,317
it was, they heard it early on.

295
00:12:02,317 --> 00:12:04,650
They didn't react to it until
it was too late-- had they

296
00:12:04,650 --> 00:12:07,220
done a little bit
extra in the beginning.

297
00:12:07,220 --> 00:12:09,770
So one thing I think we need
to think about with the class

298
00:12:09,770 --> 00:12:13,410
is thinking about how are
we helping them understand

299
00:12:13,410 --> 00:12:16,570
how to prioritize feedback?

300
00:12:16,570 --> 00:12:17,880
Right?

301
00:12:17,880 --> 00:12:20,500
What's more important?

302
00:12:20,500 --> 00:12:22,350
Could be because
here's something that's

303
00:12:22,350 --> 00:12:23,474
actually going to get used.

304
00:12:23,474 --> 00:12:26,230
It could also be here's a lesson
we want to make sure they get.

305
00:12:26,230 --> 00:12:27,396
I think they got the lesson.

306
00:12:27,396 --> 00:12:29,314
They just got it
in a harder way.

307
00:12:29,314 --> 00:12:31,730
PHILIP TAN: Sometimes they get
a lesson at a point of time

308
00:12:31,730 --> 00:12:33,880
that they can't
actually execute on it.

309
00:12:33,880 --> 00:12:37,120
But hopefully they can do it for
their next project after class.

310
00:12:39,650 --> 00:12:41,580
Something I want to
suggest, actually,

311
00:12:41,580 --> 00:12:44,120
because we've been talking
about how it's been difficult

312
00:12:44,120 --> 00:12:48,040
for us to figure out what's
going on inside the teams.

313
00:12:48,040 --> 00:12:51,410
We can see the games
and the testing,

314
00:12:51,410 --> 00:12:55,100
but the team dynamics, we
don't find out about that

315
00:12:55,100 --> 00:12:57,950
until near the end of the class.

316
00:12:57,950 --> 00:13:02,010
And something that we used
to do outside of this class,

317
00:13:02,010 --> 00:13:04,640
but in our research
work, was to get all off

318
00:13:04,640 --> 00:13:07,540
the scrum masters together
on a regular basis

319
00:13:07,540 --> 00:13:11,497
to just do an exchange of
information with each other.

320
00:13:11,497 --> 00:13:13,830
They don't even have to be
necessarily reporting to us--

321
00:13:13,830 --> 00:13:16,020
although we can
moderate those sessions.

322
00:13:16,020 --> 00:13:19,270
It's more about them sharing
their experiences with each

323
00:13:19,270 --> 00:13:22,970
other, and then it may make
it more obvious that there are

324
00:13:22,970 --> 00:13:25,849
resources out there that some
teams are taking advantage

325
00:13:25,849 --> 00:13:26,890
of and some teams aren't.

326
00:13:30,094 --> 00:13:32,260
Of course, we can use those
situations for teaching,

327
00:13:32,260 --> 00:13:37,640
but also for
teaching each other.

328
00:13:37,640 --> 00:13:40,225
SARA VERRILLI: Creating
some group, group support.

329
00:13:40,225 --> 00:13:41,850
RICHARD EBERHARDT:
We talked about that

330
00:13:41,850 --> 00:13:44,590
before this semester started,
but we couldn't figure out

331
00:13:44,590 --> 00:13:47,860
a way to do that fairly.

332
00:13:47,860 --> 00:13:49,900
I already feel
producers bear the brunt

333
00:13:49,900 --> 00:13:52,419
of the work in the class.

334
00:13:52,419 --> 00:13:53,960
They're the ones we
see all the time.

335
00:13:53,960 --> 00:13:56,480
We don't see the students
working on programming

336
00:13:56,480 --> 00:13:57,530
or design as much.

337
00:13:57,530 --> 00:13:58,980
Design a little bit
more, but programming,

338
00:13:58,980 --> 00:14:00,230
we just almost never see them.

339
00:14:00,230 --> 00:14:03,230
SARA VERRILLI: And we let
the teams use the producers

340
00:14:03,230 --> 00:14:06,110
as their face-- so giving
all the presentations,

341
00:14:06,110 --> 00:14:08,750
giving all the reports,
answering questions when

342
00:14:08,750 --> 00:14:12,640
we ask questions in class.

343
00:14:12,640 --> 00:14:15,720
Think about how we convince the
teams to share out that work

344
00:14:15,720 --> 00:14:17,370
a little bit more evenly.

345
00:14:17,370 --> 00:14:18,870
PHILIP TAN: Well,
perhaps, maybe, we

346
00:14:18,870 --> 00:14:23,735
can reduce the extent
that we require

347
00:14:23,735 --> 00:14:26,480
these producers to do
presentations in class,

348
00:14:26,480 --> 00:14:30,666
and replace that with these
more closed-off sessions

349
00:14:30,666 --> 00:14:32,790
where they don't feel like
they have to be on stage

350
00:14:32,790 --> 00:14:34,510
and put on the best
face for their team.

351
00:14:34,510 --> 00:14:36,080
They can actually just talk
about this is the problem

352
00:14:36,080 --> 00:14:37,079
I'm having with my team.

353
00:14:38,597 --> 00:14:40,180
RICHARD EBERHARDT:
So one thing I also

354
00:14:40,180 --> 00:14:41,596
wanted to make
sure that we talked

355
00:14:41,596 --> 00:14:43,230
about what was it like for you?

356
00:14:43,230 --> 00:14:45,670
How much of your
time did we demand?

357
00:14:45,670 --> 00:14:47,640
Was it a lot compared
to other projects?

358
00:14:47,640 --> 00:14:48,620
Was it a little bit?

359
00:14:48,620 --> 00:14:50,625
Was it about moderate?

360
00:14:50,625 --> 00:14:53,000
PABLO SUAREZ: I want to start
by saying one of the things

361
00:14:53,000 --> 00:14:55,760
that I really appreciated
was the time we spent--

362
00:14:55,760 --> 00:14:58,700
both face-to-face
and emailing and so

363
00:14:58,700 --> 00:15:01,550
on-- before the
beginning of the course,

364
00:15:01,550 --> 00:15:04,100
to have understanding of
what you and your students

365
00:15:04,100 --> 00:15:07,320
can offer, what we
need and can offer,

366
00:15:07,320 --> 00:15:09,160
and to manage
expectations, as you said.

367
00:15:09,160 --> 00:15:10,990
I think that was very well done.

368
00:15:10,990 --> 00:15:12,590
And it's not always the case.

369
00:15:12,590 --> 00:15:16,620
So with the students and
professors and Red Cross,

370
00:15:16,620 --> 00:15:19,480
in general, we
[INAUDIBLE], so thank you.

371
00:15:19,480 --> 00:15:21,030
That was really appreciated.

372
00:15:21,030 --> 00:15:27,135
I think that amount of time
that I personally had to give

373
00:15:27,135 --> 00:15:30,290
was about as much as I
anticipated, about as much

374
00:15:30,290 --> 00:15:33,880
as I think is reasonable
to expect for the future.

375
00:15:33,880 --> 00:15:36,770
We do have Jana, in particular,
and a few other members

376
00:15:36,770 --> 00:15:39,120
of the team-- of
the Red Cross team--

377
00:15:39,120 --> 00:15:41,900
who were available
to give a hand.

378
00:15:41,900 --> 00:15:44,460
And I think we needed
to improve because I

379
00:15:44,460 --> 00:15:46,150
did make it very
clear that I couldn't

380
00:15:46,150 --> 00:15:49,430
get five emails per
day asking questions.

381
00:15:49,430 --> 00:15:53,460
But to have some kind of ability
for the right contact when

382
00:15:53,460 --> 00:15:57,220
either engagement levels are
too low or mega confusion

383
00:15:57,220 --> 00:16:01,380
levels are too high or sense of
purpose is diluted and so on.

384
00:16:01,380 --> 00:16:04,570
So to refine that.

385
00:16:04,570 --> 00:16:09,730
I think it would be
better to think of times

386
00:16:09,730 --> 00:16:15,050
when there can be slots
of decided interaction

387
00:16:15,050 --> 00:16:18,110
between each team, and someone
from the client organization

388
00:16:18,110 --> 00:16:22,050
that doesn't have
to be during class.

389
00:16:22,050 --> 00:16:24,494
Because I would always
interfere with your plan.

390
00:16:24,494 --> 00:16:25,910
You gave us one
opportunity that I

391
00:16:25,910 --> 00:16:28,550
thought was very useful,
but a little bit late

392
00:16:28,550 --> 00:16:32,346
in the development process
and also very crammed.

393
00:16:32,346 --> 00:16:33,762
By the time we
said, OK, these are

394
00:16:33,762 --> 00:16:35,760
the kind of things you
can do, there was no time

395
00:16:35,760 --> 00:16:37,135
for them to think
and reflect on,

396
00:16:37,135 --> 00:16:40,440
OK, we can do A or B
or C. Show that to us,

397
00:16:40,440 --> 00:16:41,960
and then us providing feedback.

398
00:16:41,960 --> 00:16:44,940
So even if it were one
more chance via Skype

399
00:16:44,940 --> 00:16:49,960
but scheduled, planned, and if--
it is in the nature of our work

400
00:16:49,960 --> 00:16:52,440
to have to travel, and it
is in the nature of our work

401
00:16:52,440 --> 00:16:55,616
that travel timing changes
outside of our control.

402
00:16:55,616 --> 00:16:57,740
So where there's a default
date, and if things have

403
00:16:57,740 --> 00:16:59,020
to be changed, it is changed.

404
00:16:59,020 --> 00:17:02,420
So to know that there's going to
be an instance of feedback that

405
00:17:02,420 --> 00:17:06,830
may be too little, too late,
still but to anticipate that.

406
00:17:06,830 --> 00:17:09,079
RICHARD EBERHARDT: Yeah,
part of that is the students,

407
00:17:09,079 --> 00:17:11,240
we don't always know
the student's schedules.

408
00:17:11,240 --> 00:17:14,130
So it could possibly be
planned in the beginning

409
00:17:14,130 --> 00:17:14,910
of the semester.

410
00:17:14,910 --> 00:17:16,500
But planning it
before the semester

411
00:17:16,500 --> 00:17:17,910
would be really difficult.
Unless we just say it--

412
00:17:17,910 --> 00:17:19,810
this is the time that
you're going to have.

413
00:17:19,810 --> 00:17:21,976
If you don't take advantage
of it, you don't get it.

414
00:17:21,976 --> 00:17:24,099
PABLO SUAREZ: Right.

415
00:17:24,099 --> 00:17:26,260
It has to be we
propose it's this week.

416
00:17:26,260 --> 00:17:27,540
This is the default time slot.

417
00:17:27,540 --> 00:17:31,036
If not, that time slot, agree
on a different time slot.

418
00:17:31,036 --> 00:17:32,410
And then it's up
to the students.

419
00:17:32,410 --> 00:17:33,060
RICHARD EBERHARDT:
So you're actually

420
00:17:33,060 --> 00:17:35,450
saying you would have talked
to them a little bit more.

421
00:17:35,450 --> 00:17:38,230
PABLO SUAREZ: If
the problem was lack

422
00:17:38,230 --> 00:17:40,410
of engagement or perception
of lack of support.

423
00:17:40,410 --> 00:17:40,720
RICHARD EBERHARDT:
So it's really

424
00:17:40,720 --> 00:17:41,770
that planning and scheduling.

425
00:17:41,770 --> 00:17:43,310
So as long as that's
in place, then the time

426
00:17:43,310 --> 00:17:44,810
is there, because
it's been planned.

427
00:17:44,810 --> 00:17:45,460
OK.

428
00:17:45,460 --> 00:17:46,835
PABLO SUAREZ: And
also I think it

429
00:17:46,835 --> 00:17:50,160
would have been nice for them
to have access to, say, someone

430
00:17:50,160 --> 00:17:53,030
from the Ghana Team to
look at the color, someone

431
00:17:53,030 --> 00:17:57,180
from the Heatwave Team in
Buenos Aires-- to at least

432
00:17:57,180 --> 00:17:59,690
to get a flavor, there's
another human being there that

433
00:17:59,690 --> 00:18:01,300
eventually can use their game.

434
00:18:01,300 --> 00:18:04,630
And I didn't do enough
to help them envision

435
00:18:04,630 --> 00:18:06,730
the user-- either
the Red Cross user

436
00:18:06,730 --> 00:18:11,190
or the community level or
policy or government user.

437
00:18:11,190 --> 00:18:14,591
PHILIP TAN: I think we
need to state that strongly

438
00:18:14,591 --> 00:18:17,410
that it's not just the
information that you're

439
00:18:17,410 --> 00:18:18,000
providing.

440
00:18:18,000 --> 00:18:21,240
It's the fact that there is
a real person who can really

441
00:18:21,240 --> 00:18:23,830
benefit from the work
you're doing is much more

442
00:18:23,830 --> 00:18:25,855
powerful than some sort
of abstracted audience.

443
00:18:25,855 --> 00:18:27,230
RICHARD EBERHARDT:
So we provided

444
00:18:27,230 --> 00:18:30,990
plenty of documentation,
but that's not

445
00:18:30,990 --> 00:18:32,130
the same as a human being.

446
00:18:32,130 --> 00:18:34,472
PHILIP TAN: That's not
the emotional drive

447
00:18:34,472 --> 00:18:36,180
that you're going to
get-- oh, someone is

448
00:18:36,180 --> 00:18:37,472
counting on me doing good work.

449
00:18:37,472 --> 00:18:39,180
RICHARD EBERHARDT: So
maybe it might just

450
00:18:39,180 --> 00:18:42,200
be something for us to talk
to the Terrascope folks

451
00:18:42,200 --> 00:18:43,362
to see how they do that.

452
00:18:43,362 --> 00:18:44,820
PHILIP TAN: Well,
they specifically

453
00:18:44,820 --> 00:18:50,140
get people who can commit
on regular meetings.

454
00:18:50,140 --> 00:18:53,350
And the idea is--
my understanding

455
00:18:53,350 --> 00:18:56,070
is that it's for the
domain knowledge primarily.

456
00:18:56,070 --> 00:18:57,890
But you get that
side benefit of here

457
00:18:57,890 --> 00:19:00,610
is a real human being who's
excited about this problem,

458
00:19:00,610 --> 00:19:06,600
or maybe really concerned
about this problem,

459
00:19:06,600 --> 00:19:09,266
and that can influence
the motivation of a team.

460
00:19:09,266 --> 00:19:11,390
PABLO SUAREZ: And also, I
don't know to what extent

461
00:19:11,390 --> 00:19:13,040
we can keep cramming
what students have

462
00:19:13,040 --> 00:19:15,400
to do-- I know they already
have to do way too much--

463
00:19:15,400 --> 00:19:18,790
but to look at how
that kind of field

464
00:19:18,790 --> 00:19:22,360
is currently being communicated
in the game universe

465
00:19:22,360 --> 00:19:23,210
and outside of that.

466
00:19:23,210 --> 00:19:27,320
So PowerPoints or documents,
they're, across the board,

467
00:19:27,320 --> 00:19:29,520
generally boring
and unsuccessful--

468
00:19:29,520 --> 00:19:31,530
even if they have
the right content.

469
00:19:31,530 --> 00:19:34,030
So for them to get a sense
that if someone really

470
00:19:34,030 --> 00:19:36,770
needs to create
awareness on that topic,

471
00:19:36,770 --> 00:19:38,650
and this is the best
they have in what

472
00:19:38,650 --> 00:19:42,500
exists in the real world, how
about trying to give something

473
00:19:42,500 --> 00:19:46,860
that makes people want to
do more through their game?

474
00:19:46,860 --> 00:19:50,740
PHILIP TAN: I think saying, this
is how low the bar is, can we

475
00:19:50,740 --> 00:19:53,616
get it slightly better, is
less motivating than saying,

476
00:19:53,616 --> 00:19:54,740
what's the best you can do?

477
00:19:58,995 --> 00:20:01,120
RICHARD EBERHARDT: But it's
also choosing something

478
00:20:01,120 --> 00:20:02,620
that's going to get used.

479
00:20:02,620 --> 00:20:06,809
Why go in a direction that's--
if this is what those people

480
00:20:06,809 --> 00:20:09,350
are thinking about, if you give
them something completely not

481
00:20:09,350 --> 00:20:11,724
like that at all, they're
probably not going to know what

482
00:20:11,724 --> 00:20:12,930
to do with it.

483
00:20:12,930 --> 00:20:14,970
PHILIP TAN: But I mean,
I can imagine things

484
00:20:14,970 --> 00:20:17,360
like evaluating current
methods of teaching

485
00:20:17,360 --> 00:20:19,930
about a certain concept, to be
part of that research process

486
00:20:19,930 --> 00:20:22,590
that we've been talking about--
it's understanding the problem.

487
00:20:22,590 --> 00:20:23,590
RICHARD EBERHARDT: Cool.

488
00:20:23,590 --> 00:20:26,760
All right, so I think it's
about time to let you go.

489
00:20:26,760 --> 00:20:28,490
But thanks so much
for coming in.

490
00:20:28,490 --> 00:20:30,690
PHILIP TAN: We thank you.

491
00:20:30,690 --> 00:20:32,240
Huge gratitude.