AI and User Journeys in Higher Ed

Transcript

Joel Goodman
From Bravery Media. This is Appendix B. My name is Joel Goodman, and with me is Kristin Van Dorn. Hi, Kristin.

Kristin Van Dorn
Hi, Joel. How’s it going?

Joel
Things are going fine.

Trucking along, headed, headed into the middle of this year as we’re recording this anyway, what do you want to talk about today?

Kristin
So I think today we will talk about AI and using large language models in your content strategy.

Joel
Oh yeah, this is a great idea, right? Everyone should be doing this and writing all of their content with AI.

Kristin
Oh my gosh. No.

Joel
No, but, but that’s what all of the vendors online have been telling us to do for the last, you know, seven months or more, but no, you’re right.

Kristin
I think there has been such excitement about large language models and the possibility that they could either. Like supplant writing jobs, or they could supplement writing jobs in the sense that it’s really easy to put in a prompt and spit back relatively professional-sounding content.

And in a time when writers feel busy, when it’s been hard to read and audit our websites, it seems like, well, this is a solution, a stopgap in order to make sure that we have the content that we need available.

Joel
So why wouldn’t we do it, though? No, I think you’re right. And it’s, I’m not sure that I’m convinced that institutions are already all in on outsourcing all of their content, but I do see this curiosity plus plus, right? This sense that it’s going to be technology that does supplant the way that we’ve done content forever and ever and ever, which is hopefully you do some strategy work, and you do some research. Then you have someone write the content, and then you have someone proof the content, and then the content goes out. And it takes time, sure. But man, these AI tools, you know, you can tell it to write a thing, and it writes a thing. I don’t know that we should ever be trusting all of our content production to large language models.

Kristin
Yeah, well, you know, I think that there’s like a lot of misunderstandings about large language models. Like, you know, in some sense, people talk about large language models as being sort of the average of our content.

So if we ask, can you write, say, an About page for a college, uh, use the source material and put it in this voice and tone for this audience with these key messages being prioritized, right? Something like that. If your prompt is sophisticated, the thing is, is that I think we have this vision that it’s scanning a bunch of like websites and going, okay, these 47 websites, the next word that they use after their mission is this, and they’re sort of predicting based on an average.

And I don’t think it’s necessarily an average that they’re creating, but I also don’t think that they’re creating thoughtful quality content that’s designed specifically to the user experience that you’re trying to cultivate. I don’t believe that they’re sophisticated enough at this moment in time to capture what a 17-year-old is looking for in terms of a directed content experience.

Joel
I think one of the deficiencies, too, is that there aren’t many colleges and universities right now that have trained their own; they call them RAGs, right? Their own like offshoots of these, LLMs. And I’m not talking about taking ChatGPT from OpenAI and creating a custom GPT because that’s kind of it, but not really.

The way to do this properly, the way that all of these companies that are coming out with their own, chat interfaces and their own, LLMs, they’re taking an offshoot of the version and then they are training it separately on their own data sets apart from everything else. And they’re usually doing this, not on OpenAI’s platform.

They’re doing it on Amazon web services or somewhere else. I don’t know of any institutions apart from maybe like Harvard, MIT, like that kind of level that have done these internally. And so what you run into is you’re asking a generic commercial platform, a product to write content for you without any background as to what your institution is, apart from it, understanding in its technical way, what it’s read from your website, if it’s even scanned in the first place, asking it to write something that is going to resonate with your audience. And unless you put a lot of copy editing work into it, you know, that content isn’t really going to be fit for achieving your goals at all.

And I think part of that is the issue, you know. Kristin, you’ve seen this when, when you’ve played around with it, you know, you ask it to write some, some website content for a specific page, and all you get is, you know, the term, uh, you know, a vibrant community over and over and over again, right? And I’ve heard that same thing from other people. Like the exact same phrase, “vibrant community” comes out. This indicates that something in there is more generic than we might want or expect from our content. And, I think underestimating the amount of work that would go into making that kind of pre-written, canned, predicted content work for us in a good way is dangerous. Like, we do underestimate that a lot.

Kristin
Yeah, you know, one thing that I’ve heard a few times is that people want to use generated AI content to serve as a first draft.

Joel
Yeah.

Kristin
And a lot of times, I think a first draft is how we determine what we think belongs in web content. So we’re outsourcing the actual thinking, and then we’re putting ourselves in the role of editing mediocre content.

Or maybe it’s better than mediocre. Maybe it’s average, but we haven’t put in the hard thinking to determine if it’s the right content. Even if it sounds relatively smooth, if it looks like it directs you to the right places on the site along your user journey, um, I think we still have to ask ourselves the bare question of, is this the proper way to start this process? And identify what a user needs in this specific space and time on your website.

Joel
Yeah, I agree. I, I think I, I’ve learned over the last year that I, AI works better for me when I apply it to something that’s already been created. Like I make something and then I say, “Hey, Tell me what I could do better with this.” That’s not to say that it can’t work the other way around, but I do think that to your point like this is, it’s not just a, “Hey, you figure out the thing that I want, that I want to write.” It would be best if you did that work.

I don’t think that work necessarily has to be writing an entire full draft, but it does at least have to be figuring out: these are the key points in this order with this sort of tone. It’s really fleshing it out and giving the LLM some some kind of, you know, like art direction or creative direction to go in.

And even then, you’ll still have to go back and edit it. I think you’ll get better than mediocre at that point, or at least, you know, reasonably good. Um, but I think I’ve seen and heard from most people that they don’t start at that point. They basically say, like, I need a thing about this, and they give it like a link to somewhere, you know, and ask it to write a first draft and then go on from there.

And you’ll get content. You won’t get good content, though.

Kristin
No. I would want to approach it a bit like; I’m unsure if you’ve seen the snowflake method in writing a novel. You start with the basic story outline, which can be as short as six sentences, but you hit the major points you want to take place in your story.

Then, you take each one of those points and flesh out the details. Then, you flesh out the details of what has to happen for these story points to make sense along a hero’s journey. And then you go deeper and more descriptive. In each level, you’re making your story more fulfilling, structured, and connected, giving your audience the necessary details.

So I think when you think about your student user journey, you want to think about the information they need along that journey, where they are in that journey, and at what point you want to supply them with that information. Then, you can start to work with either a GPT style, large language model, or other editing tools and features inside of AI that can give you some of those ways of filling out that detail.

But, what you want to be careful about is that they don’t take over your user journey because they could easily if you’re not feeding it the correct information in the order that you want it across multiple pages; they can re-architect the structure and the journey that you’re trying to send users on. They can change that user experience.