It’s been almost 3 years since Code School launched its Kickstarter to create a place for developers to start learning iOS. 3 years in most technologies is a long time, but for iOS that’s 4 major versions, 6+ different phones (see what I did there?), and 2 very different languages. And of course we’ve had to change as well, which is why we’ve revamped our iOS Path!
First, let’s start from the beginning — we’ve tried out different techniques for teaching iOS in the browser. In Try iOS, you got to see a simulator running your exact code — cool, right? And in our Core iOS Course, students download Xcode projects and complete them locally. When the tests are run, Code School’s notified when the student completes the challenge, and the student can even see instant feedback in the browser! We think both of these approaches are special, and offer an educational experience unlike anything we’ve seen out there. But we still noticed a few problems with these approaches.
The (Big) Change
There are a few issues with this approach in the areas of student adoption, completion, and overall engagement that have caused us to change our approach to how we teach our iOS courses. To start, we’ve changed a number of courses to be more video based, with multiple choice, fill-in-the-blank, and (new!) code-in-the-blank challenges. Why? Because we hope these updates will allow more students to be able to make it through these courses, and fully enjoy them. So for the updated courses, we rewrote all of the challenges from scratch. That’s right, every challenge is brand new! Below are the courses that were updated:
“But the simulator was so cool! And those projects?! Why’d you change it?” Well, there’s a good reason for it. Mostly it comes down to engagement — we want people to take these courses, learn, and feel empowered by the technologies. We felt like there were better ways to do this with these courses, and our end goal is to provide you with the best place to learn. So let’s take a deep dive into why we made this decision (complete with neat graphs).
Code School vs. Other MOOCs
Completion rates for MOOCs (Massive Open Online Course) have been widely discussed. Some reports peg completion rates as low as 2%, while other reports mention rates of around 13%. These courses are often completely wide open, and while it might be a good comparison for some of Code School’s free courses, it’s not something we can compare our overall curriculum to.
Why? Intent is the main reason. Many MOOC-related students are learning topics preemptively, before they need to use the skill. That’s how I typically use many of these courses as well. So for example, someone has a new job and they need to get up to speed on Angular. They play Shaping up with Angular.js and complete it in order to use the knowledge immediately. Because of this, even our courses with relatively low completion rates will be higher than most MOOCs.
Enough talk, let’s show some numbers! Of the 5 courses we’ve revamped, the most popular one is Try iOS — the first in our Getting Started With iOS Path.
Now the hard part — getting a good comparison. Some courses are free, but most are paid. Some have videos, while others don’t. Some can be completed in 15 minutes, while others average out over 6 hours.
When you look at courses in our library, Rails 4: Zombie Outlaws stands out. Both Try iOS and Rails 4: Zombie Outlaws follow a free course, have the first level available for free, and require an intermediate skill level to get started with. Now, let’s take a deeper dive into the data and compare these 2 courses.
Try iOS is 4 minutes longer, which isn’t too much, all things considered. The average course on Code School is about 66 minutes, so both of these are right around our target time.
But that’s only videos — what about in browser challenges?
Zombie Outlaws has 30% more challenges than Try iOS, which is quite a bit more. The reason? iOS code is a bit heavier than Rails code, so students get through more challenges in the same period of time as Rails. How long does it take to make it through both these courses?
This is the most telling data point so far. For those that completed Try iOS, they spent twice as long completing each and every challenge. Try iOS is broken down into 20 short videos, but doing 2 12-minute challenges could mean you’re watching a 4-minute video, then 24 minutes of challenges. With Zombie Outlaws, you’re more likely to watch a 6-minute video, then do 4 6-minute challenges — and, hey, that’s also 24 minutes. There’s a greater sense of accomplishment and engagement by completing 4 challenges than just 2, however.
The time spent in challenges versus videos stands out as well: 13% of time in videos for Try iOS, versus 18% for Rails 4. Fifty percent more time spent watching videos might not seem like much, but that’s time you can take a break from challenges and soak up new material.
Another thing to take into account is load time. On our Try iOS course, the average submission time is somewhere around 15 seconds. It takes much longer to evaluate code in our iOS course than anywhere else due to the simulator we use. So if we take the average number of submissions for each course into account, we can see how long each student spends in different parts of the course.
So the courses are different, but how does that affect students playing through the course? Well, one thing we look at is our course funnel — where people drop off within the course. If there’s a particularly difficult challenge, we’ll likely see a dropoff at that point. Here’s the Dropoff Point Graph for the courses we’re discussing, starting with Rails 4: Zombie Outlaws:
Almost 50% of people with a Code School account who start Rails 4: Zombie Outlaws will complete it. Some of those 50% are people with a free account who don’t end up subscribing, making this even more impressive. That big dropoff in level 1? That’s mostly because many free students don’t convert to paid students. The fact that 70% of people do, or already are, is a great sign, though. Let’s see how this compares to Try iOS.
It’s obvious something is different here. With Try iOS, the completion rate is closer to 4%. This is a stunning difference. Try iOS does get more free players attempting it and leaving after level 1. But still, we’re not turning our level 1 students into entire level students.
Although 12% of students make it through level 1, we lose 50% of people on the very first challenge. We knew iOS could be intimidating, but we’ve never seen anything like this. The dropoff rate of the very first challenge of Try iOS is the same as the dropoff rate of the entire Rails 4: Zombie Outlaws course.
We’ve seen just how similar these courses are — they’re targeting similar skill levels, they’re spending a similar amount of time solving problems and watching videos, but somehow one of them causes some people to leave. Why is this happening? We’ve been batting around a number of ideas about this here at Code School, and to the best of our knowledge, here’s what we’ve come up with:
The submission time for each and every code submission to Try iOS was always a bottleneck, and a painful one. Having a student spend 5% of their overall time waiting on the server isn’t a great experience.
More Conceptual Overhead
Quicker Feedback Cycle
This is directly related to load time. The faster we can evaluate your code and give you feedback, the better. In Shaping up with Angular.js, we show a preview with every key press. In our SQL courses, we run your code with each keypress and give very specific feedback when the student asks for it. The nature of iOS courses — running on a server rather than on the client — made this task much more difficult.
We think the content of our iOS courses is pretty great. And to make these courses more empowering for our students, we’ve rewritten all of the challenges for all 5 iOS courses in an attempt to solve the 3 major concerns mentioned above. We think it should be an easier and more inviting way for students to learn iOS development from scratch. We’re also still going to provide all sample code used in the previous courses to help people get something running locally as well. And of course we’ll be comparing how far students make it through the newly revamped courses to better understand how people learn iOS, and what we can do to be the best place to get started learning iOS development.
Ready to get a head start on iOS? Play our newly updated iOS Path right now! And don’t forget to let us know what you think about these updates, or which courses you’d like to see added to our iOS Path, in the comments below (tip: click “View Discussion”)!