What Noah Pryor Learned as CTO at Teachable
Noah joined in 2014 as employee no. 1 and served as Chief Technology Officer throughout his 8-year tenure and through a successful exit.
Welcome to What I Learned, a series of fresh “exit” interviews featuring talented startup founders & operators sharing their top lessons.
Our inaugural featured guest is Noah Pryor, a technology leader and startup executive who recently joined Beehiiv as Chief Technology Officer (CTO).
Previously Noah spent 8 years at Teachable. He joined in 2014 as employee #1 and CTO and stayed in this role throughout his near 8-year tenure. In his words, the role “changed pretty dramatically every six months to a year and a half or so.”
Teachable was acquired by global edtech company Hotmart in a reported 9-figure deal announced in 2020; Teachable had ~ 150 to 200 employees. Noah stayed on to lead the engineering and data teams, scaling them up to ~85 people. He left Teachable in March of 2022 to take time off before pursuing new opportunities.
In our conversation, Noah reflects on his experience as an early employee, technology leader, and executive at Teachable. He shares deep insights on scaling product & engineering teams, leadership, the creator economy, and more.
This piece is based on a live conversation between Anu and Noah. It’s been paraphrased and edited for clarity and brevity (reviewed by the guest before publication).
Lesson #1: How fast you ship product isn’t everything.
One thing we did well was that we had a really good philosophical stance early on that we were going to be the best platform for instructors. That helped us decide where to spend our time and scope things down.
Still, we built too many features. We would build one off integrations and be really excited and proud of the way a customer could ask for a feature and we could ship it in a couple days. But then we ended up with this enormous product footprint where we were trying to be a website builder and an e-commerce platform and a learning management system. And there are way bigger companies like Shopify or Squarespace focused on just one of these things.
We were also so focused on iteration speed that we missed underlying commonalities between many parts of the product. There were really just a few big moving pieces, but we built a ton of different systems that worked in different ways. For example, we had MailChimp and ConvertKit integrations that didn't really share any code even though they basically did the same thing.
I also think we could have built more flexible systems and done a better job of making things easier to rip out and replace. We weren't super conscious of something being everywhere in the code base so really hard to fix down the road.
Lesson #2: Just because you can allow customization, it doesn’t mean you should.
One escape hatch that we used a lot early on is we let customers do a ton of customizations to the front end of the platform on the course website side, sort of like a Shopify or Wordpress theme system. We gave people a lot of power to edit or build on top of our API and do custom content embeds.
That became this millstone around our neck where it was very hard to change the underlying product without breaking people's customizations. It made people psychologically not want to make changes to certain parts of the site because we would update the lecture viewer for example and we would get 20 furious emails.
We definitely allowed too much customization too early. I got really excited that I figured out how to let people customize all the templates and just rolled it out — that was not a good decision. We should have waited a few years or done more limited customization early.
Lesson #3: Know when to stop critical-path coding and become the context shield for your team.
The first ‘phase’ at Teachable was basically the first year and a half; the company was under 10 or 15 people and effectively a flat structure. Especially when working in person, everybody has close to full context on the problems facing the company and you don't have to do nearly as much of the “context carrying” that becomes a lot of management and leadership over time. So for that first year and half or so I was pretty heads down coding working on critical path features.
The next phase was when the engineering team was getting closer to 10 people. Early on the pattern was that I would stay up late writing code and then I would sleep-in and do meetings and management stuff during the afternoons, but that fell apart eventually. I think the big lessons I took away there, other than to stop making things go through you for features and being a blocker, was to learn how to be the context shield for the team. So at that point I switched effectively to pure people and product management. I love to code so I still did stuff outside of the critical path, but not stuff that would block a big feature from going out.
Lesson #4: Hire early for people who can and want to become managers.
When the company moved from around 15 to 30 people and the tech team was at 12 engineers, we first tried to break it into subteams. One of my biggest regrets is that I had hired really intensely for people who didn't want to be managers. I just wanted people who wanted to ship code and get stuff done, but then we hit the point where we needed to start having management, and we didn't really have anyone who had any desire or aptitude for it among remotely senior people.
So the first levels of management for engineering came from just making the backend engineers report to the most senior backend person and the front end engineers report to the most senior front end person. The idea was that you don't want to report to somebody who doesn't know what you're doing at all day to day, but then they were still effectively unmanaged.
We changed who we were hiring for, but that truly was a two-year fix. In 2018, I hired a VP of Engineering and a Head of Data. That helped a lot at the top level but the engineering team was about 18 people at that point. Within a year we had four pods with engineering leads that were kind of faux-managers, but not fully managing, and that didn't work well, there were a lot of conflicts.
We ended up bringing in a couple external managers which was a real shock to the culture. It created an old guard vs. new guard dynamic. We would have management meetings in one room and then the tech leadership meetings in another room and they wouldn’t overlap much. Or there would be friction between the two teams so things that should be talked about in one meeting would be talked about in the other, and the dynamics weren’t great.
So hire earlier for people who want to grow into being a manager. If someone was a manager once, and they’re willing to come in as an IC, then go back to being a manager, that’s ideal. That’s pretty hard to find, but it was one of the big lessons.
Lesson #5: Find the balance between too little and too much management.
The goal with engineering teams generally is for each team to have their clear area of the code base. You want to minimize the amount teams have to ask other teams for help, to solve the coordination problem. Give people areas to focus on and two or three really good metrics to focus on for that area. For example, ideally the team working on the student side doesn’t need to go ask the payments team for permission before they ship code.
At Teachable it had all been one big team early on so the interface between teams took time to become robust. Each team was cross functional — it would have engineers, ideally a designer or product manager, an engineering lead, and an engineering manager. And I think that was too heavy weight, that's too many people on top, and by the end you had a QA person too. We got to the point where you had four or five people on a team before anybody was writing the code, and I think that's too heavy. I still don't have a great solution for the line level engineering managers or lead combo. If you make it on one person, that person's insanely stressed out; when you have multiple people, it becomes very heavy.
Lesson #6: If you think it’s a ‘tools’ problem, it’s actually a people problem.
There's tools for managing the work, and then the tools for the programming languages you're writing in. We moved through a bunch of different project management tools. We started with GitHub issues because we effectively didn’t have anyone who wasn't in GitHub. That kind of fell apart once we started hiring contract support people and GitHub couldn’t just give read access to people. So we moved to Trello, and then Shortcut, which worked for a long time.
We eventually ended up moving to Jira towards the end of my tenure. Jira is fine, it's super configurable, but I think there's too many ways to f*** it up. I think if you have really specific regulatory requirements or an incredible JIRA wizard on your team, it can work out really well. I think having ticketing is a good idea. But overall, I think we did a lot of churn on tools so it was a pretty marginal benefit.
I think it's really easy to be in the ‘tool will solve our problem’ mindset when it's actually always a people problem.
Lesson #7: Take big bets even when you become a ‘big’ company.
It's easy to get scared of taking big bets in later stages of a company. I think it’s in part because having those 20 customers yelling at you feels like a lot even if it's not a huge percentage of your customer base. The early customer interactions really imprint themselves on the company's DNA and you can end up being guided by them.
For us, I think one big thing was the phrase “creator first” or “instructor first.” That meant one of the principles was that we will not contact your students. It was a hard rule we were really public about upfront because Udemy on the other hand was sending out 90% off coupons to students for competitive courses. This principle made us go slower and wait a few years before we moved away from it.
There are basically two things you need to make a living as a creator — 1) you need a course or 2) you need to have an audience and you need to have content. And these are both hard and different skills. So the biggest predictor of whether or not a creator succeeds (or does not stop paying us) is them getting a sale, and then they basically never churn.
We waited way too long to try to do anything that could drive sales to our customers because we didn’t want to do direct outreach to students. And then we eventually realized we should when 90% of our current customers were like “new sales? hell, yeah, I would love new sales.’”
Lesson #8: Set standards and limits early so you can measure against them.
This is more tactical — put in standards and limits early on things. For example, you can set performance budgets for pages while they're all still fast, when you don't have very much data. That's a lot easier than trying to retrofit them.
Similarly, put limits on everything that your users can give you. For example, we didn't limit the number of lectures you could add to a course, so the de facto limit was basically when the courses page stopped loading. So make it possible to change limits if people complain, but put boundary conditions on stuff early or you'll never really be able to predict the performance characteristics of your system in production.
Lesson #9: You can’t make a ‘creator’ be successful.
The big thing I always struggled with is that you can't make people be successful, it's very much on the creator. The downside of that is that tons of people don't succeed, and it is heartbreaking. It's really hard to find the right balance between promoting your product and saying this is really going to help you, you can do it. You don't want to convince people that they're gonna make a million dollars just to get their 200 bucks, especially because it's a really sympathetic customer base.
The thing that surprised me is how hard it is to predict who will succeed. We had a lady early on who would email the CEO constantly. She’d send us pictures of her dogs when they died to apologize for not sending us emails for a couple days. She’s this really sweet elderly lady and you think she’s not going to be successful, but then she made 50 grand. There was also a “copper deficiency in goats” course that was shockingly successful. It's really easy to get cynical but you don't know what people want, and you're continuously surprised by it which is the cool part.
Lesson #10: People buy courses for the people, not for the information.
The internet lets people connect on an individual level and build personal connections. I think people will continue to make a living off that and it will expand more. With courses, the information is out there, you can buy a textbook, but people are buying your course because they want to learn it from you. There's something about how you're teaching it or your background or experience or your spin that is valuable and makes it more applicable. I think people will continue to make a living selling directly one on one to other people or smaller businesses.
Lesson #11: There is such a thing as working too much.
One thing I didn’t do well — personally, I worked too many hours. Even up to the last year I was at Teachable, I would be leaving and an engineer would hit me up and be like, “hey, do you remember writing this code?” And I would look and it would be 3am on a Tuesday that I was writing it. You can really make negative progress after a certain number of hours working, by just writing complicated things that are very hard for people to unwind, so that's something I'm trying to be a lot more conscious of.
At the time I felt like the real management struggle was keeping people from working too hard rather than trying to crack the whip and make them work harder. Yet I didn’t make the connection until later that “oh people are working too hard, they should stop.” Obviously it’s hard to build a successful startup working 10 hours a week, but the first thing you lose when you're tired is your ability to judge how much you're affected.
Lesson #12: Build systems that are flexible, not perfect.
You're totally going to cut a lot of corners early on, and that's fine. Don't stress too much about that unless you have no idea how you can rip it out or replace it later. If the code lasts two years, that's great. If it lasts eight years, you either over-engineered it up front or you made it impossible to replace. Being able to change the system is really more important than the quality of any individual part of the system up front, except those things that are really hard to change.
Lesson #13: Stay on top of paper cuts.
A paper cut is something that the CEO isn’t going to call you about in the middle of the night because it's broken — things that often cause a support ticket and the agent can answer it and the customer is satisfied. It’s easy to not fix them and just hire more customer support. Eventually it makes your product feel not very well put together and it becomes harder to make changes. Paper cuts pile up; stay on top of them because you want people to feel proud of what they're building.