The CEO of Modal, Erik Bernhardsson, is a super well regarded former CTO. In this conversation, we spoke about his best practices for increasing developer productivity, a topic that he has thought a *great* deal about, as you will soon see. We covered:
What is the secret to developer productivity?
What are the common mistakes that CTOs make?
How do the best practices change over the course of a startup’s life?
How did these best practices inform the way he developed Modal to enhance the productivity of data teams?
It’s always fun to host a leader who has reflected so deeply. You’ll find some useful frameworks in this conversation. Feel free to listen to the audio version or else read the lightly edited transcript below. Let’s dive in!
Transcript
AP: Erik, I'm so excited to have you on the podcast today to talk about developer productivity. Thanks for joining.
EB: Thank you so much.
AP: What is your background and what led you to start Modal?
EB: I've been coding for a long time, since early '90s. I grew up in Sweden. I did a lot of programming competitions in high school. Then after school I ended up joining this early stage startup, an obscure company doing music streaming, which was called Spotify, and still is. I was there for seven years and built a lot of different things, most of them data-related. In particular, I built a lot of the music recommendation system.
Then I left to end up with a different company as a CTO called Better. I was there for six years, and built up the tech team. It was a wild story in itself. And then I left three years ago and I started building my own startup, which is Modal. It's a startup building a data platform in the cloud for data teams to build mostly AI related applications, but also all kinds of other stuff.
AP: I know you have a lot of opinions about developer productivity. What would you say were the formative early experiences that shaped your views on how developers can be most productive?
EB: I've shaped it in both positive and negative ways. Part of it is being at high growth startups and seeing a lot of the good stuff and a lot of the bad stuff. And being frustrated has shaped it more than seeing the good stuff.
I'd look at other teams and wonder why they are doing it one way when I'd read that you can do it this other way much faster. What brought me into management eventually was looking at how teams are run, how they're building code and shipping products. I felt like I had a lot of theories about how to do it faster. Eventually, got a job as a CTO where I almost felt like that was my way of experimenting at a much larger scale with this. But obviously then as a CTO, I thought about it constantly, which is the job of a CTO, basically.
AP: What is the secret to developer productivity, if you had to say one thing?
EB: One thing I've seen in practice is fast feedback loops. When I think about developers, how they write code, it's basically a series of different feedback loops. You write code, you get a syntax error, you fix it, and then a compilation error. There's a a second feedback loop there, a minute maybe.
And then it's a larger feedback loop of actually making the test work or implementing behavior correctly. Maybe that takes an hour or so. And then comes a larger feedback loop of actually getting this merchant to production, getting this ad in front of users or get feedback from peers on the pull request. And that may be a few hours, or a day or so. Then you actually get it into production to see what happens. Then there's another bug — crap, I have to fix it. But now we're talking days, weeks, months.
A lot of the developer productivity comes down to how to make those feedback loops tighter. How do you take something that's maybe three days long and turn it into something that's more like five hours? Then, how do you take something that's five hours and turn it into one hour? How do you take one hour and turn it into 20 minutes?
AP: As someone who was not a developer previously, I can see how that same secret actually would be the secret to productivity of a lot of different roles. How quickly can you learn how to do your job better? If you're a sales rep, get feedback from your manager, your customers, your marketing teams, so that you can evolve your messaging faster. It's a learning loop.
EB: Absolutely, and it's funny you mentioned learning loop. If you look at the U.S. military, they have a similar thing. You build these feedback loops and you learn from the enemy.
In general this applies to almost anything with high uncertainty. If you're doing the same task over and over again, such as working in an assembly line, I don't know if feedback loops matter that much. You just have to have high throughput. But anything where there's uncertainty, you're operating on unknown terrain. If you're trying to learn from users, or you're integrating with vendors, they have undocumented features. You just have to embrace uncertainty and build a process that deals with the fact that you don't know. You just have to learn along the way.
AP: I love that. So how do you create fast learning loops at your company?
EB: One of the things I tell every company to do, is continuous integration and continuous deployment, which is a little bit scary. What I always argued for is once you merge something into master or main, it should go live in production in 20 to 30 minutes. And having an absolute commitment to that is scary for a lot of companies. But these days a lot of startups have shown that you can effectively do that and maintain a really high quality bar. You can build a process that allows for you to do that in a way where yes, you will occasionally release bugs into production, but instead of thinking about that as an absolute thing you can never reach, you instead build a process that captures those bugs very quickly. Once those bugs happen, you build a process that internalizes and learns from those things, and then extends the test suite to capture those things so it doesn't happen next time.
So continuous deployment and continuous integration, having a lot of automated testing is absolutely critical. That's probably the biggest thing.
AP: Are there certain products that you recommend using as part of that?
EB: GitHub Action is probably fine for most teams. It's the least bad product in my opinion. I don't know. All the CICD products that I've tried, I've had issues with. My crazy idea is that I always want to build my own eventually. Just because I have my pet peeves with all of them. But GitHub Action today is nice.
It's very important to just have a commitment to merging things quickly into production, not having long-lived requests and having things sitting behind feature flags, or unreleased code. It's more of a practical thing, but you need to have a lot of parity between local and production environments, so that when you're writing code locally where when you're building things, you can effectively replicate a production environment to a large extent. That way you know that once you get something working locally you can get it working in production.
This is a very core thing about Modal too. It's our approach to just always run things in production, because with data teams it's actually easier to do that. You just basically turn the production environment into the test environment.
Obviously you shouldn't reinvent the wheel. You should use vendors for almost everything you can, which today is the art of software engineering. In software engineering in the '80s, you had to build everything yourself in assembly. But today there are so many products out there. Why build something that a vendor has effectively solved for you? Part of being a CTO today is vendor selection and deciding if you're just going to buy some product instead of building.
AP: Do you have a certain framework that you use for deciding whether to buy or build?
EB: The rule of thumb is to see if the solution is part of the company's core competency, or the secret sauce of the company. Is it something that brings you a competitive advantage? For example, if you're dealing with billing, payments or invoices, that's not a core competency for most companies. They should just use Stripe.
With modern repos today, the way to go is not having a lot of small repos, but having one big repository with all the code. Google does that, Facebook does it, you can do it at scale. It's a little bit annoying for some stuff, but it actually makes your refactoring a lot easier.
And in similar terms, don't prematurely build for scale. Don't prematurely break things up in a lot of micro-services. There's a lot of urge for companies to split things up and platformize things too early, but you have to hold back. It's fine sometimes to keep things a little bit monolithic and have one big code base for everything. Over time you're going to start seeing things emerge that you can refactor out. But generally people should lean on the side of waiting with those things as much as possible.
AP: Given everything happening with AI, when most people think about developer productivity, they think of CoPilot and code generation, which seems to have dramatically on average reduced the amount of time required to build things. How are you seeing it used at Modal or among the people that you know?
EB: At Modal, we don't use it that much. Some of the engineers use it. I am just old, so I struggle with new tools, but I don't use it at all. But some people like it internally. I tend to think it's probably going to make developers, 10, 20, 30% more productive. If you look at engineers during last 20 years or 30 years, they've already gotten 100x more productive. It's great. It is another tool. Add that to the toolbox of tools we already have to make engineers more productive.
The long arc of software engineering, since we started doing it in the seventies, has always been that we keep getting more and more productive every year through new tools. And these things multiply, which is amazing. You get exponentially more productive every year. At the same time, there's actually more developers in the world, which is crazy.
I expect that trend to continue. Developers are going to get more productive. That's going to mean it's cheaper to build software. That's going to mean the demand for software engineers goes up. There are going to be more developers in the world. That's my optimistic thesis about software engineering.
AP: Really interesting that you see LLMs as a nice, but maybe only incremental, improvement to developer productivity so far, along the lines of the way other tools have impacted it. I wonder also if you think there might be a risk to developer productivity from LLMs, because what's essentially autocomplete can actually, in my experience as a writer and a thinker, prevent you from thinking deeply about the problem that you are trying to solve. Editing something that's already been created for me prevents me from thinking from first principles about what I actually want to say, and getting to the root of, "What is the idea in my head, what am I trying to say?" And therefore, the output is just not as good. And as a result, ROI on the time that I spent writing that blog post is not good. And it might've been a lot better if I just started from scratch and taken a couple extra hours. So what do you think about autocomplete as a potential hindrance to people being productive?
EB: Your personal experiences, things that you'll just learn over time, whatever makes you most productive is whatever workflow you'll probably adapt. When there's new technology, there are fears. And we're going to adopt in the wrong ways. I remember when the internet came along and there was a fear that people are going to stop thinking for themselves. They're just going to look up everything online. Which is true, about some people.
AP: It's true. I used to have a far better memory. But now I almost have lost the ability or interest in memorizing because I can just Google anything.
EB: There's probably a risk. Everyone has an old uncle or whatever who's just only reading stuff online and thinks it's all true. But with the younger generation, most people learned that it's like a bicycle. It helps you move faster. It helps your brain move faster. Using the internet as a tool, you can be more productive. In the long run, I still think these are all good things. There are short-term risks or there are wrong ways to use every new technology. But in the long run we'll learn how to leverage those for our productivity.
AP: I'm thinking about you as a former engineering leader advising other leaders on best practices for maximizing developer productivity. So maybe we can start with what are the most common mistakes that you think CTOs make in trying to maximize productivity?
EB: A lot of different things. As I mentioned before, productivity is one of those things that adds up on a logarithmic scale. So, if you do 10 things right, and you get a 20% increase in productivity from each one, that’s 1.2 to the power of 10, which is pretty substantial.
People are sometimes very paranoid about bugs. I mentioned my commitment to CICD. One of the things I've learned is, fear of bugs is actually a paradox. It sometimes makes bugs more likely, and ruins developer productivity because people are so afraid to do something like continuous deployment, deploying the production very quickly or breaking things up. It's a small incremental thing that gets merged. They impose a lot of task overhead. They want to have a lot of processes. They want every code to be reviewed by three people. I'm not really sure that actually does prevent bugs.
We have a commitment to merging things quickly and getting the product in front of users. Occasionally there are going to be bugs. But when those bugs happen, we learn from those bugs and prevent the future similar class of bugs. That ends up not just improving the quality overall, but it also vastly improves the developer productivity.
Then you can just iterate much quicker with a higher confidence. We have the assurance that we have automated testing, monitoring and alerting. If something breaks, we don't have to rely on a human to look at the code to find it. We can rely on all of these systems we've built that are automated. That's the most important thing. Committing to CI/CD early on, and just building all the test processes and all the automated testing that's required to keep quality high.
At slightly bigger scale, one thing I see is that companies are very nervous about platformizing. Or not nervous about it exactly, but they tend to do it prematurely. They look at all these different teams and they see that some teams have the same concerns. They think they should break it out into a platform. It comes from a good place, but doing it too early is very dangerous.
Now you're introducing dependencies between a lot of different teams, and you're creating this new team that's not actually incentivized to work on user features. And so to a large extent, up to at least a hundred people, it's actually better to only have fully vertical teams. That is, you have vertical teams that only focus on specific user problems, and very little internal platform teams. It obviously depends on what product you're building. If you're a deep technical infrastructure company, that might look different than if you're building a e-commerce product.
As much as possible, having teams focused on vertical features is very important. Software is inherently embracing uncertainty. It's almost like research, by definition. Because if it wasn't uncertain then a big company would just do it.
If you look at what a startup does by virtue of being a startup, it means you're building something that other people think is impossible. Which means there's a lot of uncertainty. Which means you can't really plan out front. You have to be a little bit exploratory. You're going to start working on something and then realize it's not a really good approach. Then you'll abandon this direction and go in these other directions.
And a lot of companies want to have more control. They want to have more visible roadmaps. They want to have longer commitment. They want to put five people on this project and know it's going to take X months. Ironically, trying to have all this control over what everyone's doing means you're actually choosing not to do the big uncertain stuff, which is often much higher value.
So it's more important to embrace that uncertainty and have a planning process that's very nimble and very adaptive. I'm very biased towards startups anyway. When I talk about these things, I realize I'm a startup degenerate. I've worked at startups for 20 years. So, some of this may not translate directly to very large companies, but when you're building the startups, it's so important to embrace this uncertainty, plan for it. Or not plan for it.
AP: I love that framing. Let's talk more about developer productivity at different sized companies. In general, I find that there are three stages of a startup. There's early on when you're searching for product-market fit. Then there's building repeatable go-to-market motion. Then there's scaling that repeatable playbook. I didn't come up with these three stages, a lot of folks have talked about them. I'm wondering if you see different tactics for driving developer productivity in each of those stages.
EB: It's actually quite different. In fact, sometimes it's like opposing forces. What's good at one stage is actually really bad at another stage, and vice versa. At the very early stages you should have almost zero planning. Just hire a bunch of people who have a deep commitment and deep understanding of what the company's trying to do. And then just lean back and let them self-organize. If you find the right team at a stage of ten, they can deeply internalize what the company needs to do. You can tell them the big priorities.
This may seem a little aspirational and maybe naive. But what I found is that you can get pretty close to this ideal of letting people self-organize if you hire the right people. People who are very entrepreneurial and very commercial.
At this stage it's also very important that you lay a foundation. Committing to CI/CD for instance, is something that's very important to do early on. Commit to incremental changes. Commit to having the right production stack that makes it easy to move fast with a reasonable level of confidence. Obviously once you get a little bit bigger, it changes. Once you start having 15 or 20 engineers, you need to start having teams. That starts to make things a lot harder because now people in each team don't necessarily understand the whole picture anymore.
That being said, the same principles apply. You have to be very careful about introducing process overhead. At this stage, you might start using a task tracker, which is maybe good. But don't necessarily take this task tracker to an extreme, where everything has to be documented and go into a centralized backlog every three weeks. Those are the things that destroy developer activity.
If you have only about 20 developers, they should still have the ability to think of things and apply a quick fix. You want to have the absolute minimum amount of process for them to do something that's obviously good for the product. You've got to resist the urge to introduce too much process.
Once you start hitting 100 engineers, that's where things get very different. You should start to think about platformizing, not specializing. Start to think about building internal platforms, hiring specialized people, having a little bit more of a process for planning. Be more data-driven, have more KPIs and OKRs. A lot of these things end up shifting when you hit the Dunbar number of 150.
AP: Erik, you're an awesome podcast guest because you've come with all these frameworks that you've clearly have been thinking about for so long. This was awesome. I'd love to just pick your brain on hiring. How do you hire to ensure that your teams are productive?
EB: It's always the same at almost every stage, up to maybe a hundred or a few hundred people. You always want people who are very scrappy and very commercial who create a culture of curiosity about understanding what's needed. When you look at ICs and developers, that's probably always true.
That being said, at early stage startups, you probably want to bias more towards generalists. You want people who will know a little bit about everything. As you mature, you need to specialize a little bit more. Early on you shouldn't even have front end and back end people. You should have full stack people.
But once you start getting 15 or 20 people, then you could have a little bit of specialized people. Maybe you can hire a front end engineer and maybe you can have a mobile engineer, you can start to split things up a little bit.
AP: How do you get a sense of whether an engineer that you're interviewing is likely to be productive individually?
EB: That's an incredibly hard assessment problem and something I can talk about for hours. Interviewing is inherently a very noisy assessment problem. We're going to spend three hours together. I'm going to try to predict whether you're going to be successful at this company in the future or not. That's just near impossible to do well. it's more of an empirical science, frankly.
You focus on people who first of all have a base level of technical competency. That's very important not to skip. You need to make sure they're generalists and they can write code in different languages. And they understand basic system design. We also want to understand people's projects that they worked on in the past. You'll want to know about the most complicated thing they've ever built. What did they actually do, versus what did their team do? Who came up with the idea? Was it them or was it someone else?
I like to dig into those things and try to understand if this person cares about building things. Have they come up with their own ideas? Have they made significant contributions in the past? There are many questions like that where you can dig in and figure out who actually is willing to roll up their sleeves, crank out code, and build amazing stuff.
AP: Erik, you've obviously thought a ton about how to make developers more productive. You are building a product for data teams now, and I'm sure that you're thinking through how to apply these best practices for data teams too, to make them more productive. How are you translating those best practices using your product?
EB: The reason I ended up working on Modal is that I started thinking about how to make data teams more productive. Modal is a very infrastructural product. I've worked on a lot of different parts of the stack. And data teams, especially versus other types of software engineering teams, have been a bit behind in terms of having the right tools and the right workflows.
Front-end engineers have amazing tools these days. You pull up the code on one monitor and pull up the website on another monitor. You have all this hot reload. You can iterate very quickly if you're a front-end engineer. And now they're starting to have their own tools for sale and specialized vendors that focus on front-end engineering. You could say the same thing about back-end engineering.
When you look at how data teams operate, there's been a lot of friction. They've been held back in terms of the value that they deliver. They have a lot of ideas, but it's hard to get things in production. It's hard to iterate. I started thinking about that problem three or four years ago. It's about getting code in production, and dealing with a lot of the idiosyncrasies of data teams that they're dealing with. They often need to use specialized hardware like GPUs. They basically have to run things in production at all times, because in production is where the data is.
They're often dealing with weird academic code, especially with AI. There are so many different environment installation problems. So, you have to embrace containerization. But on the other hand, data teams don't really like using Docker for various reasons. You have to deal with large fan out, running things like a very large scale for brief periods. Kubernetes isn't quite right for data teams. Docker isn't quite right. And I started thinking about what I can build for data teams that's better than all these things.
Data teams want to take code that they have on their local computer and run it in the cloud. And I started wondering, how do I make that as fast as possible? How do I make that feel like you're writing code locally? How do I make this feedback loop super tight? And that actually turns out to be this very deeply hard, technical problem.
You need to take code, put it in a container, and launch it in the cloud very quickly. So, we started building that and had to go very deep in the rabbit hole of Linux, file system, page caching, containers and more. And we built something that now enables people working in data and AI to iterate very quickly, ship value very quickly, and get things out in production very quickly. And now we're thinking about how to build the rest of this product. We laid the infrastructure layer. How do we build the rest of the things, to truly make the whole toolbox platform for data teams to be as productive as other types of software engineers?
AP: Any last tips for engineering leaders to find maximized productivity in their teams?
EB: If you don't do continuous deployment, you should absolutely do it. It's a little bit scary at first, but just commit to it and think through all the ramifications. Better test suites, easier testing and isolation, more alerting, more monitoring, easier roll backs, easier incremental changes. Once you think through all the things that are needed for continuous deployment and commit to all those things, that's by far the biggest thing for developer productivity at most companies.
AP: Erik, thank you so much for joining us today.
EB: Thank you so much. It was great.
