The guys talk about strategies for creating systems, documentation, and automation to separate yourself from your business so you can hire employees, get it ready for sale, or even take some time off. Balancing good customer service while being efficient with your time is also discussed along with reasons Honeybadger doesn't use automation for customer service.
Full Transcript:Starr:
00:01 I probably should have muted that, so you couldn't hear the toilet flushing.
Josh:
00:05 I don't know whose it was, so... You just needed... There, there you...
Josh:
00:11 Okay, so, Starr. There is, there's your intro.
Announcer:
00:16 They're just three amigos making their way in the crazy old world of software as a service.
Announcer:
00:22 Welcome to Founder Quest!
Josh:
00:26 Oh, that reminds me, I was going to... during that decision-making thing, I was going to say we'd be a lot cooler, though, if we used a blockchain, like to decentralize, since, you know, we're totally like a, you know, a remote, very decentralized company, like we should have a blockchain for a decision-making process.
Ben:
00:42 For real, scan that audio trail.
Starr:
00:43 Yeah.
Ben:
00:44 To make sure that-
Starr:
00:44 The future. The future's now.
Ben:
00:46 ... make sure that Starr doesn't go back and change the decision that we made?
Starr:
00:52 What?!
Josh:
00:52 Uh-huh (affirmative).
Starr:
00:52 Why am I getting this flack?
Ben:
00:53 Well, you know, because you know that I would be the one that would actually be doing that sort of thing, so that's why I'm the...
Josh:
00:58 Well, it's more to protect against Ben, yeah, so it's like a digital gavel.
Ben:
01:01 I'm the totally random element in this outfit, that's for sure.
Josh:
01:05 Yeah. The wild card.
Ben:
01:07 Yes!
Josh:
01:08 The joker.
Ben:
01:08 The joker!
Josh:
01:10 And plus I could buy more video cards, so that I have more weight, my decisions have more weight.
Ben:
01:14 There you go.
Starr:
01:16 All right, so let's catch people up. So last week, we talked about some issues involving systems, like what are our systems for decision-making? And we talked about our quarterly conclaves, our process for doing that, and so this week we're going to be talking about systems and continuing the conversation, this is one long conversation that's just been split up into two.
Starr:
01:36 And we're going to be talking about managing employees, we're going to be talking about daily operations, about ops and all that stuff.
Starr:
01:45 Yeah, let's get going! So like what... we started with nothing, we started with no systems, errors would come in, and Ben would see them and he would manually write out an alert email, and send those out on Gmail. And since then, we've like, we have systems out the, uh... I can't say it on iTunes, I'm sorry, but we've got lots of systems!
Starr:
02:07 So how do we coordinate a bunch of, like three of us are independent workers, we've hired a bunch of independent workers, like how do we coordinate between those?
Ben:
02:15 I think that the technical term you were looking for there was "wazzoo."
Josh:
02:17 Wazzoo?
Starr:
02:18 Oh, okay, most definitely was it.
Ben:
02:20 You know, one of the things that was really crazy early on was, it accelerated so rapidly. Like, I remember, in the early-early days, when we first started this out, and most of the day I was thinking, you know, because we had jobs, so Starr and I were working for a start-up, and I was thinking, "Ah, this should be great! I have two incomes streams, right? Like I have my day job, and then Honeybadger just will be doing its thing on the side, it'd be a cash machine, it'd be awesome!" And then, it didn't go that way. Like-
Josh:
02:51 Then reality struck.
Ben:
02:52 Yeah, the reality struck, where, like, Starr and I are sitting there, at our day jobs, and all of the sudden Honeybadger's on fire, and it was like, "Oh, we got to go take a lunch break right now!", you know?
Starr:
03:00 Yeah.
Ben:
03:01 And so, like eventually, that just... its like the pressure was too much, right? We couldn't do both, and so we had to dive in on Honeybadger. But a lot of that was because things were just growing so rapidly, and traffic was coming in, and things were falling apart, and like in that one server that we bought initially, right, had to become two, and so on. But-
Starr:
03:21 Yeah.
Starr:
03:21 And we did not build this thing for a scale, people. We did not prematurely optimize.
Ben:
03:25 We did not, no. We definitely did not.
Ben:
03:28 And, so, you know, a lot of the, you know, that people joke about, the bailing wire and duct tape, and the changing the engine wire going down the road kind of thing, and that's definitely what we were doing, and that was high-stress. And over time, it became just too much. Like I was totally getting burned out.
Ben:
03:45 And so I was talking to a friend of mine, who also has a start-up, not exactly the same kind of operational burden that we have, but he had a lot of customers who were smaller, and so he's also dealing with some of his own constraints and his own pressures for scaling his business, and one of the tips that he gave me was, when you do something, document it. Like, don't just fix something, like if you fix something, document what happened, why there is a problem, and what you did to solve it. And then if you find yourself like having to deal with that again, automate that. Like, so, step one, do it. Step two, document it. Step three, automate it.
Ben:
04:23 And when I really got in the groove with that idea, that just changed my world. I got into the point where I was actually like documenting things that I was doing, so that if I was away for a day or whatever, you two could like read my documentation, and find out where things were. And then, as time went on, automated so that you didn't have to care, because it was just taken care of. And that was just... that changed my life, so I just had to throw that out there.
Josh:
04:48 Yeah, and we've brought that to basically every other part of the business as much as possible. Like we're all about documenting everything and automating anything that can be automated, so, yeah, it's not just operations at this point, like we're trying to do that for whatever's possible to that with.
Ben:
05:06 One of the side effects was, and aside from me being able to like, go away for a vacation-
Josh:
05:12 Yeah.
Ben:
05:12 ... which was really nice. But the other side of it was, just feeling more comfortable with where the business was as an entity separate from ourselves. Like we mentioned briefly in the last episode about how we wanted the business to be able to operate independently from us, so that we could go away, and that can be a short-term thing, like a vacation, or could be a long-term thing, like when you want to sell the business, right?
Josh:
05:32 Uh-huh (affirmative).
Ben:
05:32 Or when you want to replace yourself with employees. Like, in the early days, we couldn't hand anything off and not think about it, because none of us had documented, it was all in our heads, we were doing our own things that we knew how to do but nobody else knew how to do them, and over time, as we did document that, and automate that, it was much more conceivable for us to actually bring on an employee to do X, Y and Z, because now we had it pretty well planned out, it wouldn't be just like a discovery process, like "Oh, good luck, figure how to do it!", you know?
Josh:
06:01 It's kind of interesting to me, hearing like the documentation automation story from the operations side.
Josh:
06:07 We talked about, in the last podcast, we talked about, you know, like the books about scaling and creating business systems and business systems are basically the same deal. Like most of those books talk about, you know, like you document, and then you build the systems, which you're kind of like usually like checklist or something, and then the automation comes through having, you know, getting other people to do that, do those things.
Josh:
06:29 And I think, you know, we have tried to do that to some extent, but as developers we kind of, we try to prefer like, you know, getting computers to do those things, or outsource services whenever possible, because we can just, you know, if we can program something to do that, it kind of just fits our, you know, our small lean style better.
Starr:
06:48 Totally! So what are some of examples of, like we're talking kind of generally.
Starr:
06:52 So what are some specific examples of things we did that we no longer have to do because they were documented and subsequently automated?
Josh:
07:24 One of our classic examples, and this might be on the tip of Ben's tongue too, but is a, and I know we've talked about it before, is Printfection and how we've automated, you know, our swag to fulfillment. Because, I don't know about you guys, I was getting really tired of like going out and buying envelopes and folding shirts and putting shirts in envelopes, and then, you know, taking the envelopes to the post office to mail to people.
Josh:
07:25 And now, you know, we basically like, we have documented how we were doing that, and then we found a service that actually like can let us do that through an API, which is kind of like the best thing in the world to me.
Starr:
07:37 Can we get like a sponsorship from them? I feel like we really should. I feel like we're repping them more than we're repping our own company.
Ben:
07:45 Here you go! A specific example from our pipeline, over the last little while.
Ben:
07:51 So we used Elasticsearch for indexing the notice payloads that come into the system, so that-
Starr:
07:58 Yeah, the notice payloads are like the main sort of data in the error that happened, right?
Ben:
08:03 Right, right.
Ben:
08:04 And so we index, like where the error happened, and who it happened to, and all that stuff is in like a really big Elasticsearch cluster and it works great, like 99% of the time, but there are cases where we might try to submit a batch of payloads to be indexed by Elasticsearch, and it returns an error, and the whole batch just fails.
Ben:
08:28 And our batch is typically like a 100 payloads at a time, and there might be a case where, you know, Elasticsearch is taking a little siesta, and a few hundred payloads just don't get indexed right away. And so we have a system that kicks information about that particular error into Slack. It's like, "Oh, okay, Elasticsearch had an issue, and these payloads did not get indexed!"
Ben:
08:51 And when I first implemented that system, like that was it, like it would notify the Slack channel, and I'd have to go in there manually and find those IDs of those payloads that did not get indexed, and go back and manually tell Elasticsearch, "Yeah, go index those payloads!" And that worked fine, initially.
Ben:
09:08 And then, over time though, you're don't want to be doing that all the time, so the next step was, "Okay, we know which batches did not get processed, so let's record those batches in a database," in our case we're using DynamoDB for this, and then whenever we detect that some batches fail, like when records show up in that DynamoDB table, we know there is a failure. We can just use that table to drive Elasticsearch and remind it, "Hey please, hey, go re-index those payloads," right?
Ben:
09:37 So, that's where we are today. So now we don't have to use a Slack channel, like we can say, "Okay," and we can pull those records on the DynamoDB table. And then the next step in the automation, which I haven't done yet, is "Okay, well, we know that these things can be re-indexed, maybe a few minutes later, so let's just have an automated process, that once it sees the record show up in DynamoDB, like it waits five minutes, and then tries the reindexing, keeps on trying that until it's done, right?"
Ben:
10:02 So that's one example of something that we've done to like, something that I did manually, and then I did document "How do you actually do this restoration process?" And then, we're working on the automation part.
Starr:
10:13 You know what this really reminds me of? This really reminds me of the whole like, lean start-up approach to building products for other people, where you start with a minimum viable product, or maybe that minimum viable whole product is like, you responding to e-mails in a way that looks like a computer, and then, based on what you learn from that, then you, you know, slowly automate it and build it into a real product. Only, you know, you're the customer and the programmer.
Josh:
10:55 Back on the business end, what about customer service stuff? Because I know that we have a lot of customers, who, you know, got customer-service requests, they got pretty repetitive up about it over the years, and a lot of them might we've been able to completely eliminate by creating like self-serve tools, and, you know, things that allow customers to help themselves, and sometimes that is like an actual tool that we build, sometimes it's actually just putting, you know, writing some extra documentation that answers that question preemptively, and you can kind of look at documentation as, you know, as far as, customer-facing documentation as a kind of as a system in itself.
Starr:
11:14 That's a good point! Because when you mentioned customer service, I got this little lump in my stomach, because I was like, "Well, he's going to talk about how we don't really have canned replies or anything," we've talked for a long time about having canned replies that we can sort of just send off to people, but it hasn't really worked for us very well, because we get a lot of questions along the same lines but they're, you know, they're all very specific, you know?
Josh:
11:42 Well, yeah, and we don't want... a lot of them are specific, but some of them you could, you know, you could answer them with a click or-
Josh:
11:50 canned reply, but we really want to help people. I think that helping people is really important in our support, and we don't want, you know, if we can give people a more specific answer, just something more, like, a creative solution, I think that we're not really happy with just a canned reply that we would have for everyone.
Starr:
12:12 What I was saying is that that's not the only way you can automate this customer support. You can actually kind of do it around the back way, by eliminating the reasons why people are contacting you for customer support. And if you can do that by, say, putting a link to the appropriate documentation right next to the thing in your user interface that causes people problems, do it that way, or maybe you can make a tool that automates something they're asking you to do manually.
Josh:
12:37 Just to be clear, I wasn't saying like, "We don't want to be personable," I was saying, like, as a developer, I can spot canned replies from a mile away, personally, and they piss me the hell off. So-
Starr:
12:49 Well, I mean-
Josh:
12:50 as a customer, since we have developers that are our customers, you know, I want to be able to help them and give them a good, you know, an actual solution versus an "Oh, yeah, here's the, you know,"
Ben:
13:01 You know, that kind of mentality, like we are who we serve, right?
Ben:
13:07 That actually guides a lot of the decisions that we make. Like, the help desk system that we use. I remember when we first started and I was looking at the options for what kind of support system we were going to use. There were ones that had this autoreply feature, which is, you know "Hey, we've received your request, your request is number blah blah blah, we'll get back to you, you know, shortly," and that drives me crazy. I hate that! Like, I hate this automated thing that says, "We got your request,"
Josh:
13:36 Yeah.
Ben:
13:37 And I understand why some people like that, you know? It's like "They didn't get lost in the ether, blah blah blah," but I'd rather just like respond to people, like me, you know?
Josh:
13:46 Yeah.
Ben:
13:47 So we ended up not choosing one that had that feature because I didn't want that kind of experience for the customers.
Josh:
13:52 Yeah, I mean, we, to be fair, we do have a couple canned replies, and a couple of them are very, extremely useful because we have a few like highly technical edge-cases that, if you're going to help someone with this issue, you know, you're going going to go spend an hour digging on it. You know, JavaScript source maps is a pretty hairy one that comes to mind.
Josh:
14:13 And I have a canned reply for, specifically, for those types of issues, which like, you know, is very, it lays out all the different things that they could do to potentially troubleshoot their issue, just because the issues are so, there's such a variety of potential solutions to it. And I think that's pretty helpful for the rest of you guys, who don't really want to have, you know, the experts in the Source Map V3 spec that is currently published in a Google doc, by the way.
Ben:
14:41 And the other thing that we've done, you know, regarding the customer service stuff, is there are some things that we just have to do on our end to fix things, you know, right? Because sometimes data gets out of whack and there's nothing that the customer can do, it's on us, right? So we've documented-
Josh:
14:55 Yeah.
Ben:
14:55 ... "Here are all the steps that you need to follow when someone asks about this particular problem," right? And that's been very helpful-
Josh:
15:01 Yeah.
Ben:
15:01 ... I think, and sharing our load and making that more efficient.
Josh:
15:05 Yeah, so we've use documentation in our support process to help us provide better support, but we do it with standards, I think is.
Starr:
15:12 Yeah. I mean, life would be a little bit easier if our customers didn't have such like, sensitive bullshit meters. Right?
Josh:
15:21 Yeah.
Starr:
15:21 Let me find a target market that's a little bit more gullible next time, maybe?
Josh:
15:27 Yeah, but like... be a little more hand-wavy. Yeah.
Josh:
15:32 But yeah, I can think of, I think a number of cases where I think Ben has done a lot of this, but yeah, automated a solution, or built something into the app that lets people do a process that they were asking us to do before.
Ben:
15:45 Yeah.
Josh:
15:46 Like transferring projects, for instance.
Starr:
15:48 Yeah. Because a lot of customer support is, at least for us, is really just sort of like a band-aid over something that the app itself is missing, or like the user interface is confusing in some way, and so the real solution, you know, is to fix the user interface.
Josh:
16:03 Yeah, yeah. I'll say that really we have like two, there's two types of typical support requests, and that's it's like, "Something's missing," they just want to do but they can't, in the interface, and then there's super technical like, "Let me help you debug your app" type of thing. And we've tried to get a little better at like, saying "no" in certain cases, because in the past we've been tempted to like...
Starr:
16:28 Yeah, "Let's talk about this!" Let's go into this. This is interesting!
Josh:
16:31 You know, like, jump on a screen-share or something, and you know, pair a program with them to... I'm exaggerating, we haven't done that.
Starr:
16:38 But I mean, we-
Josh:
16:39 Ben might have!
Starr:
16:41 We've figured out some intense issues for people, right?
Starr:
16:44 Like, what are some of the issues that we've like helped people debug?
Ben:
16:46 Well, there was that one with Rational in the Ruby World, right?
Josh:
16:49 Well, yeah, that's the... that was the math, the math-
Ben:
16:53 The math engine, right. Or-
Josh:
16:55 It was a standard lab.
Josh:
16:56 Well, the math, I'm trying to remember, so the math and gem, or not gem, that was a standard library thing in Ruby. It's not required, it's an optional thing though, and if you require this math, if you require math and, and this isn't in Ruby anymore, thank God, because they took it out I think in like Ruby 2... I forgot what it was, like...
Ben:
17:15 2.5 or something.
Josh:
17:17 Yeah. Recently! Because I saw it, and I was like, jumping for joy.
Josh:
17:23 If you require this thing, it changes all decimal numbers to rational numbers or something, or...
Ben:
17:29 It changes division.
Josh:
17:30 Division, that's-
Ben:
17:31 It gets you returns as a rational number instead of-
Josh:
17:35 Yeah, instead of like floating point.
Starr:
17:37 So normally when you divide, you get a number that's like, 1.125 blah blah blah, repeated. But then, like a rational number's is like a fraction, right?
Ben:
17:46 They're like, 22 over 7.
Josh:
17:47 Yeah.
Josh:
17:48 And in Ruby is like an entirely different object, it has totally different, you know properties on, you know, how it interacts and like what it converts to, and-
Ben:
17:55 And it blew up our gem.
Josh:
17:57 It just... yeah. But it blew it up in such an obscure like, just like very subtle way. And this was back when we were doing metrics collection, by the way, I think it was in the metrics service when we were doing some like statistical calculations, and it was like, it wasn't in a way where it was like an error, it was like one of those errors where it's like, it's not an exception, it's not blowing anything up, but it's giving you the wrong value, and you just cannot figure out, like why you would be getting this value back, because it's like... there's no rational explanation, pun intended.
Starr:
18:36 Yeah, and I remember we've had some other cases where people include gems that redefine certain library standard methods.
Josh:
18:42 Yeah. And then, yeah monkey patching, because that's basically what math-end was doing, monkey patching in Ruby, is when they're basically rewriting, you know, rewriting things dynamically at runtime, and it's incredibly difficult to find where those things are unless you're familiar with the application. And in a lot of cases, like we've actually, like, you know, like stepped through that thing and, you know, that process in people's apps, which I think is well beyond what most companies are willing to do.
Starr:
19:09 So these people are paying us like maybe as low as 50 dollars a month.
Josh:
19:14 Yeah, I was going to say, like I've never had such a low hourly rate.
Starr:
19:18 So 50 dollars a month, and we're like, you know, sometimes spending hours on customer service. So that's something we're trying to get a little bit better at.
Josh:
19:25 And we have!
Josh:
19:27 Yeah, we've gotten better at that. And like I said, as we've... I think it's helped us to some extent, though, that we got in deep into those issues for a time, because it's helped us build much more comprehensive and deep documentation about the types of issues that people can run into. It's not, you know, it's not really our fault, it's just programming. Like, you're going to run into issues with things.
Josh:
19:52 So yeah, we kind of know what to expect and then we can build that into our documentation now instead of like sitting down and spending a bunch of time on it, hopefully.
Starr:
20:00 Totally, because like the number of issues that can come up, especially in Ruby, where someone can redefine the language out from underneath you.
Josh:
20:07 Yeah.
Starr:
20:08 Like, you can't depend on anything in Ruby, and that's cool in some cases, but also, if you're trying to provide a library for somebody to use that always works, it's going to be a little bit frustrating.
Ben:
20:20 One of the things, in talking about things, blowing up underneath you, that we've done more recently, is our continuous deployment set-up.
Ben:
20:29 Like, you know, for a long time we've done continuous integration, right? Where we have a big set of tests that we run for our code, and Josh is really good about keeping on me, in particular, about making sure that I'm writing new tests when I'm writing new code, and so we have this big test suite that gets run, you know, whenever you want on your machine, but also gets run automatically for every check-in, every push.
Starr:
20:53 Yeah, so every time you add new code and you share that code, it gets tested.
Ben:
20:56 So we find out in Slack, you know, if I wrote the build or whatever, but for the longest time we just manually deployed using Capistrano. It worked just fine.
Starr:
21:06 And note even just Capistrano, this is Capistrano-
Ben:
21:09 Oh, yeah.
Starr:
21:09 ... like, 2., like a version of Capistrano that stopped being developed four years ago?
Ben:
21:15 It works, it's like they say, if it ain't broke, don't fix it, you know?
Starr:
21:17 Yeah.
Ben:
21:17 But we recently decided we brought on, you know, a new developer hire and we thought, "You know, maybe it's time to make this process a little more automated so that's little bit more hands-off," and so now we have CircleCI, we're using the continuous deployment feature function, where we can actually have it trigger jobs that actually do the deployment for us. And that's been really nice to see that code shows up when it's supposed to, we're not like wondering, "Oh, did it get there?" You know, "Can I deploy this stuff, is it safe?" if like sometimes, as we are siloed in how we do our development, sometimes like, "Oh, did you really want that to go to production yet, or you know, that's still waiting.?"
Starr:
21:55 Now, we have like some sort of issue too, where like something was...
Ben:
21:59 It was the asset compilation in Rails. It would like-
Josh:
22:02 Yeah.
Ben:
22:03 ... compile on like one of our web servers, but not on the other, and so then there'd be some inconsistent results, you know, when you were browsing the site, the CSS would be missing, or the JavaScript would be missing, so...
Starr:
22:12 Yeah.
Ben:
22:12 You know, just having... so, as part of that continuous distribution, we have the assets pushed out, you know, automatically by Circle and that really smoothed that over.
Starr:
22:22 Yeah, so it turns out deploying like applications for multiple servers, and having them all be like in sync all the time, isn't always super trivial.
Ben:
22:30 This is actually a... You're talking about having things change out from under you, this is actually exactly that. Like you upgraded Webpacker, right? Because we're trying to keep up current with our dependencies, and the behavior changed from the version that we were using to the version that we upgraded to. And the behavior change was that those asset file names, when it compiles the assets, they changed based on the path. And when you deploy into two different servers, and we use time-stamped deployment directories, like Capistrano does, well, one server may have been one second ahead of the other server, and so now they have different path names, and now those assets when they got compiled had different names in their file names.
Starr:
23:12 Oh, interesting!
Josh:
23:13 Hm.
Ben:
23:14 Doing the continuous deployment thing made that more consistent, and I went and found that little behavior change in Webpacker after like hours of pulling my hair out, trying to figure out what was going on, and, yeah, probably made that consistent again and that solved that problem. But having that continuous deployment helped pave over that weirdness.
Starr:
23:34 Well, I didn't know that, Ben. Thank you!
Ben:
23:35 Thank you.
Starr:
23:36 Thank you for your service!
Josh:
23:37 I've been loving the continuous deployment, by the way, that I-
Starr:
23:40 Oh, me too!
Josh:
23:41 ... I mean, it seems like it's a small thing, like you can just run cap-deploy or whatever, but it's... it really... those little automations seem to add up over time.
Ben:
23:49 Yeah.
Josh:
23:50 Yeah, seems to add up to a lot of time saved.
Starr:
23:52 I don't know about you guys, but when I deploy manually I can't help but just like stare at the output as it deploys, and I don't know what I'm expecting to find, I'm not going to like jump in and fix something that goes wrong if I just can't just like not just like sit there.
Josh:
24:07 ... the environments like having it always in the same environment, deployed from the same environment helps, because it's more like, you're not thinking like, you know, "This is running in my laptop, there's a number, like a bunch of things that could cause this to fail and I literally need to sit here and watch this to make sure it doesn't blow up."
Ben:
24:24 Right.
Josh:
24:25 At least that's like, for me, I find myself doing the same thing as Starr, and I think usually I'm sitting there, I'm not quite convinced that it's actually going to work.
Ben:
24:35 And it's helped scale the business, right?
Ben:
24:36 Because the precipitating event for this was hiring Kevin, and like, "Oh, Kevin has to be able to deploy now!" Well, do we tell Kevin to cap-deploy whenever we want, or do we just say-
Starr:
24:47 VPN's, and all that stuff?
Ben:
24:49 Yeah, exactly. Why don't we just say, "Push it to Master and Circle will take care of it for you." I mean, that's much easier-
Josh:
24:54 Yeah.
Ben:
24:54 ... much more scalable on the people sense.
Josh:
24:57 Yeah. And it helps us enforce our rules around master. So-
Ben:
25:01 Right.
Josh:
25:01 ... it's, if you commit directly to master, it's definitely going out.
Ben:
25:05 It's going to production!
Starr:
25:07 Oh, wait, what?
Josh:
25:10 So have you been force-pushing to master Starr?
Starr:
25:13 That's, I just made a batch that just force-pushes everything, like I just-
Josh:
25:17 Right!
Starr:
25:18 I just type in, "git push", and it just automatically force-pushes to master.
Josh:
25:21 Okay, okay, cool.
Ben:
25:22 You know what, confession time though: my favorite commit message-
Josh:
25:25 Yeah?
Ben:
25:25 ... my favorite commit message that I make is, "Oops."
Josh:
25:29 What?
Ben:
25:29 When I commit that thing to master, and like things break, and like, "Oops!" So I commit the fix, there it is. "Oops."
Josh:
25:36 Yeah.
Starr:
25:37 Yeah, we can do this because we're the bosses, so... all you dev-ops, all of you ops people out there, who were like pulling your hair out, well, hmm, yeah, too bad.
Ben:
25:45 So, you know, one thing that's changed recently, and this came up during the interview process when I was talking to developers, it was, you know, we talked in our previous episode about how we work pretty independently. Like the three of us, we pick things that we want to work on, and then we go and work on that thing until it's done, and then we go do the next thing right? And so the question I got a lot of times was, "Well, how was that going to work when you have a developer who's not one of you three right? He doesn't share your brain space."
Ben:
26:11 And so that's been a bit of a change in the business. So I guess we could talk a little bit about how we've, you know, transitioned into that new reality.
Starr:
26:20 Yeah, how does that work?
Ben:
26:22 Well, I think having the conclaves on a quarterly basis definitely helps, you know? Setting the vision for what we want to have happen over the next few months definitely gives us, or gives me, anyway, some talking points and some context that I can share with like Kevin, and I can say "Here are the things that we want to accomplish, like here are the goals that we have for the next, you know, year, or the next half-year," and, compare that to the list of issues that we have and get GitHub you know, enhancement requests and, you know, bugs and things like that. So, you know, go wild, go in there, figure out like what makes sense, what you feel has impact to get us to where we want to go to get us to, you know, make some progress on the goals that we set out in the conclave.
Ben:
27:08 And that's been really helpful, you know, because I find I don't want to be in a situation where there's micro-managing happening, like I don't want to tell someone what to do every day and that person I'm sure doesn't want to be told what to do every day?
Starr:
27:20 Yeah, we kind of hire for this too, we hire for somebody who could work fairly independently.
Ben:
27:26 Right.
Ben:
27:26 And to be able to do that, though, that person has to have some context, right? Some framework from which they can make decisions about "What am I going to work on today?" So I think that having the Conclaves has been really helpful for that.
Josh:
27:39 Yeah.
Starr:
27:40 Oh, totally.
Starr:
27:41 Yeah, and we, ah... I think actually maybe doing the podcast has also been a little bit helpful too, because I know some of the... I know Kevin and Ben F. have mentioned that they listened to some episodes, so...
Josh:
27:54 Yeah, they can kind of get the inside scoop before it's in our heads.
Starr:
27:58 Yeah. So how do we...
Starr:
28:01 So now that we are like bosses, like what...
Starr:
28:05 I'm not really involved in the bossing, I don't do much bossing myself, I think Ben is like the big boss here. Well, most of the time. So, how do you do this? How do you... like how do you manage these good folks and make sure that, you know, everybody's happy and doing what they should be doing and stuff?
Ben:
28:25 What we do is we have... we do have a monthly one-on-one, so I meet with Ben F. and with Kevin monthly via Zoom, we chat about what happened the previous month, the things that they were able to accomplish, we chat about the the things we want to accomplish in the next month.
Ben:
28:42 It's really driven by them, I try not to set the agenda, except for they know that I'm going to ask about things that happened before and I'm going to ask about what they want to do next. But after that, it's really up to them to discuss like, how are they feeling, how they like, you know, being a Honeybadger, and what kind of things they want to work on, and what longer term goals they have, things they want to learn, maybe skills they want to develop.
Ben:
29:05 And, yeah, for us, I think we've built a business around having fun, learning stuff and, you know, building cool things and serving our customers well. And we want to have that same culture with all of our employees; we want them to have fun, we want them to learn new things, we want them to join us in serving our customers well, and hopefully that will be, you know, engaging for everybody and so far its worked out really well.
Starr:
29:30 We don't have sort of the separate track for employees, like we want to be one company that acts in one way.
Josh:
29:37 Yeah, we talked about kind of all wanting to be peers.
Starr:
29:39 Would you say we're a flat organization?
Josh:
29:41 Yeah, I would say, yeah. Exactly.
Starr:
29:43 All right.
Starr:
29:44 I'm going to read a book about this now.
Ben:
29:47 You know, I've always avoided the corporate work environment where there's a lot of just crud you have to deal with, you know? I've avoided working for big companies, I've avoided working for cruddy bosses, and so obviously I don't want to make my own business into that kind of cruddy environment.
Josh:
30:04 It'd be fine if we did, though, where... if everyone has a direct report and we are, you know... Ben's like... Ben, you can be the big boss, and then, yeah.
Starr:
30:14 No I think it needs to be circular where Ben then reports to the lowest person on the totem pole?
Josh:
30:18 Oh, okay!
Josh:
30:19 Oh, that would be interesting, to see where the business goes, and with that sort of management structure.
Starr:
30:24 Yeah, I mean, it's a wheel, so I'm guessing it's going places!
Ben:
30:27 But it has its benefits, for sure, but it has its own challenges as well.
Ben:
30:33 And, you know, we're still learning, we're trying to figure this out, and, you know, both Ben F. and Kevin have to flexible, right? And they have to be willing to, I guess, step a little bit into the dark because, you know, they are basically trodding the path for the first time, and hopefully, and they have been very patient with us, and in the mistakes that we make, and dealing with them, and they've been very good about like taking the initiative and not having to be directed in everything they do, and realizing that hey, yeah, we're just three guys who are trying to figure this out.
Starr:
31:04 Yeah... I've really enjoyed working with them, I'm glad we brought them on board.
Josh:
31:07 Yeah.
Starr:
31:08 So what's up next, like what do we want to do in terms of systems, and automation, like what's going to take our business to the stratosphere?
Josh:
31:16 Yeah, what's... you're asking us like what's your next thing to hype, right? Because you want to be the hype man?
Starr:
31:22 Oh, yeah, yeah, I'm going to be the worst hype man.
Josh:
31:27 Yeah... I don't know, um...
Ben:
31:29 Yeah, nevermind.
Ben:
31:29 You know, we're not at the four-hour workweek yet.
Josh:
31:32 Yeah, I know.
Ben:
31:33 So, I guess we have some more automation to do, or maybe some more people to hire or something to get to that point?
Josh:
31:40 Yeah.
Ben:
31:40 But yeah, I think definitely, continue to automate stuff is a good way to go.
Josh:
31:43 We've talked about, we've been... we've talked about recently like, ways that we could more like, streamline our content production process-
Starr:
31:51 Uh-huh (affirmative).
Josh:
31:51 ... and I think there's something there, you know? That, um...
Josh:
31:54 Just right now, we've all been a lot better at producing content, and I think that's really awesome, and now that we actually... you know, we're actually doing stuff now, that's always a good opportunity to start documenting and then automating and streamlining it, so.
Starr:
32:09 Also, for example, like outsourcing is good, you know? We can do more-
Josh:
32:14 Yeah...
Starr:
32:14 ... more on that end, and sort of moving things into managed services, like for example, we used to host all of our own blog content and stuff like that, using Jekyl, or whatever, static site generator, and now we started using Netlify for all that, and we have four or five websites on Netlify, and we just don't have to do any of the stuff around managing, you know, the hosting for them anymore, and that's great.
Starr:
32:39 Like, I'm very excited about, you know, whatever we can put on our managed services, even if the cost is twice as much, like, I'm all for.
Josh:
32:47 Yeah. Totally.
Starr:
32:49 You know, occasionally teases me, when it's like we're going to put our Postgres into a managed Postgres service.
Josh:
32:58 That would be the day.
Starr:
32:59 My heart just starts beating.
Starr:
33:00 There's always good reasons why we don't quite do it yet, but, you know, one day maybe we'll get there.
Ben:
33:05 Yeah. I've definitely had to learn how to give up that control, that sense of control over the things that were running, because I'm the Ops person, I like to make sure that things are my way, and...
Josh:
33:16 Yeah.
Ben:
33:16 But yeah, we're making progress, like Starr, you've convinced me that there are many things that I don't have to control, I don't have to run myself, like that Elasticsearch cluster that I was talking about, and like, we don't run that, like that's a hosted by Amazon kind of thing.
Starr:
33:29 Yes. Yeah, now we use that.
Ben:
33:32 It's beautiful! Makes me weep. Like, I can upgrade versions, all I do is click a button! And I'm like, "Thank you, Starr," as I click the button.
Starr:
33:39 Really? You are?
Ben:
33:40 Yes.
Starr:
33:41 I don't think... I don't know, I don't think I was like that instrumental, but you know, if you want to give me credit, I'll take it.
Starr:
33:46 Um, it's going on my resume!
Ben:
33:49 There you go.
Josh:
33:50 Well, you know, like we could circle this all back around to, you know, to like the business advice.
Josh:
33:58 But giving up control is an essential theme in all of these like, systems, you know, business systems, stuff to, like that's the classic small business owners' problem, like... You have to give up control, you have to be willing to like let yourself, take yourself out of the picture, and let someone else run the show, if you're going to be able to like take yourself out of the picture, or like go on vacation, or, you know, sell your business someday. Like chances are, you're not going to be too happy if the only way you can sell your business is if you go with it and work for it for the rest of your life. So you have to have step back, yeah.
Starr:
34:35 The whole point of systems is that they allow people who aren't you to do things, right?
Josh:
34:40 Right, yeah, exactly.
Ben:
34:42 Yep.
Starr:
34:43 And yeah, speaking of, so also one other thing, one other system I'm thinking about, maybe invoking, is getting somebody to edit this damn podcast, because that takes me a long time.
Josh:
34:53 Yeah, totally.
Starr:
34:54 I'm a little worried they won't be able to, like, bring the sass, and they'd-
Josh:
34:57 Well, that's the thing, you like, you're doing such a great job, you know, with it, and you have such specialized knowledge about it that, you know-
Ben:
35:04 That's... that's-
Josh:
35:04 ... you'll probably never find someone to take over for you, Starr.
Ben:
35:07 That's the eternal struggle!
Starr:
35:09 I know!
Ben:
35:09 I can do it better myself!
Josh:
35:10 Yeah, exactly, I'm being kind of facetious there, but...
Starr:
35:13 I think I may... there may be like, there may be, a... there may be a middle ground for us. Like I can go in and do a couple of like, big edits.
Josh:
35:22 Yeah.
Starr:
35:22 Things I want there, and then just be like, "Okay, you go take out the "ohs" and "ahs", and make us sound beautiful.
Josh:
35:28 Sure. Yeah. Well, I think like, having, like... you're building up a pretty good library of examples of like what we want the show to sound like, right? Like, that's part of what you're doing by doing it yourself right now, so, when you find the right person to be the editor, we can say, you know "Go listen to our," you know, "our huge back catalog, and see what it should sound like, and then make it sound like that."
Starr:
35:51 You know, this is a topic that I see coming up again and again when it comes to like outsourcing things, and doing systems, is that, you know, when it comes to having some sort of creative element in the process, is like you really have to give people the autonomy to, for example, like I do sometimes, I just delete whole chunks of things that we say, because I... they run on, or whatever, they're not... they don't really make the podcast sort of flow as well as I'd like it, and it's like you got to give people control over that stuff, and really find people who will take advantage of that control, and actually do it.
Josh:
36:29 Yeah. You do make me sound really, really smart, Starr, and I appreciate that, and I do, I worry that if we have someone else come in that doesn't know what to cut, they'll want to leave in that, you know, everyone's going to find out what we really know, what we really think.
Starr:
36:46 Here's the thing, Josh. Here's the thing. I don't make you sound smart. Because that's just how you sound.
Josh:
36:51 Aww!
Starr:
36:52 I'm just going to be, yeah, you're wearing your ruby slippers this whole time, Josh.
Josh:
36:58 Well, thanks. Thank you, Starr.
Starr:
37:02 I love that we can end the show on such a nice, positive note!
Starr:
37:05 This wraps up our special two-part episode on Systems. Next week Ben and Josh will be back from RailsConf, so we'll be doing our normal thing, and until then, why not head over to iTunes, give us one of those five-star reviews; we'll take it. Head over to Twitter, join the conversation. We're @FounderQuest, we'd love to hear your questions, comments, you know, whatever, just, you know, be nice, be supportive, be loving and we'll catch you the next time. Thanks!
Announcer:
37:29 FounderQuest is a weekly podcast by the founders of Honeybadger. Zero instrumentation, 360 degree coverage of errors, outages and service degradations for your web apps. If you have a web app, you need it. Available at Honeybadger.io.
Announcer:
37:44 Want more from the founders? Go to FounderQuestpodcast.com! That's one word. You can access our huge back catalog, or sign up for our newsletter to get exclusive VIP content. FounderQuest is available on iTunes, Spotify and other purveyors of fine podcasts. We'll see you next week!
S1E12