Season Two of FounderQuest is upon you! The guys discuss what they did while on summer break, releasing breadcrumbs on Honeybadger and why they help when debugging, launching the new blog, Amazon RDS, and Airwolf!
Links:Write for our
blog.
Full Transcription:Josh: We can't all be book-learned like you star.
Starr: Yeah, I'm book smart.
Josh: Yeah.
Starr: You guys are just the street smart ones.
Josh: We just read what's ever on the front page of Amazon.
Starr: No, y'all know how the streets work, you know how to get stuff done, you know, outside of formal channels if you know what I mean?
Announcer: Never forget that you have the tools to build a life on your terms. Forget the haters. This is Founder Quest.
Starr: We were going to talk about I mean we've been gone for a couple of weeks, several weeks, and I think we're just going to talk about what we're going to do. I've decided, we shouldn't say that we had a vacation because that's not the appropriate...
Josh: How I had multiple vacations and I did some work.
Starr: No, hush, hush, hush.
Josh: Okay.
Starr: I think we shouldn't call it a vacation. I think we should say that this is season two of Founder Quest.
Josh: Oh, I see where you're going. Okay, so season two go.
Starr: Yeah, nobody says like serial had a vacation.
Josh: Right.
Starr: There just on, you know, they had a new season.
Josh: So are we going to be a serial podcast now?
Starr: Well, we come serially so yeah sure.
Josh: Okay, cool.
Starr: Yeah, sure, it's just whenever we want to go on vacation we'll just say there's a new season.
Josh: Nice.
Starr: so people have to say tuned.
Ben: I can't wait for season three.
Starr: I hope we don't get canceled. That would suck.
Josh: Yeah.
Ben: So how were your vacations, Josh?
Starr: Yeah. How many vacations did you take?
Josh: I think I took two. The first one was a kind of a, I mean I was planning to take some time off, but it was not as planned as this last one. Yeah, in August I took a couple of weeks off just to kind of recuperate and just connect and stuff. This last week, I just got back from visiting my kids grandparents, my parents in Arizona. That was a fine. It wasn't, I don't know, it was still a lot of work because the kids. Taking two little toddlers on an airplane that was the first time we'd done that. Yeah, it was fine. Got some sun, they have a pool. We spent a lot of time by the pool.
Ben: Nice.
Starr: I see a little bit of a thousand-yardstare in your eyes.
Josh: Yeah, I think.
Starr: The children on the airplane thing.
Ben: Oh, Starr you did that too, right? You just got back?
Starr: Yeah, I got back from also a grandparent vacation, but this time the grandparents, my partner's parents were like, Hey, we've got this timeshare in Mexico in Puerto Vallarta and so you should come down and I've got to tell you, boys. I was skeptical because it seemed like we were doing all the work right because normally like paying for accommodations is not the most daunting part of an international vacation with a three-year-old. I was skeptical, but it turned out to be super nice and relaxing because they have the timeshare they have, it was very nice. It was a sort of like an all-inclusive resort. They had this giant like a four-bedroom suite with balconies and a hot tub on a balcony and like I got to sit during the day for several hours for multiple days in a row and just like read by myself. I haven't gotten to do this unless I just like skip work and do it or something.
Josh: That's what I did by the way.
Starr: It's not the same. Right, because you feel like you need to get back. Yeah, it was just great. I mean it wasn't a cultural trip. I lived in Mexico for a year. I feel like I have a good appreciation of the Mexican people, country and culture already. A little bit of a resort vacation. It was kind of welcome I think.
Josh: I also got to spend some time reading by the pool while the kids were napping and that was very nice. What about you Ben?
Ben: I did not take any vacations with toddlers, but I did drop off my older son at college. He just started college this fall. That was a pretty wild experience. Let me tell ya. And I got some house work done, like had contractors doing work on my house, so I'm flat broke now because between college and, Oh and then my car broke down. I had to replace a bunch of stuff on it.
Josh: Oh man.
Ben: One here actually another, it's been, it's been crazy, but you know, it was a good Summer I got to relax a bit and yeah, I didn't go anywhere. I didn't do anything but read some books and you know, I hit the trail with my bike several times, which is nice. We had a good summer here in Kirkland. So yeah, checking out of work at, at lunch time and going riding is a good thing.
Josh: You're going to take some time off later this year?
Ben: Yeah. And go weeks. I'll be heading to visit. My parents haven't seen him at their house in quite a while, so that'll be fun.
Starr: Yeah, that's great. That's like a big life milestone when you think about it. It's a little bit of a permanent vacation.
Ben: Yeah.
Starr: You know, it's like they're out for good, you know, I mean, until they, you know, decided to become an artist and then come back because you know art, the only pay much. You got a couple of years.
Ben: It's interesting, the house is definitely quieter, that's for sure.
Josh: Yeah.
Ben: You know, kids these days. He texts his mom regularly and you know, when I went off to college it was, I never talked to my parents because I couldn't afford to call them cause all long-distance phone calls and there are no emails.
Josh: That's right, yeah.
Ben: It was like to write them a letter.
Josh: Go down to the telegraph, go down to the telegraph station.
Starr: Yeah.
Ben: So it's a, it's a little different these days than there was back in my day. But it's, it's still fun and you know, he's still in the dorm and doing all the college stuff, so that's cool.
Starr: That's good, I've got a plan for IDA too, to pay for college and that's just like have her, you know, join the circus instead I've got a friend, Bernie. I think he might be able to get her an in, a little foot in the door.
Josh: Yeah. Well I think, I think connections are really important in getting your kids into college these days. I hear. So it's good to have those.
Starr: It's not meritocracy people. I'm sorry to break this to you. The world is not a meritocracy.
Ben: Yeah. Only chumps like me actually pay for college.
Josh: Well you know what I did, I, I've been saving for my kid's college since before both of them was born. That's just what I do. I figure I'll just, I'll save a certain amount and if it goes to college, great. If it goes to some other kind of education, you know it doesn't have to be college and hopefully, it's not, to be honest. We'll see.
Ben: Worst case scenario you get a pool.
Starr: We're going to be one of those micro people who to be like micro and feeling like you don't need to go to college.
Josh: Yeah.
Starr: Or Pre-school it's okay. So Josh like you came back and like immediately were interviewed like by some pretty like serious famous podcasters, right?
Josh: I know, Yeah, I'm a minor celebrity now. I'm pretty sure, cause I was on indie hackers. I interviewed with indie hackers a couple of weeks ago. I think he came out last week, last Monday and it was a lot of fun. Me and Courtland, Courtland Allen is the host there and he's a great interviewer. We had a lot of fun just kind of going through the Honeybadger story,
Starr: Refresh my memory. I think they're the people who always sort of get, they're always sort of back when I read hacker news. Before I realized it was a trash fire. They're always getting sort of featured on hacker news and stuff because it's like, you know, learn about someone's a small product team and they may make this much money now and blah blah blah. So that like that's the sort of thing it is, right?
Josh: Yeah. It's like a podcast for honestly, it's a, you know, a lot of pretty, pretty similar to our audience with basically like Bootstrappers, I think they lean heavily towards them because of the hacker part, you know, it developed Bootstrap developers that are trying to do their own thing. Yeah, that's right where we fall in a lot of their other interview, like all the other people on the podcast or fall into similar space like us, I think.
Ben: Yeah, listen to that. It was good. You did a good job on that.
Josh: Thanks, yeah, I don't know. I'm always, you're always like, you don't know how it's how it goes. Like right after you, I always think the worst and I'm like, you know, review every thing I said in my head. Yeah, I think it turned out turned out pretty good.
Starr: Oh yeah, when we started this one. This is like I said, I remember Josh that you were always like, Oh I was so dumb sounding and you know I was editing them and I was like, no, you didn't sound dumb. You sounded pretty smart. It's the same thing. I had the same thoughts and I would go into the editing process and see how I sound, I was so off the rails scattered. Then I listened to it and it's like, no. I say I'm pretty sure I knew what I was doing.
Josh: The process of recording this podcast has helped a lot just to make me a little more comfortable sitting down and talking. I actually enjoy it now.
Starr: You guys haven't necessarily had this experience sitting down and like listening to your own voice every week for like an hour or so. It's really sort of enlightening in a strange way. I can't really put words to it. It's just like, Oh.
Josh: Mmhmm.
Starr: it's like, it's like you learned kind of what you really sound like as opposed to what you always sort of imagined. You sound like in your head, it gives you a little bit of perspective.
Josh: I know a number of people who podcast and just won't listen to their episodes. They just, you know, record and ship it in and it's gone.
Starr: They're missing out on a wonderful growth opportunity.
Josh: I was not sure if I was going to do that or if I should listen and I ended up listening to them. Now I think yeah, like you said, listening does help you kind of a perfect your cadence and you know, the um's and AH'S and stuff. I like to think I'm getting a little bit better at dropping those. As I say like five of them.
Starr: It's all right. I mean you're better now. Not perfect, right?
Josh: Yeah, every little bit is an improvement.
Starr: Yeah, when we first started, I had a real problem because I was so used to like having a complete thought-form before I started talking. It turns out like on radio or podcasts, you kind of just have to start talking and trust in the universe to give you something to say, because otherwise it's just going to, yeah, it's just going to be so stilted and, and slow and it takes forever to edit together.
Josh: I think that in The Office, Michael Scott's, that was Michael Scott's formula for telling jokes, I think it is just like start a train of thought.
Starr: Just start talking.
Josh: Then hope it turns out, yeah.
Starr: Wait, I haven't seen the American office. Which one is Michael Scott?
Josh: He's Steve Carell's character the main one. Yeah, the main one. The boss.
Starr: You're comparing me to a satirical character Josh. Yeah, okay sure.
Josh: Yeah. I mean you're a total Michael Scott. Just a shoo-in.
Starr: It's okay. I love you too. So let's see, what have we been up to like with the product and stuff? I mean, we haven't been all just completely on vacation. We've been like, we shipped a major feature like breadcrumbs. Did we talk about this in an earlier podcast?
Starr: I don't think so. We talked about CSPs. I think breadcrumbs was still in stealth mode, so we hadn't discussed it.
Starr: Oh god.
Josh: Yeah.
Starr: Every time people say self mode know it. I just think of like Knight Rider and it just sounds so cool. I want everything to be on self mode. So what are breadcrumbs? Can somebody explain breadcrumbs to me?
Ben: Oh, sorry. I got kind of distracted by the whole Kight Rider thing. I was thinking about Airwolf and how cool that show was.
Starr: It's okay. We're all distracted by night rider. I mean it had the Hoff, right? I've never rider but no.
Ben: It's like Knight rider but a helicopter instead of a car?
Starr: Oh, okay.
Josh: It was called Air Wolf?
Ben: yeah.
Josh: Wow.
Ben: It's sweet. Check it out.
Starr: Oh man. The eighties, the eighties were very Weird.
Ben: All right so breadcrumbs.
Josh: where they weirder than now?
Starr: The breadcrumbs, yeah.
Ben: Yeah, so breadcrumbs the feature allows you to track the things that happened and your application right before an error happened. Maybe a user clicked on a certain thing or certain queries got to run or whatever. So it allows you to some additional context. We already had the notion of contexts. When you're reporting an error you can report things like the user ID or maybe an order ID or whatever was like in flight at the time that error happened. Breadcrumbs extends that a bit and say, okay you can stuff a bunch of activities that happen that were leading up to the event and to this basis basically in a ray they get sent to us. You can say, Oh well they encounter this page or they ran this query or whatever. And that helps set the scene for how that air could have happened. It's pretty handy information.
Starr: Can I tell you of a place where I could have used breadcrumbs like sort of recently, but they were just not quite, you know, shipped and, and ready. We use Honeybadger internally. You know, as you guys know to keep track of our internal errors, cause we do have a couple. I was going and one of my morning routine things is I go and check the errors specifically for like the front end Java script stuff because that's the stuff I am most likely to break. And I noticed there was this weird specific class of error where our front end code, we full confession, we use jQuery still because it's just awesome when we built the app. Yeah, you're right. You know what? I should own this. We love it. We're never getting rid of it. No, we're using jQuery and for some reason jQuery was interpreting Ajax requests response as Javascript.
Starr: Then there was, it was saying there was a syntax error in the Javascript and in case you'd never had this happen to you before. This is one of those JavaScript errors that give you no information other than like something bad happened and it's just like syntax errors like crate what, where, what's the syntax error, please tell me. So I had to go and sort of do a little custom instrumentation to log every data request that happened in the application and still get to figure out which one was returning. I could figure out which one happens just before the error to figure out to see like why it was returning.
Ben: Mmhhmm.
Starr: To be honest, we don't really have a lot of like Ajax responses. It should be returned in JavaScript. We have a couple, which is another great technique and yeah, breadcrumbs would've been really nice because then it would've let us, it would have, let me just see sort of maybe automatically that, well, you know, here's all the responses we got and I wouldn't have had to look at that manually.
Starr: It turns out, you only know the punchline. It turns out that jQuery, because it's so awesome and so user friendly, unless you tell it exactly what type of response to expect from an Ajax call, it will do its best to figure it out for you. For some reason our engine X error pages that you get when you know you're throttled or whatever. For some reason it was interpreting those as Java script that it needed to execute even though there's not. I checked the headers, I checked the content, there was nothing in it that looks like Java script. I have no idea why.
Josh: I like a dollar sign in the, in the body.
Starr: Oh there may be cause I'm all about the Benjamins.
New Speaker: If there's a dollar sign. It's jQuery.
Starr: Okay.
Josh: Yeah.
Starr: Damn, Hmmm. Anyway, that would've been useful.
Josh: That's exactly the use case for breadcrumbs. Yeah and we launched them so far we've launched them up for Ruby and Elixir. I'm currently working on the JavaScript limitation. Pretty soon you won't have to hack all of that networking code together to see what's happening.
Starr: Well, you know, what am I going to do when like Honeybadger just does everything for me? This is a problem they're going to have.
Josh: Yeah, just a vacation and well yeah, record seasons-
Starr: More vacations?
Josh: of Founder Quests.
Josh: Yeah.
Starr: One other thing that we've been doing is or I've been doing mostly is revamping the blog. If you dear listeners, go and look our blog Honeybagder.io/blog. You'll see a brand new spiffy design.
Josh: Looks amazing.
Starr: It looks amazing. Yeah, no lie, it does. That's one of the things I've been working on because we're going to have a big new effort, a big new push around blogging. It's interesting because the genesis of this is actually kind of interesting because one of the things I really like to do is occasionally download a whole just ton of .csvs full of data around all aspects of our business and then just sort of like disappear for a couple of weeks and crunch it and, and try and figure out what it means.
Starr: I noticed that in sort of the times when we happen to be the most active sort of blogging, we saw some results from that in terms of revenue and all that. They are the type of results though that are really kind of hard to see like in the moment. I remember checking stuff and checking stats like in the moment when we are blogging after six months or a year. It was pretty obvious. So yeah, so my big new push is blogging and we got so new blog design is sort of the first step because we want the single look great and we want it to look attractive to people. You know, the second step, which I'm working on now is finding authors because I like to write and you know, I plan on writing, but keeping up the level and the pace of blog posts that we really need.
Starr: I did it once and it wasn't sustainable.
Josh: Yeah.
Starr: It was just stressful and I ran out of things I knew about to write.
Josh: you blogged a lot Star.
Starr: Yeah I was like doing, you were like a blogging machine. I was doing like a technical blog post every day. It was your full time job. So now I'm trying to maybe recreate sort of the success that we had in that by in a way where that is sort of more like a system, more like a business should be right. Where we sort of hire people to do some work and we do some of it in house. And so it's kind of this big new thing. I'm excited about it and we'll see how it goes.
Josh: Tell me about like the quality of the blog posts and the authors that you're looking for. Cause I don't know, maybe we have some listeners who would be interested in writing us?
Starr: That's a great idea!
Josh: I mean we are on this podcast here where developers listen.
Starr: Ya so, I've been working on a sort of author recruitment document, which I shared with you guys yesterday and it includes a lot of this. I guess I'll publish that and we'll probably link it in the show notes and everything. Basically what we're looking for, the type of content we're looking for and that we've had success with in the past sort of has to do with teaching people sort of fundamentals and teaching people how things work on a level deeper than a lot of tutorials usually go because there's so much content on the web. It's just about how like how do I build a form? How do I make a to do app in Angular? Does Angular still exist? I don't know.
Josh: I'm pretty sure it does.
Starr: There's this, a lot of this very sort of tactical content, this like how to do a specific thing that's useful and everything. But well I found personally, and I think how this relates to the mission of Honeybadger as a company is we help people deal with errors and in their systems and fix their errors. One of the things I found to be really useful is to have a sort of very broad knowledge of things and to try and sort of understand how systems work. A lot of times the thing that's causing your error is a misunderstanding of something sort of fundamental to the system. It's a leaky abstraction where like the operating system and the way it works is somehow influencing the way your code needs to work and your code didn't take that into account.
Starr: I think the blog is all about just sort of providing people that context and providing this sort of rich learning experience that teaches people things that they're going to stick with them like for a year or two. It's not just something that's going to they're going to sort of forget in or it's going to expire because like a new version of node came out in three weeks. That's the sort of thing we're looking for and we're looking to find people who we can work with on a sort of a longterm basis to do lots of articles. We're going to pay very well compared to most places that pay for blog posts. We're going to start at, you know.
Josh: The price is free right? Like, we're, we're just-
Starr: Oh yeah, you're doing it for free.
Josh: I mean you get the privilege of guest posting on the Honeybadger blog. That's, I mean that's, that's exposure right there.
Starr: Exposure is the ultimate currency. That's right. So no, we're going to pay him money. I talked with Peter Cooper about this, he said this is a very sort of generous or of starting rate. We're going to start at $500 for sort of average article, whatever that means. We're still kind of working it out, you know, more complex stuff we'll pay more for or we're not interested in nickel and dimeing people and we're interested in having sort of excellent content that really helps people. I don't know if you're interested in working with us on that, working with me specifically, then yeah, get in touch. We will link to that on the show notes in Twitter and whatever.
Josh: Yeah, I can't wait.
Starr: What else is going on, like?
Ben: So a lot of what I do is not visible to our customers. A lot of the work that I do is in the background. It's, you know, as long as you don't notice any problems that I'm doing my job, right? I like what star was saying about how we're looking for people to write content that's not about the tactical, here's how you do with thing, but more about here's how you understand stuff. In contrast, like when I write for our blog, I typically write about and here's how to do the thing because I'm doing this kind of thing in the background. Oh, this is kind of cool to share, right? I love writing about that kind of stuff.
Josh: Mmhmm
Ben: I think I'll continue doing that.
Josh: Yeah, which is totally useful as well.
Ben: It's fun I like learning stuff. Then, you know, one of the, to me, I think one of the things is whenever you learn something, it's a good time to write about it because as you teach other people, you learn it better and you start to poke holes in your and your own experience and you see where you may have messed something up.
Ben: Recently, I've spent the past a week or two redoing our internal Honeybadger instance. So sorry I mentioned that you know we had used honey Badger to track honey Badger. Our instance was a little out of date because we've been making some significant changes over the past year. We switched from Solar to Elastic Search. We started using Lambda a lot more. There are a bunch of like moving parts to our production Honeybadger instance that hadn't quite made it to our internal Honeybadger instance. In the past week or two, I've replaced our internal Honeybadger instance with a brand new stack. While I was in there I thought, Hey, why don't I automate all this so you know, while I'm doing some work, let me do some extra work that no one's going to notice. I built an entire like Terraform infrastructure that boots up a new region for us. It creates the VPC.
Josh: Mmhmm
Ben: It creates all the security groups, the elastic search cluster, the RDS instance, the EC2 instance. Gosh, you just run, apply and all of a sudden we have buckets and Lambdas and everything. It's pretty cool.
Josh: I don't know Ben, that sounds like a huge waste of time. Like, I don't see how that could ever be useful to be honest. I mean, yeah, that's awesome.
Starr: Can I ask you a question that may betray my ignorance or maybe just my memory? Are we using RDS for like our main Postgres?
Ben: For our production, since we are not, we are a self-hosting Postgres. We use Petroni name to have automated fail-over and that's been working like a champ.
Starr: Okay. So the way that you work so often Ben is you sort of slide things in and so I know we talked about using RDS for our main Postgres and so I was just like did this just to use like do this just nonchalantly?
Ben: I placed everything in Cassandra.
Josh: That's my favorite like yeah, just I come back and there's Casandra like everything is running, out of cassandra, I'm like-
Josh: Well, it sounds like this RDS chain, like maybe this is this kind of like a good test environment for trying out these ideas and then potentially we could implement those changes like go the other direction and roll that out in production?
Starr: I feel like I should explain that RDS is Amazons managed database hosting options that you would use instead of like running your server and running your own Postgres on top of it.
Ben: What I did is I created this in a separate AWS account. I created a new account so that it would be just, you know, from scratch and there won't be any conflicts with our production environment or our testing environment and yeah, the goal was to be able to just run Terraform and you could get a full-stack and you could do all kinds of experiments with it, right? We can, and you name it, you can try it out or even just the like one of us wants to have that full environment. That's not our laptop, right? You could have that too, they could do things, you know, test a new version of Postgres. For example, like it's running this internal instance is running Postgres 11 and we're still on 9.6 In our production environment. Yeah, I get to test that.
Ben: I'm glad you mentioned that because I tested the new partitioning scheme and Postgres has I think Postgres 10 introduced this and post wrestle that and made it even better. 9.6 That we're using in production. We use the PC department, it manages our petitions for us.
Ben: Basically, what a partition is Starr, I know you'll want like to explain this, a partition splits up your big database table into smaller tables so you can more effectively manage the data that's in there. You can have queries against smaller tables because it's partitioned say by day or by month.
Ben: In older versions of Postgres that was good as it didn't have native partitioning, you had to use triggers. An insert would happen to a parent table and the trigger would say, Oh Nope, don't put it in the parent table. Put it in one of the child tables. The new partitioning that was introduced in version 10 and made better in version 11 now has that built into post-grads so you don't have to use triggers that understands how you can split a table and send multiple child tables. And so just the past couple of days, it did that in our, in our new internal environment. Now it's using the native Postgres partitioning rather than part man. Stuff like that will eventually roll back into our production environment after being tested in this stand a lone environment. Yeah, that was fun!
Starr: Nice! Good work! Yeah, I think Founder Quest we are sort of on track for, you know, being back. I think we're back. Are we back?
Josh: We're back.
Starr: Okay, Well now when you go to MicroComf next here, everybody can come up to you and say, I know what you did last summer. I've been saving that one out this whole time.
Josh: Well done.
Starr: I've just been dying, I've been waiting for my spot. All right, well should we, should we wrap this one up or what?
Ben: You know it's not happening in Vegas next year?
Josh: That's right.
Starr: Oh, I know. Yeah. What's up with that? It's like I didn't want it to in Vegas, but I didn't want it to be like a layover plane ride away. I'm just going to bitch about whatever happens,
Ben: You know, we went to Minneapolis for Rails Conf. It was really nice. Yes, you can get a direct flight there.
Ben: So yeah.
Starr: Oh, you can? Okay. I didn't know that. I think I may have missed a direct flight last time I went there, but Oh wait I went to Madison not Minneapolis. Oh my God. The Midwesterners are just going to be throwing cheese at me.
Starr: All right, well it was great talking with you guys and yeah, if you're ready to wrap it up, I think we can do that.
Josh: Yeah,
Starr: Alright thanks bye.
Josh: Bye
Announcer: FounderQuest is a weekly podcast by the founders of Honeybadger. Zero instrumentation 360 degree coverage of errors, outages, and service degradations for your web apps. If you have a web app, you need it available at Honeybadger.io. Want more from the founders? You can go do Founderquestpodcast.com that's one word. You can access our huge back catalog or sign up for our newsletter to get exclusive VIP content. FounderQuest is available on iTunes, Spotify, and other purveyors of fine podcasts. We'll see you next week!
S2E1