In their first episode, Starr, Josh and Ben talk about the company hack-week where they built a small proof-of-concept product with Elixir and Phoenix.
Full Transcript:
Josh: Okay so you guys tell me if this too stupid but
Starr: Well if you have to ask
Ben: I like how this is starting.
Starr: Sold.
Announcer: They're just three amigos making their way in the crazy old world of software as service. Welcome to Founder Quest.
Starr: Yeah so ah lets talk about the hackathon guys.
Josh: Cool.
Ben: Okay.
Starr: So last week we had our first ever honey badger hackathon and the idea was that we would take a break from working on our from our mundane lives you know working on air tracking, up time tracking stuff like that and work on something completely different to kind of clear out the cobwebs and have a little fun. Yeah we chose to build a product in Elixir and Phoenix because we don't really um use those two often so it's a nice chance to do something different.
Starr: So what do we hope to achieve by doing this? Like what was the goals?
Josh: So I think like just I mean mainly have fun um one of the things I really liked about the hackathon was we also this was the first thing we did coming back from a three week vacation for Christmas over which I think we all worked on an Elixir Udemy course, so it was a chance to practice some of the stuff we learned in the course and get back into the swing of things.
Starr: Yeah that was nice. Are we gonna do that vacation all the time? Is that going to be a regular thing?
Josh: Yeah I think so. Will break it up by hackathons but otherwise I'm good with just being on vacation.
Starr: Okay can we have a hackathon driven company? Is that a thing?
Josh: Yeah I think so.
Josh: Would we be like hip then, would we be popular?
Starr: Probably. What do you think Ben?
Ben: Can we be more popular? I mean you know. There are limits right?
Josh: Yeah Ben always brings humbleness.
Starr: Don't want to fly too close to the sun.
Starr: Let's think back to so we first were talking about the hackathon at one of our conclaves. If you don't know us we work remotely, we do everything in Slack and stuff. We meet up once a quarter for what we call conclaves at an undisclosed location in western Washington.
Starr: So we went through lots of different ideas for the hackathon.
Josh: Who was it that came up with the idea we went with?
Starr: I think it might have been Ben. We're talking about like the model of deploying the applications that we're interested in building and I think we were talking about things that are easy to do onsite or on a single server, they're like a self hosted type thing and that's what kind of led us to talking about Elixir and stuff, but I think it was Ben that came up with the
Ben: I think you know we do a lot in our day jobs with high traffic sites. We do a lot of processing and one thing that was really interesting was as far as Elixir is concerned is that the high concurrency that it can support so we're like “oh what can we build that would be interesting that would be in our wheelhouse but still kind of fun” and we did that. Like you said we did Elixir over the Christmas break but we also did the advent of code and
Starr: Oh yeah the, I didn't finish that.
Josh: Yeah.
Ben: I didn't finish it either .
Josh: I did like one tenth of what I expected.
Starr: I did like two.
Ben: But we don't have to talk about that. But we started with the right intentions. I know that for me, I was like doing them first in Ruby and then I would do an Elixir and see how different it was. Having the idea to play with that was a lot of the fun motivation behind the hackathon.
Josh: Yeah so eventually I think Ben was like “lets build a segment replacement”, because we use segment to send various,
Starr: Well Segment is sort of works like a repeater, you send events that happen to your users, you send your user data to Segment and then it sends out to your vast array of third party services that consume that data like Intercom, like I think we use Mix Panel, we use Drip. Maybe Google analytics, I don't know.
Ben: Yeah.
Josh: Yeah. Like it costs a lot of money, right?
Starr: Yeah it costs a lot of money well, a fair amount of money. It depends
Josh: Yeah right.
Starr: We basically only use it to broadcast, request to other services.
Josh: Right.
Starr: So it seems like it should be pretty cheap but its not pretty cheap.
Starr: Yeah and we've had some trouble. We've been, we've talked about building some sort of internal thing to do this for us, just because we're not fully utilizing its full capability yet either. I think the core of what it is like a centralized customer information database and warehouse really. And then it handles, like you said sending all those events to all the different places like third party software and service tools, even to the point where it can even replay events which I think is a cool feature that we're not using at all.
Josh: So is it actually a database though? Can you go in and query your users straight into Segment?
Ben: Well one of the destinations that you can configure is like a [inaudible] database, which we do, we dump [inaudible] so you can go and query the events. I think one of things that was interesting about using that at the hackathon project was that its very similar to what we do, we take in a bunch of events, and we spit them out to different places like Slack or Github issues, or whatever so we thought "hey we kind of know what this is like we can build something that will take in events and spit out things".
Josh: Yeah so we didn't end up replacing Segment. Note for future hackathon planners.
Starr: Yeah you probably can't duplicate a complex product built by a whole team of people with a couple guys working a week.
Ben: And I guess we should qualify what our hackathon was because I think some people might think when you say hackathon and think “oh like you start on a Friday, and its red bull craziness until Monday morning”, but ours is nothing like that. We just did it during the week.
Josh: Yeah it was just a normal work week.
Ben: During normal working hours.
Josh: I don't know, you guys did, I was chugging the red bull. I was like "where is everybody?"
Starr: That explains a lot.
Starr: So we kind of divided up the responsibility. I ended up doing sort of the back end routing of the requests as they come in, sort of fanning them out to different notifiers that would send the requests then on to Intercom, to Drip, to stuff like that. Ben worked on deployment and Josh worked on the API or intake side of things. And also I worked on the notifiers as well but I feel like Josh did the most research and had the best graphs out of all of us into the problem domain.
Josh: Yeah what I started with was working out the data model which kind of helped with that. Segment is really awesome in their documentation in that they share a lot about how they do things on the back end anyway. They have a really amazing API spec which kind of outlines how their data model works and I borrowed mostly from that but did a few interesting things to kind of make it faster for us just so we could get more decoding and not trying to figure out the whole domain and architecture and everything.
Ben: I think one of the things that you did that was really cool talking about the API and stuff was that you built those Jason schema documents.
Starr: Oh that was really nice.
Ben: Yeah that really helped, I think make it really smooth to build out the code once you had that in place.
Josh: Yeah the Jason schema, they weren't
Starr: Should we explain a little bit? What we actually did?
Josh: Sure, I can. Basically like I mentioned there's a nice API spec that segment has on their website. Its got like example payloads of like the kind of things that you can send to Segment, which is basically what models your customer data.
Starr: So do you think like I'm a person that's coming to your website or your app, I've signed up and I'm using your app and you want to tell all the tools you use that I've just showed up and logged in. You can send an event to Segment called identify which basically includes a bunch of information that you have about me, like my email address and that sort of stuff. That creates the customer record and then from there Segment would forward it on to Drip or Intercom or any of the other tools we're using.
Josh: And so these events are the core of the data model. They basically describe everything that could happen with a customer in your product. And Jason schemas are handy for describing the model of what that data is. So the data is modeled in Jason so its key values and objects and stuff and Jason schema is a standard that you can use to describe Jason data. Its like a structured format so you basically define what each property in your Jason object is, and from there you can actually run it through a validator that would check your actual data object against the schema and if it doesn't match up it will throw an error.
Starr: Yeah so the idea is that if you submit a bad API request, it will just reject it outright instead of sending it on to the model layer or further code within errors because it was wrong. You basally validate it when it comes in.
Josh: Yeah and so that's, once I had the schemas which kind of helped to make sure we understand what the data model is then the next step was to, I added a little, actually I found this little Elixir package called X Jason schema and luckily it pretty much did all the work for me to actually take an object and verify it against a schema. I plugged, I put that into a little, in Phoenix, I put into a plug which is kind of like a middleware that when a request comes in I can actually just run that payload through the schema validator and if it was wrong then it would return an error to the user basically.
Starr: I went into this not really knowing much about Elixir, not much about OPT which is sort of the foundations that Elixir and [inaudible] are built on. And I found it really fascinating to learn about all this. It was a bit hectic at first. At first, I felt like I'm not making any progress at all, this is incredibly slow, why am I doing this? But it actually helps to know the language and the platform that you're using before trying to build things in it. Eventually by the end of the week I was actually surprised at how quickly we were able to stand up, I wouldn't call it a product, but a robust system. By the end of it, we had a system that would take in events, process them, where for any given user you always use the same queue and events always went in the same order so that you didn't have the problem where the same user sends events in and they get assigned to two different queues and one queue is running faster than the other, so then they get processed out of order.
Starr: I was super impressed by Erlang and Elixir by the end of the week.
Josh: Yeah I really liked the code that you came back with, the repeater code which is what we ended up calling it. It didn't seem like a whole lot of actual code that was written but what it actually did was kind of crazy to me. The way it basically had like a routing system that would control the flow basically of data through the repeater system.
Starr: Yeah it used a built, well I guess its not built in, it's a package but its called Jenstage which lets you process streams of data in a way that takes into consideration the fact that processing doesn't happen immediately.
Josh: It was all pretty cool.
Starr: What were you going to say Ben?
Ben: One of the things that I thought was neat, one of things that we had thought was interesting going with Elixir for this was, you know we're used to the idea and the [inaudible] band, you take in some work, you shove it into a sidekick queue and then you have a job somewhere that's processing it. The only problem is that you have to now worry about Redis, and keep that up and deploy it and so on but the thing I think was really cool about how Starr did this is that its all self contained. You're running the application and its handling that queuing, that processing just in one place so you can reason about what's going on more easily and you can handle the back pressure that happens when you're trying to deliver to a target that can handle the work that's coming in. I thought that was pretty neat.
Starr: Like for our main product, honey badger, we do stuff mostly in Ruby. We work in AWS, in any given system that we push up, we may have several separate services that support it, like we may have a data store, may have caching, and its just kind of crazy to me that in Elixir and Erlang you can put all of that, or a lot of it, into the application itself. That's just really cool to me.
Josh: By the end of the week we had like what two actual destinations done, Drip and Intercom?
Starr: Yeah that was it, we had two destinations up, we could handle.
Ben: Well started [inaudible] too.
Starr: Did I? Oh my god, I forgot about that.
Josh: That's true.
Ben: Yeah on Friday you busted out the
Josh: I was, it was all the red bull, it just
Starr: That's right.
Josh: I was in a red bull fueled haze of productivity.
Starr: Yeah so by the end of the week I think we had two types of events done, which are the main two, we had the identify event that I mentioned, which is like basically telling the system that a user exists and that their here and then a track event, which is telling the system when a user performs an action. And that's where it gets really interesting is sending the data to other systems like Drip or Intercom because those systems will use these events to do things like, in Intercom for instance, when someone signs up, we start triggering various onboarding actions like emails and stuff that go out to them and that's all handled through events. That's like the end result of this data coming through our system and going out to these sources. On the actual destination side, we had Drip, Intercom, Poststress, we had a, I forgot to mention the debugger that we that I built.
Josh: Oh yeah that was really nice.
Starr: Yeah so we could see the events actually coming in like when you would post an event to the API it would put it out to the actual web browser using a Phoenix channel over a web socket, so it would actually show the formatted Jason data, and the browser basically in real time, which was
Josh: Yeah that was amazing.
Starr: And that was all
Josh: Like two lines of code, right? It was nothing.
Starr: Yeah just a couple lines, mostly boiler plate. It was more than a couple lines but it was mostly boiler plate Phoenix code so that was pretty cool.
Starr: Can I tell you guys a secret?
Josh: Of course.
Starr: The whole event routing thing I did that was boiler plate too. I copied most of it straight from the Elixir docs. And most of the time I spent on it was just trying to understand how it all worked.
Josh: Oh my gosh, Starr.
Starr: Am I fired?
Ben: Well while we're talking about copying and pasting, so
Starr: Yeah what did you copy and paste Ben?
Ben: Yeah so what I copied
Starr: We are experienced senior engineers here.
Ben: So my job was the upside so I had to figure out “how we gonna get this into production” and I'm somewhat colored by having to keep honey badger up and running at all times so sometimes I over engineer but I held back and I just copied and pasted a bunch of docker stuff from the distillery documentation which is fantastic. They have great examples on how to deploy a various providers and so I decided to run with the docker version because you can run a docker container anywhere, right?
Ben: SO the docker that I got from Distillery was pretty cool, they talked about how you can build, and this was kind of new to me, I haven't spent a lot of time with Docker, so I got to learn about this, which was fun. You can build your one image like a build image and then you can use that build image in another image so that you don't have to have all of those build artifacts along with the final image that you send out. So you know with Elixir you got to do a lot of compiling, got to have Erlang and all that stuff installed and you might not need all those things in your final production image right? You can, step one is you compile the app using Mix and get everything into a release a [inaudible] and then the second step, the second image that you make grabs that tarball puts it in the right place and then packages it up. Instead of having a few hundred megabytes worth of image you can have like forty or fifty, thought that was pretty cool.
Ben: So yeah there's documentation on the Distillery side about how to do that and it works out pretty well. The one trick being that you do you really want to configure your configuration of like for example, Intercom and Drip have API keys that you have to use right to be able to talk to their services so the only bit of snag is that you want to have the Elixir application configured by a little code that actually hangs outside of the compilation step because those API keys aren't necessarily available when you're compiling. But they will be available when you run it in production. Toss a little code to the side that loads those things from the environment at run time rather than compile time so that you can configure on the fly with those API keys.
Starr: So on the whole, what do you guys think of hackathon? Would you do it again?
Josh: Totally.
Starr: Awesome. It was great being able to work on something that didn't have to do serious stuff in production or we could fail and it didn't matter. It's kind of nice.
Josh: Yeah. I had a lot of fun.
Starr: Alright well I guess that wraps up our podcast. I think Ben has somewhere he needs to be so
speaker 4: Thunder Quest is a weekly podcast by the founders of honey badger. Zero instrumentation, three hundred and sixty degree coverage of errors, outages and service degradation for your web apps. If you have a web app you need it. Available at www.honeybadger.io.
speaker 4: Want more from the founders? Go to founder quest podcast dot com. That's one word. You can access our huge back catalog or sign up for our newsletter to get exclusive VIP content. Founder Quest is available on iTunes, Spotify and other purveyors of fine podcasts. We'll see you next week.
S1E1