SF Logging MeetUp: Apache Kafka edition

Clark Haskins and Todd Palino, both SREs from LinkedIn reprised their talk about operating Apache as a Service from this year's Apachecon. They presented an overview of how the Kafka system works, things that are available as open source and projects that they are working on internally that may or may not make it as open source projects.

Man, it’s been a busy month! Earlier this month, you might’ve read about the DevOps internal mini-conference we pulled together. Greg Wester will post about our DevOps Hack Day early next week. This week, we hosted the San Francisco Logging Meetup group which I organize each month. Since working here, the Meetup group has really been picking up steam.  In the last month alone, we’ve added almost 100 members.

On Monday night, we held the latest installment here at salesforce.com.  Response to the Meetup was pretty overwhelming with a long waiting list and a good sized group in attendance.

Clark Haskins and Todd Palino, both SREs from LinkedIn reprised their talk about operating Apache as a Service from this year’s Apachecon.  They presented an overview of how the Kafka system works, things that are available as open source, and projects that they are working on internally that may or may not make it as open source projects.   You could really tell how much pride each of them had in the service they support, even though each of them had only been supporting the service for a relatively short time.  In that time, they have made lots of valuable contributions to the project and have really brought their years of operations experience to their jobs every day. Everyone has benefited as a result. They also took great pride in the fact that they sit with most of the main contributors to the project and that really helps them to deliver the service to the rest of the company at a high level.

Some things we learned from Clark and Todd:

  • If you’re going to operate on the most bleeding edge of the Apache Kafka project, you want to be sitting with the developers who are writing it!
  • No matter how much planning you do, someone is always going to come up with a novel use case for your service you hadn’t planned for; be welcoming but firm.
  • As an operator, you need to be the one to fix the problems that are most important to you.
  • In the old days, migrating between Kafka releases was HARD! It’s gotten so much better over the years.

The evening was definitely a success and we’re already putting the finishing touches on our July Meetup which will also be held at salesforce.com. Being able to bring great speakers and great content to the community at a company that really prioritizes giving back, is another reason that makes working at Salesforce a #dreamjob.

Full Disclosure:  We run Apache Kafka internally in production at salesforce.com!

Leave your comments...

SF Logging MeetUp: Apache Kafka edition