Netflix's internal data pipeline processes hundreds of billion messages a day and has recently been upgraded to use Apache Kafka for collecting data from all their applications. Jonathan Bond (Senior Software Engineer at Netflix) will explain the challenges of building a pipeline to handle a trillion messages a day, and how end users can configure it to get the data to the right teams at the right time. Delivery of messages is critical to a successful pipeline; Jonathan will show how the Netflix pipeline is monitored and how failures are handled to exceed four nines availability.
Bio
Jonathan Bond is a Senior Software Engineer at Netflix where he uses Apache Kafka to implement data pipeline and messaging systems. Prior to Netflix, he was an engineer at IBM where he worked on transaction processing, messaging, enterprise service bus, complex event processing and business rules.