Serving Machine Learning Models

A Guide to Architecture, Stream Processing Engines, and Frameworks

Boris Lublinsky, Principal Architect at Lightbend

Audience: Architects, Developers

Technical level: Intermediate

Machine learning is certainly one of the hottest topics in software engineering today, but one aspect of this field demands more attention: how to serve models that have been trained. Typically, two different groups are responsible for model training and model serving. Data scientists often introduce their own machine-learning tools, causing software engineers to create complementary model-serving frameworks to keep pace. It’s not a very efficient system.

This practical report demonstrates a more standardized approach to model serving and model scoring. Author Boris Lublinsky, Principal Architect at Lightbend, introduces architecture for serving models in real time as part of input stream processing. This approach would also enable data science teams to update models without restarting existing applications.

Using Python, Beam, Flink, Spark, Kafka streams and Akka code examples (available on GitHub), Lublinsky examines different ways to build this model-scoring solution, using several popular stream processing engines and frameworks.

You’ll explore:

  • Methods for exporting models, using Predictive Model Markup Language (PMML) and TensorFlow as examples
  • Implementing Lightbend’s architecture with stream processing engines: Spark, Flink, and Beam
  • Implementing the same solution with stream processing libraries: Kafka Streams and Akka Streams
  • Methods for monitoring the architecture with queryable state

Grab your copy

Please enter your information to receive your E-book chapter(s) of Serving Machine Learning Models and be signed up for the Lightbend Newsletter. Once you've entered your information and submitted the form, the PDF will be emailed to your address.

*Required: The information you provide will be used in accordance with the terms of our privacy policy. **Required Opt-In: I would like to receive the monthly newsletter, educational resources (white papers, ebooks, webinars, reports), and event information via email. You can unsubscribe at anytime or manage your email preferences here.


Boris Lublinsky, Principal Architect at Lightbend

Boris Lublinsky is a Principal Architect at Lightbend. Boris has over 25 years experience in enterprise, technical architecture, and software engineering. He is an active member of OASIS SOA RM committee, co-author of Applied SOA: Service-Oriented Architecture and Design Strategies (Wiley) and author of numerous articles on Architecture, Programming, Big Data, SOA and BPM.

About Lightbend

Lightbend (@Lightbend) is leading the enterprise transformation toward real-time, cloud-native applications. Lightbend Platform provides scalable, high-performance microservices frameworks and streaming engines for building data-centric systems that are optimized to run on cloud-native infrastructure. The most admired brands around the globe are transforming their businesses with Lightbend, engaging billions of users every day through software that is changing the world. For more information, visit lightbend.com.