From 81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba Mon Sep 17 00:00:00 2001
From: Brian O'Neill <bone@alumni.brown.edu>
Date: Thu, 12 May 2016 20:10:33 +0100
Subject: [PATCH] =?UTF-8?q?[SPARK-14421]=20Upgrades=20protobuf=20dependenc?=
=?UTF-8?q?y=20to=202.6.1=20for=20the=20new=20version=20of=20KCL,=20and?=
=?UTF-8?q?=E2=80=A6?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
## What changes were proposed in this pull request?
When running with Kinesis Consumer Library (KCL), against a stream that contains aggregated data, the KCL needs access to protobuf to de-aggregate the records. Without this patch, that results in the following error message:
```
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
```
This PR upgrades the protobuf dependency within the kinesis-asl-assembly, and relocates that package (as not to conflict with Spark's use of 2.5.0), which fixes the above CNFE.
## How was this patch tested?
Used kinesis word count example against a stream containing aggregated data.
See: SPARK-14421
Author: Brian O'Neill <bone@alumni.brown.edu>
Closes #13054 from boneill42/protobuf-relocation-for-kcl.
---
external/kinesis-asl-assembly/pom.xml | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/external/kinesis-asl-assembly/pom.xml b/external/kinesis-asl-assembly/pom.xml
index e057b78abd..6fb88ebae5 100644
--- a/external/kinesis-asl-assembly/pom.xml
+++ b/external/kinesis-asl-assembly/pom.xml
@@ -63,7 +63,12 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
- <scope>provided</scope>
+ <version>2.6.1</version>
+ <!--
+ We are being explicit about version here and overriding the
+ spark default of 2.5.0 because KCL appears to have introduced
+ a dependency on protobuf 2.6.1 somewhere between KCL 1.4.0 and 1.6.1.
+ -->
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
@@ -147,6 +152,15 @@
<include>*:*</include>
</includes>
</artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google.protobuf</pattern>
+ <shadedPattern>kinesis.protobuf</shadedPattern>
+ <includes>
+ <include>com.google.protobuf.**</include>
+ </includes>
+ </relocation>
+ </relocations>
<filters>
<filter>
<artifact>*:*</artifact>
--
GitLab