GCP noSuchMethodError 해결

JAR 를 만들고 Spark 실행을 하면, 아래와 같이 에러가 발생 할 때가 있다.

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;C)V
    at com.google.common.io.BaseEncoding$Alphabet.<init>(BaseEncoding.java:458)
    at com.google.common.io.BaseEncoding$Base64Encoding.<init>(BaseEncoding.java:940)
    at com.google.common.io.BaseEncoding.<clinit>(BaseEncoding.java:322)
    at com.google.api.client.util.Base64.decodeBase64(Base64.java:101)
    at com.google.api.client.util.PemReader.readNextSection(PemReader.java:99)
    at com.google.api.client.util.PemReader.readFirstSectionAndClose(PemReader.java:128)

이는 spark에 있는 라이브러리가 예전 버전이라서, JAR dependency의 라이브러리에 있는 함수가 존재하지 않아서 발생하는 에러이다.

아래와 같이 maven shade plugin을 사용하여 uber-jar를 생성하면 에러가 해결된다.

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>com.test.App</mainClass>
                </transformer>
              </transformers>
              <relocations>
                <relocation>
                  <pattern>com</pattern>
                  <shadedPattern>repackaged.com.google.common</shadedPattern>
                  <includes>
                    <include>com.google.common.**</include>
                  </includes>
                </relocation>
              </relocations>
            </configuration>
          </execution>
        </executions>
      </plugin>

package 를 해보면, 아래와 같이 jar들이 shaded 되는걸 확인할 수 있다.

[INFO] Including com.google.cloud:google-cloud-storage:jar:1.98.0 in the shaded jar.
[INFO] Including com.google.cloud:google-cloud-core-http:jar:1.91.2 in the shaded jar.
[INFO] Including com.google.cloud:google-cloud-core:jar:1.91.2 in the shaded jar.

참조

https://cloud.google.com/dataproc/docs/guides/manage-spark-dependencies?hl=ko

https://stackoverflow.com/questions/56824843/getting-error-while-running-scala-spark-code-to-list-blobs-in-storage