Error in CDAP > 6.2

[hayj]

When using this plugin (batch sink in my case) which works on CDAP 6.2, I get this error on CDAP > 6.2:

2021-04-07 15:37:12,609 - ERROR [Executor task launch worker for task 0:o.a.s.e.Executor@91] - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.reflect.UndeclaredThrowableException: null
	at com.mongodb.hadoop.util.$Proxy71.getConfiguration(Unknown Source) ~[na:na]
	at com.mongodb.hadoop.output.MongoRecordWriter.<init>(MongoRecordWriter.java:58) ~[mongo-hadoop-core-2.0.2.jar:na]
	at com.mongodb.hadoop.output.MongoRecordWriter.<init>(MongoRecordWriter.java:75) ~[mongo-hadoop-core-2.0.2.jar:na]
	at com.mongodb.hadoop.MongoOutputFormat.getRecordWriter(MongoOutputFormat.java:46) ~[mongo-hadoop-core-2.0.2.jar:na]
	at io.cdap.cdap.etl.spark.io.TrackingOutputFormat.getRecordWriter(TrackingOutputFormat.java:45) ~[hydrator-spark-core2_2.11-6.5.0-SNAPSHOT.jar:na]
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
	at org.apache.spark.scheduler.Task.run(Task.scala:100) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) ~[io.cdap.cdap.spark-assembly-2.1.3.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_282]
Caused by: java.lang.IllegalAccessException: Class com.mongodb.hadoop.util.CompatUtils$CompatibleInvocationHandler can not access a member of class io.cdap.cdap.etl.spark.io.TrackingTaskAttemptContext with modifiers "public"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) ~[na:1.8.0_282]
	at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296) ~[na:1.8.0_282]
	at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288) ~[na:1.8.0_282]
	at java.lang.reflect.Method.invoke(Method.java:491) ~[na:1.8.0_282]
	at com.mongodb.hadoop.util.CompatUtils$CompatibleInvocationHandler.invoke(CompatUtils.java:43) ~[mongo-hadoop-core-2.0.2.jar:na]
	... 13 common frames omitted
2021-04-07 15:37:12,661 - WARN  [task-result-getter-0:o.a.s.s.TaskSetManager@66] - Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.reflect.UndeclaredThrowableException
	at com.mongodb.hadoop.util.$Proxy71.getConfiguration(Unknown Source)
	at com.mongodb.hadoop.output.MongoRecordWriter.<init>(MongoRecordWriter.java:58)
	at com.mongodb.hadoop.output.MongoRecordWriter.<init>(MongoRecordWriter.java:75)
	at com.mongodb.hadoop.MongoOutputFormat.getRecordWriter(MongoOutputFormat.java:46)
	at io.cdap.cdap.etl.spark.io.TrackingOutputFormat.getRecordWriter(TrackingOutputFormat.java:45)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:100)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalAccessException: Class com.mongodb.hadoop.util.CompatUtils$CompatibleInvocationHandler can not access a member of class io.cdap.cdap.etl.spark.io.TrackingTaskAttemptContext with modifiers "public"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
	at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)
	at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)
	at java.lang.reflect.Method.invoke(Method.java:491)
	at com.mongodb.hadoop.util.CompatUtils$CompatibleInvocationHandler.invoke(CompatUtils.java:43)
	... 13 more

2021-04-07 15:37:12,668 - ERROR [task-result-getter-0:o.a.s.s.TaskSetManager@70] - Task 0 in stage 0.0 failed 1 times; aborting job

[albertshau]

thanks for reporting this, I've also opened https://cdap.atlassian.net/browse/PLUGIN-666 to track this.

Which version of the plugin did you see this with?

[hayj]

The plugin version is 2.0
The CDAP versions are 6.2 OK, 6.3 NOK, 6.5 (develop) NOK
This old plugin from hydrator is deprecated I think, right? https://github.com/cdapio/hydrator-plugins/tree/release/5.1/mongodb-plugins
Why don't use a Mongo JDBC in a generic plugin? Is it a better practice in general? I can't find any free JDBC for mongo but it's easy to implement do you think?

[albertshau]

Yeah the old plugin is deprecated, though I believe it was moved from hydrator-plugins to this repository as a base.

I'm not actually very familiar with Mongo so not sure on the jdbc driver. If you can find one, the generic database source/sink can be used instead of this plugin.