Spark Structured Stream qubole Kinesis connector errors out with "Got an exception while fetching credentials"

Question

I am using the following code to write to Kinesis from a spark structured stream code. It errors out with following error. The AWS credentials have admin access. I am able to use aws console using that. What could be the issue here?

22/03/16 13:46:34 ERROR AWSInstanceProfileCredentialsProviderWithRetries: Got an exception while fetching credentials org.apache.spark.sql.kinesis.shaded.amazonaws.SdkClientException: Unable to load credentials from service endpoint

val finalDF = rawDF.select(expr("CAST(rand() AS STRING) as partitionKey"),
   to_json(struct("*")).alias("data"))
finalDF.printSchema()
val query = finalDF.writeStream
  .outputMode("update")
  .format("kinesis")
  .option("streamName", "sparkstream2")
  .option("endpointUrl", "https://kinesis.us-east-1.amazonaws.com")
  .option("region", "us-east-1")
  .option("awsAccessKey", "")  // Creds removed
  .option("awsSecretKey", "")  // Creds removed
  .option("checkpointLocation", "chk-point-dir")
  .start()
query.awaitTermination()
spark.stop()

Printschema output looks as follows root |-- partitionKey: string (nullable = false) |-- data: string (nullable = true)

I am using the connector from qubole https://github.com/qubole/kinesis-sql

Answer 1

I had this issue too - add on .option("awsUseInstanceProfile", "false"). The kinesis-sql package doesn't handle AWS credentials by default as one would expect. I found this GitHub issue for a lead here.