'Missing data after uploading CSV to s3 (Kotlin)
after copying table into csv file, i uploaded csv file to s3 (batch).
i checked csv file has 100 rows, but when is uploaded on s3, it has 95 rows.
missing number is random like 30->26 , 7420590 -> 7420566 , 7730590 -> 7730577.
it seems like the size of file doesn't matter
what is wrong with this?
override fun run(baseDate: String) {
val sqlSession: SqlSession = datastoreSqlSessionFactory.openSession(ExecutorType.BATCH, true)
val con = sqlSession.connection
val copyManager = CopyManager(con.unwrap(BaseConnection::class.java))
val file = File("./buildings.csv")
val stream: OutputStream = BufferedOutputStream(FileOutputStream(file.absolutePath))
val bucketName = "manage-batch-dev"
val key = "/data"
val copySql = String.format(
"copy buildings_table to stdout"
)
try {
copyManager.copyOut(copySql, stream)
s3.upload(bucketName, key, file.absolutePath)
} catch (e: Exception) {
log.error("BuildingsOutlinesSummariesCopyOutBatch Error:${e.message}", e)
throw e
} finally {
stream.close()
sqlSession.close()
}
file.delete()
}
and s3.upload
@Throws(IOException::class, AmazonServiceException::class)
fun upload(bucketName: String, key: String, filePath: String): PutObjectResult {
try {
val request = PutObjectRequest(bucketName, key, File(filePath))
val result = client.putObject(request)
return result
} catch (e: AmazonServiceException) {
throw e
} catch (e: java.lang.Exception) {
throw e
}
}
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
