I'm not finding a way of changing the file name generated by glue jobs. It creates files called like 'run-xxxxx' but I want to modify that and use a specific name. Is this possible? PD: I'm using Python script (not scala)
Spark (and all other tools Hadoop ecosystem) use filenames as a mean to parallelise reads and writes; a spark job will produce as many files in a folder as there are partitions in it's RDD/Dataframe (often named part-XXX. When pointing Spark to a new datasource (be it S3, local FS or HDFS), you always point to a folder containing all the part-xxx files.
I don't know what kind of tool you're using, but if it depends on a filename convention then you'll have to rename your files (using your FS client) after the spark session has finished (it can be done in the Driver's code). Be aware that spark may (and usually does) produce multiple files. You can overcome that by calling coalesc on your DataFrame/RDD.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments