wrong schema while reading csv file as a dataframe

ben

trying to read a csv file into a dataframe simple code

df = spark.read.csv("1.csv")

i got

    df.printSchema()
root
 |-- _c0: string (nullable = true)

also i try this

db = spark.read.csv("1.csv", header=True, inferSchema= "True")
db.printSchema()
root
 |--                   id                  |                      date                      |                              cases                               |                      country                      |                       deaths                       |   cities   |    per_cap     | 

Thanks in advance for your help

Steven

apparently, your line seperator is a pipe |.

try:

db = spark.read.csv("1.csv", sep='|', header=True, inferSchema= "True")

for col in db.columns:
    db = db.withColumnRenamed(col, col.strip())

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

How to skip lines while reading a CSV file as a dataFrame using PySpark?

skip second row of dataframe while reading csv file in python

Error while reading csv file and returning dataframe in python

reading csv to dataframe with dynamic custom schema with pyspark

reading CSV file into julia DataFrame

Wrong file path while reading from UI

How to drop malformed rows while reading csv with schema Spark?

How to Handle different date Format in csv file while reading Dataframe in SPARK using option("dateFormat")?

dataframe importing column's first value as column name while reading a CSV file

How to keep the digit limit if a column in Dataframe while reading from csv file?

In GCP python cloud function, dataframe is putting ' ' in the end while reading csv file

pyspark load csv file into dataframe using a schema

Unable to define schema for a csv file in dataframe

In Scan EOF error while reading CSV file

KeyError while reading a CSV file in Python

Error while reading csv file in R

Reading from a CSV file while it is being written to

snowflake handle null while reading csv file

Error while reading a CSV file in Spark - Scala

Python: Replace values while reading CSV file

Strange characters while reading gzipped CSV file

Error while reading csv file using python

Strange character while reading a CSV file

reading csv file to pandas dataframe as float

Reading CSV file in loop Dataframe (Julia)

Reading in csv file as dataframe from hdfs

Reading a csv file with a list of elements into pandas dataframe

Reading uploaded csv file into pandas dataframe

How to specify schema while reading parquet file with pyspark?

TOP Ranking

  1. 1

    Failed to listen on localhost:8000 (reason: Cannot assign requested address)

  2. 2

    pump.io port in URL

  3. 3

    How to import an asset in swift using Bundle.main.path() in a react-native native module

  4. 4

    Loopback Error: connect ECONNREFUSED 127.0.0.1:3306 (MAMP)

  5. 5

    Compiler error CS0246 (type or namespace not found) on using Ninject in ASP.NET vNext

  6. 6

    BigQuery - concatenate ignoring NULL

  7. 7

    Spring Boot JPA PostgreSQL Web App - Internal Authentication Error

  8. 8

    ggplotly no applicable method for 'plotly_build' applied to an object of class "NULL" if statements

  9. 9

    ngClass error (Can't bind ngClass since it isn't a known property of div) in Angular 11.0.3

  10. 10

    How to remove the extra space from right in a webview?

  11. 11

    Change dd-mm-yyyy date format of dataframe date column to yyyy-mm-dd

  12. 12

    Jquery different data trapped from direct mousedown event and simulation via $(this).trigger('mousedown');

  13. 13

    maven-jaxb2-plugin cannot generate classes due to two declarations cause a collision in ObjectFactory class

  14. 14

    java.lang.NullPointerException: Cannot read the array length because "<local3>" is null

  15. 15

    How to use merge windows unallocated space into Ubuntu using GParted?

  16. 16

    flutter: dropdown item programmatically unselect problem

  17. 17

    Pandas - check if dataframe has negative value in any column

  18. 18

    Nuget add packages gives access denied errors

  19. 19

    Can't pre-populate phone number and message body in SMS link on iPhones when SMS app is not running in the background

  20. 20

    Generate random UUIDv4 with Elm

  21. 21

    Client secret not provided in request error with Keycloak

HotTag

Archive