How to Join Multiple Columns in Spark SQL using Java for filtering in DataFrame

Gokul Published at Java

270

Gokul :

DataFrame a = contains column x,y,z,k

DataFrame b = contains column x,y,a

a.join(b,<condition to use in java to use x,y >) ???

I tried using

a.join(b,a.col("x").equalTo(b.col("x")) && a.col("y").equalTo(b.col("y"),"inner")

But Java is throwing error saying && is not allowed.

zero323 :

Spark SQL provides a group of methods on Column marked as java_expr_ops which are designed for Java interoperability. It includes and (see also or) method which can be used here:

a.col("x").equalTo(b.col("x")).and(a.col("y").equalTo(b.col("y"))

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at2020-06-24

Comments

0 comments

How to concatenate spark dataframe columns using Spark sql in databricks

How to filtering pandas dataframe by multiple columns

How to convert single column to multiple columns using spark dataframe

Zip and Explode multiple Columns in Spark SQL Dataframe

how to concat all columns in a spark dataframe, using java?

How to construct a column based on other columns using dataframe in Spark Java?

Dropping multiple columns of Spark DataFrame in Java

Spark DataFrame and renaming multiple columns (Java)

pyspark dataframe filtering on multiple columns

How to join DataFrame with multiple conditions on different columns?

filtering or subsetting a dataframe using multiple columns matching values in a list

How to exclude multiple columns in Spark dataframe in Python

How to split column in Spark Dataframe to multiple columns

Split a column in multiple columns using Spark SQL

How to join using a nested column in Spark dataframe

SQL : Filtering with multiple columns in a subquery

join in a dataframe spark java

How to join two tables by multiple columns in SQL?

dplyr filtering on multiple columns using "%in%"

How to concatenate null columns in spark dataframe in java?

How to pivot on multiple columns in Spark SQL?

How to select all columns of a dataframe in join - Spark-scala

How to explode an array into multiple columns in Spark Java

perform join on multiple DataFrame in spark

pySpark join dataframe on multiple columns

pandas dataframe filtering multiple columns and rows

Conditional filtering on multiple columns for a pandas dataframe

Filtering pandas dataframe with multiple Boolean columns

DataFrame filter with multiple columns not working using && operator in Spark 2.2 (scala)

TOP Ranking

Article

How to Join Multiple Columns in Spark SQL using Java for filtering in DataFrame

How to Join Multiple Columns in Spark SQL using Java for filtering in DataFrame

pump.io port in URL

How to import an asset in swift using Bundle.main.path() in a react-native native module

Failed to listen on localhost:8000 (reason: Cannot assign requested address)

Double spacing in rmarkdown pdf

SQL Server : need add a dot before two last character

C++ 16 bit grayscale gradient image from 2D array

JMeter: Why get error when try to save test plan

JWT gives JsonWebTokenError "invalid token"

How to make thrown errors visible outside of a Promise?

How to tell if iOS Today Widget is being updated in the background?

Calling Doctrine clear() with an argument is deprecated

Capybara Selenium Chrome opens About Google Chrome

How to update azerothcore-wotlk docker container

Adding Ripple Effect to RecyclerView item

mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on '127.0.0.1:3306' (111 Connection refused)

Error while applying filter on dataframe - PySpark

Unable to add slack to bluemix project

MyPy fails dataclass argument with optional list of objects type

How can I validate and parse phone numbers to extract their country calling code and area code?

Single Sign-On in Spring by using SAML Extension and Shibboleth

python how to create many-to-many of lists inside one list