Here we are going to use the SQL col function, this function refers the column name of the dataframe with dataframe_object.col. It returns all data that has a match under the join condition (predicate in the `on' argument) from both sides of the table. I was getting "AssertionError: joinExprs should be Column" Instead, I used raw sql to join the data frames as shown below df.registerTempTable ("df") df3.registerTempTable ("df3") sqlContext.sql ("Select df.name,df3.age from df outer join df3 on df.name = df3.name and df.age =df3.age").collect () Share PySpark - join - myTechMint 1 min read. pyspark.sql.DataFrame.crossJoin — PySpark 3.1.1 documentation PySpark Filter is applied with the Data Frame and is used to Filter Data all along so that the needed data is left for processing and the rest data is not used. 3. We can alter or update any column PySpark DataFrame based on the condition required. It adds the data that satisfies the relation to . PySpark where Clause - Linux Hint It needs to join the following: Always on dset_cob_dt and tlsn_trd_id ; In addition if the meas_data.tlsn_leg_id is not null it needs to join on tlsn_leg_id as well ; and in addition to that also on tlsn_vrsn_num if similar to the last one meas_data.tlsn_vrsn_num is not null. Update NULL values in Spark DataFrame. Where () is a method used to filter the rows from DataFrame based on the given condition. DataFrame join with OR condition - Cloudera Community - 125960 Let us understand the usage of BETWEEN in conjunction with AND while filtering data from Data Frames. Inner join. apache spark - Pyspark SQL conditional join issues - Stack Overflow I'm using a DataFrame with StructType and ArrayType columns here because I'll be covering examples with both struct and array types. It is also known as simple join or Natural Join. DataFrame.crossJoin(other) [source] ¶. Method 1: Using drop () function. 2. pyspark when otherwise multiple conditions The filter function was added in Spark 3.1, whereas the filter method has been around since the early days of Spark (1.3).