Friday 2 December 2016
Filter Spark Dataframe column using Scala
If you are using a Spark dataframe, you can filter Spark DataFrame using Scala with Spark.Sql in an easy way like below:
Filter a DataFrame which contains ""
DataFrame.registerTempTable("tempDfTable")
SqlContext.Sql("""Select * from tempDfTable where tempDfTable.col!="" """)
Filter a DataFrame column which contains null
DataFrame.registerTempTable("tempDfTable")
SqlContext.Sql("""Select * from tempDfTable where tempDfTable.col is null """)
You can write this statement with Single Quotes also
SqlContext.Sql("Select * from tempDfTable where tempDfTable.col is null")
Filter a DataFrame column which will not contain null values
DataFrame.registerTempTable("tempDfTable")
SqlContext.Sql("""Select * from tempDfTable where tempDfTable.col is not null """)
Filter a DataFrame which contains ""
DataFrame.registerTempTable("tempDfTable")
SqlContext.Sql("""Select * from tempDfTable where tempDfTable.col!="" """)
Filter a DataFrame column which contains null
DataFrame.registerTempTable("tempDfTable")
SqlContext.Sql("""Select * from tempDfTable where tempDfTable.col is null """)
You can write this statement with Single Quotes also
SqlContext.Sql("Select * from tempDfTable where tempDfTable.col is null")
Filter a DataFrame column which will not contain null values
DataFrame.registerTempTable("tempDfTable")
SqlContext.Sql("""Select * from tempDfTable where tempDfTable.col is not null """)
Subscribe to:
Post Comments
(
Atom
)
No comments :
Post a Comment