Links

Snowpark Extensions

Library of extensions that adds functions to the API
Snowpark Extensions is a support library that extends the standard Snowpark library by adding different functionalities that are present in Apache Spark but are not currently supported by Snowpark.
The goal of this library is to facilitate the conversion process of projects from Apache Spark to Snowpark.
Check here for more details about the repo.

References

The tool automatically adds the Snowpark Extensions reference to the project configuration file. In this way, the build tool used by the input project automatically takes care of resolving this reference.
Based on the extension of the project configuration file, the tool adds the reference as follows:

build.gradle

dependencies {
implementation 'com.snowflake:snowpark:1.6.2'
implementation 'net.mobilize.snowpark-extensions:snowparkextensions:0.0.9'
...
}

build.sbt

...
libraryDependencies += "com.snowflake" % "snowpark" % "1.6.2"
libraryDependencies += "net.mobilize.snowpark-extensions" % "snowparkextensions" % "0.0.9"
...

pom.xml

<dependencies>
<dependency>
<groupId>com.snowflake</groupId>
<artifactId>snowpark</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>net.mobilize.snowpark-extensions</groupId>
<artifactId>snowparkextensions</artifactId>
<version>0.0.9</version>
</dependency>
...
</dependencies>

Imports

The tool includes these two imports in any processed file for using the extensions.
import com.snowflake.snowpark_extensions.Extensions._
import com.snowflake.snowpark_extensions.Extensions.functions._

Code example

In the following code, hex and isin are supported by Spark, but these are not supported by Snowpark. The code will work because hex and isin are functions included as extensions.

Input code

package com.mobilize.spark
import org.apache.spark.sql._
object Main {
def main(args: Array[String]) : Unit = {
var languageArray = Array("Java");
var languageHex = hex(col("language"));
col("language").isin(languageArray:_*);
}
}

Output code

package com.mobilize.spark
import com.snowflake.snowpark._
import com.snowflake.snowpark_extensions.Extensions._
import com.snowflake.snowpark_extensions.Extensions.functions._
object Main {
def main(args: Array[String]) : Unit = {
var languageArray = Array("Java");
// hex does not exist on Snowpark. It is a extension.
var languageHex = hex(col("language"));
// isin does not exist on Snowpark. It is a extension.
col("language").isin(languageArray :_*)
}
}