SPRKPY1029
pyspark.sql.readwriter.DataFrameReader.parquet has a workaround
Warning.
This issue appears when the tool detects the usage of pyspark.sql.readwriter.DataFrameReader.parquet which has a workaround.
Input code:
accounts = spark.read.parquet("dbfs:/mnt/datalake/DMS/raw_data/account.parquet", sep=',', inferSchema=True, header=True, multiLine=False, escape='\\', quote='"')
Output code:
#EWI: SPRKPY1029 => pyspark.sql.readwriter.DataFrameReader.parquet has a workaround, see documentation for more info
accounts = spark.read.parquet("dbfs:/mnt/datalake/DMS/raw_data/account.parquet", sep=',', inferSchema=True, header=True, multiLine=False, escape='\\', quote='"')
parquet(path: str, mode: Optional[str], partitionBy: Optional[Union[str, List[str]]], compression: Optional[str])
Action: The optional parameters "schema", "sep" and "quote" are not supported in snowpark. So try to replace those uses by the following ones: sep(',') -> option("field delimeter", ","), escape = '\\' -> option("escape", "\\"), Quote='"' -> option("FIELD_OPTIONALLY_ENCLOSED_BY",'"'_. InferSchema parameter is not supported at all.
import snowflake.snowpark.DataFrameReader as dfr
accounts = spark.read.parquet("dbfs:/mnt/datalake/DMS/raw_data/account.parquet",
dfr.option("field delimeter", ","), header=True, multiLine=False,
escape=dfr.option("escape", "\\"), quote=dfr.option("FIELD_OPTIONALLY_ENCLOSED_BY",'"'))
- For more support, you can email us at [email protected] or post a message to our forums. If you have a contract for support with Mobilize.Net, reach out to your sales engineer and they can direct your support needs.
Last modified 4mo ago