Links

SPRKPY1026

pyspark.sql.readwriter.DataFrameReader.csv has a workaround

Category

Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.readwriter.DataFrameReader.csv which has a workaround.
Input code:
stringmap = (spark.read.csv("dbfs:/mnt/datalake/VMS/raw_data/stringmap.csv", sep=',', inferSchema=True, header=True, multiLine=False, escape='\\', quote='"')
.select('attributename','attributevalue','objecttypecode','value'))
Output code:
#EWI: SPRKPY1026 => pyspark.sql.readwriter.DataFrameReader.csv has a workaround, see documentation for more info
stringmap = (spark.read.csv("dbfs:/mnt/datalake/VMS/raw_data/stringmap.csv", sep=',', inferSchema=True, header=True, multiLine=False, escape='\\', quote='"')
.select('attributename','attributevalue','objecttypecode','value'))

Scenario:

def csv(path: str, mode: Optional[str], compression: Optional[str], sep: Optional[str], quote: Optional[str], escape: Optional[str], header: Optional[Union[bool, str]], nullValue: Optional[str] , escapeQuotes: Optional[Union[bool, str]], quoteAll: Optional[Union[bool, str]], dateFormat: Optional[str], timestampFormat: Optional[str], ignoreLeadingWhiteSpace: Optional[Union[bool, str]], ignoreTrailingWhiteSpace: Optional[Union[bool, str]], charToEscapeQuoteEscaping: Optional[str], encoding: Optional[str], emptyValue: Optional[str], lineSep: Optional[str]) Action: The optional parameters "schema", "sep" and "quote" are not supported in snowpark. So try to replace those uses by the following ones: sep(',') -> option("field delimeter", ","), escape = '\\' -> option("escape", "\\"), Quote='"' -> option("FIELD_OPTIONALLY_ENCLOSED_BY",'"'. InferSchema parameter is not supported at all.
import snowflake.snowpark.DataFrameReader as dfr
stringmap = (spark.read.csv("dbfs:/mnt/datalake/VMS/raw_data/stringmap.csv", sep=dfr.option("field delimeter", ","),
header=True, multiLine=False, escape=dfr.option("escape", "\\"), quote=dfr.option("FIELD_OPTIONALLY_ENCLOSED_BY",'"'))
.select('attributename','attributevalue','objecttypecode','value'))

Recommendation