磚矮子ime 9.0 (Unsupported)

Correction

A previous version of these release notes incorrectly stated that Apache Parquet dependencies were upgraded from 1.10 to 1.12. In fact, Parquet dependencies remain at version 1.10. The incorrect release note has been removed.

New features and improvements

CREATEFUNCTIONsquare(xDOUBLE)RETURNSDOUBLERETURNx*x;SELECTsquare(2);

SELECT*FROMt1,LATERAL(SELECT*FROMt2WHEREt1.c1=t2.c1)

Bug fixes

• You can no longer overwrite a view by creating a view with the same name.

Library upgrades

• Upgraded Python libraries:

  • certifi from 2021.5.30 to 2020.12.5

  • chardet from 3.0.4 to 4.0.0

  • Cython from 0.29.21 to 0.29.23

  • decorator from 4.4.2 to 5.0.6

  • ipython from 7.19.0 to 7.22.0

  • joblib from 0.17.0 to 1.0.1

  • jupyter-client from 6.1.7 to 6.1.12

  • jupyter-core from 4.6.3 to 4.7.1

  • kiwisolver from 1.3.0 to 1.3.1

  • matplotlib from 3.2.2 to 3.4.2

  • pandas from 1.1.5 to 1.2.4

  • pip from 20.2.4 to 21.0.1

  • prompt-toolkit from 3.0.8 to 3.0.17

  • protobuf from 3.17.3 to 3.17.2

  • ptyprocess from 0.6.0 to 0.7.0

  • pyarrow from 1.0.1 to 4.0.0

  • Pygments從2.7.2 to 2.8.1

  • pyzmq from 19.0.2 to 20.0.0

  • requests from 2.24.0 to 2.25.1

  • s3transfer from 0.3.6 to 0.3.7

  • scikit-learn 0.23.2 0.24.1

  • scipy from 1.5.2 to 1.6.2

  • seaborn from 0.10.0 to 0.11.1

  • setuptools from 50.3.1 to 52.0.0

  • statsmodels from 0.12.0 to 0.12.2

  • tornado from 6.0.4 to 6.1

  • virtualenv from 20.2.1 to 20.4.1

  • wheel from 0.35.1 to 0.36.2

• Upgraded R libraries:

  • Matrix from 1.3-3 to 1.3-4

Apache Spark

磚矮子ime 9.0 includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in磚矮子ime 8.4 (Unsupported)以及以下額外的bug修複d improvements made to Spark:

• [SPARK-35886][SQL][3.1] PromotePrecision should not overwrite genCodePromotePrecision should not overwrite genCode

• [SPARK-35879][CORE][SHUFFLE] Fix performance regression caused by collectFetchRequests

• [SPARK-35817][SQL][3.1] Restore performance of queries against wide Avro tables

• [SPARK-35841][SQL] Casting string to decimal type doesn’t work if the…

• [SPARK-35783][SQL] Set the list of read columns in the task configuration to reduce reading of ORC data

• [SPARK-35576][SQL][3.1] Redact the sensitive info in the result of Set command

• [SPARK-35449][SQL][3.1] Only extract common expressions from CaseWhen values if elseValue is set

• [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match

• [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions

• [SPARK-35278][SQL] Invoke should find the method with correct number of parameters

• [SPARK-35226][SQL] Support refreshKrb5Config option in JDBC datasources

• [SPARK-35244][SQL] Invoke should throw the original exception

• [SPARK-35213][SQL] Keep the correct ordering of nested structs in chained withField operations

• [SPARK-35087][UI] Some columns in table Aggregated Metrics by Executor of stage-detail page shows incorrectly.

• [SPARK-35168][SQL] mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

• [SPARK-35127][UI] When we switch between different stage-detail pages, the entry item in the newly-opened page may be blank

• [SPARK-35142][PYTHON][ML] Fix incorrect return type forrawPredictionUDFinOneVsRestModel

• [SPARK-35096][SQL] SchemaPruning should adhere spark.sql.caseSensitive config

• [SPARK-34639][SQL][3.1] RelationalGroupedDataset.alias should not create UnresolvedAlias

• [SPARK-35080][SQL] Only allow a subset of correlated equality predicates when a subquery is aggregated

• [SPARK-35117][UI] Change progress bar back to highlight ratio of tasks in progress

• [SPARK-35136]Remove initial null value of LiveStage.info

• [SPARK-34834][NETWORK] Fix a potential Netty memory leak in TransportResponseHandler

• [SPARK-35045][SQL] Add an internal option to control input buffer in univocity

• [SPARK-35014]Fix the PhysicalAggregation pattern to not rewrite foldable expressions

• [SPARK-35019][PYTHON][SQL] Fix type hints mismatches in pyspark.sql.*

• [SPARK-34926][SQL][3.1] PartitioningUtils.getPathFragment() should respect partition value is null

• [SPARK-34630][PYTHON] Add typehint for pyspark.version

• [SPARK-34963][SQL] Fix nested column pruning for extracting case-insensitive struct field from array of struct

• [SPARK-34988][CORE][3.1] Upgrade Jetty for CVE-2021-28165

• [SPARK-34922][SQL][3.1] Use a relative cost comparison function in the CBO

• [SPARK-34970][SQL][SECURITY][3.1] Redact map-type options in the output of explain()

• [SPARK-34923][SQL] Metadata output should be empty for more plans

• [SPARK-34949][CORE] Prevent BlockManager reregister when Executor is shutting down

• [SPARK-34939][CORE] Throw fetch failure exception when unable to deserialize broadcasted map statuses

• [SPARK-34909][SQL] Fix conversion of negative to unsigned in conv()

• [SPARK-34845][CORE] ProcfsMetricsGetter shouldn’t return partial procfs metrics

• [SPARK-34814][SQL] LikeSimplification should handle NULL

• [SPARK-34876][SQL] Fill defaultResult of non-nullable aggregates

• [SPARK-34829][SQL] Fix higher order function results

• [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

• [SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

• [SPARK-34630](PYTHON] [SQL]添加typehint pyspark.sql.Column.contains

• [SPARK-34763][SQL] col(), $”” and df(“name”) should handle quoted column names properly

• [SPARK-33482][SPARK-34756][SQL] Fix FileScan equality check

• [SPARK-34790][CORE] Disable fetching shuffle blocks in batch when io encryption is enabled

• [SPARK-34803][PYSPARK] Pass the raised ImportError if pandas or pyarrow fail to import

• [SPARK-34225][CORE] Don’t encode further when a URI form string is passed to addFile or addJar

• [SPARK-34811][CORE] Redact fs.s3a.access.key like secret and token

• [SPARK-34796][SQL][3.1] Initialize counter variable for LIMIT code-gen in doProduce()

• [SPARK-34128][SQL] Suppress undesirable TTransportException warnings involved in THRIFT-4805

• [SPARK-34776][SQL] Nested column pruning should not prune Window produced attributes

• [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

• [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

• [SPARK-34719][SQL][3.1] Correctly resolve the view query with duplicated column names

• [SPARK-34766][SQL][3.1] Do not capture maven config for views

• [SPARK-34731][CORE] Avoid ConcurrentModificationException when redacting properties in EventLoggingListener

• [SPARK-34737][SQL][3.1] Cast input float to double inTIMESTAMP_SECONDS

• [SPARK-34749][SQL][3.1] Simplify ResolveCreateNamedStruct

• [SPARK-34768][SQL] Respect the default input buffer size in Univocity

• [SPARK-34770][SQL] InMemoryCatalog.tableExists should not fail if database doesn’t exist

• [SPARK-34504][SQL] Avoid unnecessary resolving of SQL temp views for DDL commands

• [SPARK-34727][SQL] Fix discrepancy in casting float to timestamp

• [SPARK-34723][SQL] Correct parameter type for subexpression elimination under whole-stage

• [SPARK-34724]通過使用getMetho [SQL]修複解釋評價d instead of getDeclaredMethod

• [SPARK-34713][SQL] Fix group by CreateStruct with ExtractValue

• [SPARK-34697][SQL] Allow DESCRIBE FUNCTION and SHOW FUNCTIONS explain about || (string concatenation operator)

• [SPARK-34682][SQL] Use PrivateMethodTester instead of reflection

• [SPARK-34682][SQL] Fix regression in canonicalization error check in CustomShuffleReaderExec

• [SPARK-34681][SQL] Fix bug for full outer shuffled hash join when building left side with non-equal condition

• [SPARK-34545][SQL] Fix issues with valueCompare feature of pyrolite

• [SPARK-34607][SQL][3.1] AddUtils.isMemberClassto fix a malformed class name error on jdk8u

• (火花- 34596)[SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode

• [SPARK-34613][SQL] Fix view does not capture disable hint config

• [SPARK-32924][WEBUI] Make duration column in master UI sorted in the correct order

• [SPARK-34482][SS] Correct the active SparkSession for StreamExecution.logicalPlan

• [SPARK-34567][SQL] CreateTableAsSelect should update metrics too

• [SPARK-34599][SQL] Fix the issue that INSERT INTO OVERWRITE doesn’t support partition columns containing dot for DSv2

• [SPARK-34577][SQL] Fix drop/add columns to a dataset ofDESCRIBENAMESPACE

• [SPARK-34584][SQL] Static partition should also follow StoreAssignmentPolicy when insert into v2 tables

• [SPARK-34555][SQL] Resolve metadata output from DataFrame

• [SPARK-34534]Fix blockIds order when use FetchShuffleBlocks to fetch blocks

• [SPARK-34547][SQL] Only use metadata columns for resolution as last resort

• [SPARK-34417][SQL] org.apache.spark.sql.DataFrameNaFunctions.fillMap fails for column name having a dot

• [SPARK-34561][SQL] Fix drop/add columns from/to a dataset of v2DESCRIBETABLE

• [SPARK-34556][SQL] Checking duplicate static partition columns should respect case sensitive conf

• [SPARK-34392][SQL] Support ZoneOffset +h:mm in DateTimeUtils. getZoneId

• [SPARK-34550][SQL] Skip InSet null value during push filter to Hive metastore

• [SPARK-34543][SQL] Respect thespark.sql.caseSensitiveconfig while resolving partition spec in v1SETLOCATION

• [SPARK-34436][SQL] DPP support LIKE ANY/ALL expression

• [SPARK-34531][CORE] Remove Experimental API tag in PrometheusServlet

• [SPARK-34497][SQL] Fix built-in JDBC connection providers to restore JVM security context changes

• [SPARK-34515][SQL] Fix NPE if InSet contains null value during getPartitionsByFilter

• [SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

• [SPARK-34473][SQL] Avoid NPE in DataFrameReader.schema(StructType)

• [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs

• [SPARK-34373][SQL] HiveThriftServer2 startWithContext may hang with a race issue

• [SPARK-20977][CORE] Use a non-final field for the state of CollectionAccumulator

• [SPARK-34421][SQL] Resolve temporary functions and views in views with CTEs

• [SPARK-34431][CORE] Only loadhive-site.xmlonce

• [SPARK-34405][CORE] Fix mean value of timersLabels in the PrometheusServlet class

• [SPARK-33438][SQL] Eagerly init objects with defined SQL Confs for commandset-v

• [SPARK-34158]Incorrect url of the only developer Matei in pom.xml

• [SPARK-34346][CORE][SQL][3.1] io.file.buffer.size set by spark.buffer.size will override by loading hive-site.xml accidentally may cause perf regression

• [SPARK-34359][SQL][3.1] Add a legacy config to restore the output schema of SHOW DATABASES

• [SPARK-34331][SQL] Speed up DS v2 metadata col resolution

• [SPARK-34318][SQL][3.1] Dataset.colRegex should work with column names and qualifiers which contain newlines

• [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

• [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

• [SPARK-34310][CORE][SQL] Replaces map and flatten with flatMap

• [SPARK-34083][SQL][3.1] Using TPCDS original definitions for char/varchar colums

• [SPARK-34233][SQL][3.1]的字符填充修複肺水腫二進製比較arison

• [SPARK-34270][SS] Combine StateStoreMetrics should not override StateStoreCustomMetric

• [SPARK-34144][SQL] Exception thrown when trying to write LocalDate and Instant values to a JDBC relation

• [SPARK-34273][CORE] Do not reregister BlockManager when SparkContext is stopped

• [SPARK-34262][SQL][3.1] Refresh cached data of v1 table inALTERTABLE..SETLOCATION

• [SPARK-34275][CORE][SQL][MLLIB] Replaces filter and size with count

• [SPARK-34260][SQL] Fix UnresolvedException when creating temp view twice

• [SPARK-33867][SQL] Instant and LocalDate values aren’t handled when generating SQL queries

• [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

• [SPARK-34221][WEBUI] Ensure if a stage fails in the UI page, the corresponding error message can be displayed correctly

• [SPARK-34236][SQL] Fix v2 Overwrite w/ null static partition raise Cannot translate expression to source filter: null

• [SPARK-34212][SQL] Fix incorrect decimal reading from Parquet files

• [SPARK-34244][SQL] Remove the Scala function version of regexp_extract_all

• [SPARK-34235][SS] Make spark.sql.hive as a private package

• [SPARK-34232][CORE] Redact SparkListenerEnvironmentUpdate event in log

• [SPARK-34229][SQL] Avro should read decimal values with the file schema

• [SPARK-34223][SQL] FIX NPE for static partition with null in InsertIntoHadoopFsRelationCommand

• [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too

• [SPARK-34203][SQL] Convertnullpartition values to__HIVE_DEFAULT_PARTITION__in v1In-Memorycatalog

• [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation

• [SPARK-34133][AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching

• [SPARK-34187][SS] Use available offset range obtained during polling when checking offset validation

• [SPARK-34052][SQL][3.1] store SQL text for a temp view created using “CACHE TABLE .. AS SELECT …”

• [SPARK-34213][SQL] Refresh cached data of v1 table inLOADDATA

• [SPARK-34191][PYTHON][SQL] Add typing for udf overload

• [SPARK-34200][SQL] Ambiguous column reference should consider attribute availability

• [SPARK-33813][SQL][3.1] Fix the issue that JDBC source can’t treat MS SQL Server’s spatial types

• [SPARK-34178][SQL] Copy tags for the new node created by MultiInstanceRelation.newInstance

• [SPARK-34005][CORE][3.1] Update peak memory metrics for each Executor on task end

• [SPARK-34115][CORE] Check SPARK_TESTING as lazy val to avoid slowdown

• [SPARK-34153][SQL][3.1][3.0] Remove unusedgetRawTable()fromHiveExternalCatalog.alterPartitions()

• [SPARK-34130][SQL] Impove preformace for char varchar padding and length check with StaticInvoke

• (火花- 34027)[SQL][3.1] Refresh cache inALTERTABLE..RECOVERPARTITIONS

• [SPARK-34151][SQL] Replacesjava.io.File.toURLwithjava.io.File.toURI.toURL

• [SPARK-34140][SQL][3.1] Move QueryCompilationErrors.scala to org/apache/spark/sql/errors

• [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector

• [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

• [SPARK-34118][CORE][SQL][3.1] Replaces filter and check for emptiness with exists or forall

• [SPARK-34114][SQL] should not trim right for read-side char length check and padding

• [SPARK-34086][SQL][3.1] RaiseError generates too much code and may fails codegen in length check for char varchar

• [SPARK-34075][SQL][CORE] Hidden directories are being listed for partition inference

• [SPARK-34076][SQL] SQLContext.dropTempTable fails if cache is non-empty

• (火花- 34084)[SQL][3.1] Fix auto updating of table stats inALTERTABLE..ADDPARTITION

• [SPARK-34090][SS] Cache HadoopDelegationTokenManager.isServiceEnabled result used in KafkaTokenUtil.needTokenUpdate

• [SPARK-34069][CORE] Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL

• [SPARK-34091][SQL] Shuffle batch fetch should be able to disable after it’s been enabled

• [SPARK-34059][SQL][CORE][3.1] Use for/foreach rather than map to make sure execute it eagerly

• [SPARK-34002][SQL] Fix the usage of encoder in ScalaUDF

• [SPARK-34060][SQL][3.1] Fix Hive table caching while updating stats byALTERTABLE..DROPPARTITION

• [SPARK-31952][SQL] Fix incorrect memory spill metric when doing Aggregate

• [SPARK-33591][SQL][3.1] Recognizenullin partition spec values

• [SPARK-34055][SQL][3.1] Refresh cache inALTERTABLE..ADDPARTITION

• [SPARK-34039][SQL][3.1] ReplaceTable should invalidate cache

• [SPARK-34003][SQL] Fix Rule conflicts between PaddingAndLengthCheckForCharVarchar and ResolveAggregateFunctions

• [SPARK-33938][SQL][3.1] Optimize Like Any/All by LikeSimplification

• [SPARK-34021][R] Fix hyper links in SparkR documentation for CRAN submission

• [SPARK-34011][SQL][3.1][3.0] Refresh cache inALTERTABLE..RENAMETOPARTITION

• [SPARK-33948][SQL] Fix CodeGen error of MapObjects.doGenCode method in Scala 2.13

• [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

• [SPARK-33029][CORE][WEBUI] Fix the UI executor page incorrectly marking the driver as excluded

• [SPARK-34015][R] Fixing input timing in gapply

• [SPARK-34012][SQL] Keep behavior consistent when confspark.sql.legacy.parser.havingWithoutGroupByAsWhereis true with migration guide

• [SPARK-33844][SQL][3.1] InsertIntoHiveDir command should check col name too

• [SPARK-33935][SQL] Fix CBO cost function

• [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql

• [SPARK-34000][CORE] Fix stageAttemptToNumSpeculativeTasks java.util.NoSuchElementException

• [SPARK-33992][SQL] override transformUpWithNewOutput to add allowInvokingTransformsInAnalyzer

• [SPARK-33894][SQL] Change visibility of private case classes in mllib to avoid runtime compilation errors with Scala 2.13

• [SPARK-33950][SQL][3.1][3.0] Refresh cache in v1ALTERTABLE..DROPPARTITION

• [SPARK-33980][SS] Invalidate char/varchar in spark.readStream.schema

• [SPARK-33945][SQL][3.1] Handles a random seed consisting of an expr tree

• [SPARK-33398]Fix loading tree models prior to Spark 3.0

• [SPARK-33963][SQL] CanonicalizeHiveTableRelationw/o table stats

• [SPARK-33906][WEBUI] Fix the bug of UI Executor page stuck due to undefined peakMemoryMetrics

• [SPARK-33944][SQL] Incorrect logging for warehouse keys in SharedState options

• [SPARK-33936][SQL][3.1] Add the version when connector’s interfaces were added

• [SPARK-33916][CORE] Fix fallback storage offset and improve compression codec test coverage

• [SPARK-33899][SQL][3.1] Fix assert failure in v1 SHOW TABLES/VIEWS onspark_catalog

• [SPARK-33901][SQL] Fix Char and Varchar display error after DDLs

• [SPARK-33897][SQL] Can’t set option ‘cross’ in join method

• [SPARK-33907][SQL][3.1] Only prune columns of JsonToStructs if parsing options is empty

• [SPARK-33621][SPARK-33784][SQL][3.1] Add a way to inject data source rewrite rules

• [SPARK-33900][WEBUI] Show shuffle read size / records correctly when only remotebytesread is available

• [SPARK-33892][SQL] Display char/varchar in DESC and SHOW CREATE TABLE

• [SPARK-33895][SQL] Char and Varchar fail in MetaOperation of ThriftServer

• [SPARK-33659][SS] Document the current behavior for DataStreamWriter.toTable API

• (火花- 33893)[CORE] Exclude fallback block manager from executorList

• [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends

• [SPARK-33889][SQL][3.1] Fix NPE fromSHOWPARTITIONSV2表上

• [SPARK-33879][SQL] Char Varchar values fails w/ match error as partition columns

• [SPARK-33877][SQL] SQL reference documents for INSERT w/ a column list

• [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

• [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

• [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

• [SPARK-33834][SQL] Verify ALTER TABLE CHANGE COLUMN with Char and Varchar

• [SPARK-33853][SQL] EXPLAIN CODEGEN and BenchmarkQueryTest don’t show subquery code

• [SPARK-33836][SS][PYTHON] Expose DataStreamReader.table and DataStreamWriter.toTable

• [SPARK-33829][SQL][3.1] Renaming v2 tables should recreate the cache

• [SPARK-33756][SQL] Make BytesToBytesMap’s MapIterator idempotent

• [SPARK-33850][SQL] EXPLAIN FORMATTED doesn’t show the plan for subqueries if AQE is enabled

• [SPARK-33841][CORE][3.1] Fix issue with jobs disappearing intermittently from the SHS under high load

• [SPARK-33593][SQL] Vector reader got incorrect data with binary partition value

• [SPARK-26341][WEBUI] Expose executor memory metrics at the stage level, in the Stages tab

• [SPARK-33831][UI] Update to jetty 9.4.34

• [SPARK-33822][SQL] Use theCastSupport.castmethod in HashJoin

• [SPARK-33774][UI][CORE] Back to Master” returns 500 error in Standalone cluster

• [SPARK-26199][SPARK-31517][R] Fix strategy for handling … names in mutate

• [SPARK-33819][CORE][3.1] SingleFileEventLogFileReader/RollingEventLogFilesFileReader should bepackageprivate

• [SPARK-33697][SQL] RemoveRedundantProjects should require column ordering by default

• [SPARK-33752][SQL][3.1] Avoid the getSimpleMessage of AnalysisException adds semicolon repeatedly

• [SPARK-33788][SQL][3.1][3.0][2.4] Throw NoSuchPartitionsException from HiveExternalCatalog.dropPartitions()

• [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command

• [SPARK-33786][SQL] The storage level for a cache should be respected when a table name is altered

• [SPARK-33273][SQL] Fix a race condition in subquery execution

• [SPARK-33653][SQL][3.1] DSv2: REFRESH TABLE should recache the table itself

• [SPARK-33777][SQL] Sort output of V2 SHOW PARTITIONS

• [SPARK-33733][SQL] PullOutNondeterministic should check and collect deterministic field

• [SPARK-33764][SS] Make state store maintenance interval as SQL config

• [SPARK-33729][SQL] When refreshing cache, Spark should not use cached plan when recaching data

• [SPARK-33742][SQL][3.1] Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions()

• [SPARK-33706][SQL] Require fully specified partition identifier in partitionExists()

• [SPARK-33740][SQL] hadoop configs in hive-site.xml can overrides pre-existing hadoop ones

• [SPARK-33692][SQL] View should use captured catalog and namespace to lookup function

• [SPARK-33669]Wrong error message from YARN application state monitor when sc.stop in yarn client mode

• [SPARK-32110][SQL] normalize special floating numbers in HyperLogLog++

• [SPARK-33677][SQL] Skip LikeSimplification rule if pattern contains any escapeChar

• [SPARK-33693][SQL] deprecate spark.sql.hive.convertCTAS

• [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results

• [SPARK-32680][SQL] Don’t Preprocess V2 CTAS with Unresolved Query

• [SPARK-33676][SQL] Require exact matching of partition spec to the schema in V2ALTERTABLE..ADD/DROPPARTITION

• [SPARK-33670][SQL] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED

• [SPARK-33663][SQL] Uncaching should not be called on non-existing temp views

• [SPARK-33667][SQL] Respect thespark.sql.caseSensitiveconfig while resolving partition spec in v1SHOWPARTITIONS

• [SPARK-33652][SQL] DSv2: DeleteFrom should refresh cache

Maintenance updates

See磚矮子ime 9.0 maintenance updates.

System environment

• Operating System: Ubuntu 20.04.2 LTS

• Java: Zulu 8.54.0.21-CA-linux64

• Scala: 2.12.10

• Python: 3.8.10

• R: 4.1.0 (2021-05-18)

• Delta Lake1.0.0