火花不支持添加新的列或刪除現有列嵌套結構。特別是,withColumn和下降的方法數據集類不允許您指定一個列名不同於任何頂級列。例如,假設您有一個數據集使用以下模式:

% scala val模式=(新StructType)閥門(閥門“元數據”,(新StructType) (“eventid”、“字符串”,真正的)閥門(“主機名”,“弦”,真正的)閥門(“時間戳”,“弦”,真正的),真的)閥門(閥門“物品”,(新StructType)(“書”,(新StructType)。添加(“費用”,“雙”,真的),真正的閥門)(“紙”,(新StructType)。add(“頁麵”,“int”,真的),真的),真的)schema.treeString

模式看起來像:

根|——元數據:結構(可空= true) | |——eventid:字符串(nullable = true) | |——主機名:字符串(nullable = true) | |——時間戳:字符串(nullable = true) |——項目:結構(可空= true) | |——書:結構(可空= true) | | | -費用:雙(nullable = true) | |——紙:結構(可空= true) | | |——頁麵:整數(nullable = true)

假設你有DataFrame:

% scala val抽樣:抽樣(行)= sc.parallelize (Seq(行(行(“eventid1”、“hostname1”,“timestamp1”)、行(行(100.0)、行(10)))))val df =火花。createDataFrame(抽樣模式)顯示器(df)

你想增加費用列,這是嵌套的書,1%。更新費用列,您可以從現有的列和重建數據集更新的列如下:

% = df scala val更新。selectExpr (" " " named_struct(元數據,元數據,“物品”,named_struct(‘書’,named_struct(“費用”,items.books。費用* 1.01),“紙”,項目。紙))named_struct named_struct”“”) .select(美元)”。元數據”,“美元named_struct.items”) updated.show(假)

然後你會得到結果:

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - + | | |項目元數據+ = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = + = = = = = = = = = = = = = = = = = + | | (eventid1, hostname1, timestamp1) [[101.0], [10]] | + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - +