Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0301· OpenAIAi2 WildChat-4.8M (AI2)

scala spark多列怎么转多行

  1. User

    scala spark多列怎么转多行

  2. Assistant

    如果您已经有一张多列的Spark DataFrame,想要把其中多列转成多行,可以使用Spark DataFrame的stack方法。

    比如,我们有这样一张含有多列的DataFrame:

    val df = Seq(
      (1, "John", "Doe"),
      (2, "Jane", "Smith"),
      (3, "Bob", "Johnson")
    ).toDF("id", "first_name", "last_name")
    

    现在想要把first_name和last_name这两列转成多行,可以使用stack方法:

    import org.apache.spark.sql.functions._
    
    val stackedDf = df.selectExpr("id", "stack(2, 'first_name', first_name, 'last_name', last_name) as (col, val)")
      .select("id", "col", "val")
    
    stackedDf.show()
    

    这样就把first_name和last_name这两列转成多行,每行包含id、列名(即col)和对应的值(即val)。

    输出结果:

    +---+----------+------+
    | id|       col|   val|
    +---+----------+------+
    |  1|first_name|  John|
    |  1| last_name|   Doe|
    |  2|first_name|  Jane|
    |  2| last_name| Smith|
    |  3|first_name|   Bob|
    |  3| last_name|Johnson|
    +---+----------+------+
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.