spark select withColumn 例子
- User
spark select withColumn 例子
- Assistant
假设我们有一个名为“people”的数据框,其中包含以下列:name(字符串类型),age(整数类型)和job(字符串类型)。我们想要添加一个新列“isAdult”,该列将包含一个布尔值,指示每个人是否已经成年。
使用withColumn函数,可以轻松地添加此新列:
from pyspark.sql.functions import when, col # 创建一个名为“people”的数据框 people = spark.createDataFrame([("Alice", 25, "Engineer"), ("Bob", 18, "Student"), ("Charlie", 50, "Manager")], ["name", "age", "job"]) # 添加一个新列“isAdult”,根据年龄判断是否成年 people = people.withColumn("isAdult", when(col("age") >= 18, True).otherwise(False)) # 显示数据框 people.show()运行结果:
+-------+---+--------+-------+ | name|age| job|isAdult| +-------+---+--------+-------+ | Alice| 25|Engineer| true| | Bob| 18| Student| true| |Charlie| 50| Manager| true| +-------+---+--------+-------+通过使用withColumn函数,我们成功地添加了一个新列“isAdult”,并根据每个人的年龄确定了相应的布尔值。
- User
spark select withColumn 例子
- Assistant
假设我们有一个名为“sales”的数据框,其中包含以下列:date(日期类型),category(字符串类型),revenue(整数类型)。我们想要添加一个新列“profit”,该列将包含一个计算公式,用于计算每个销售类别的利润。
使用withColumn函数,可以轻松地添加此新列:
# 创建一个名为“sales”的数据框 sales = spark.createDataFrame([("2022-01-01", "Electronics", 100), ("2022-01-02", "Clothing", 50), ("2022-01-03", "Electronics", 200)], ["date", "category", "revenue"]) # 添加一个新列“profit”,根据公式计算利润 sales = sales.withColumn("profit", col("revenue") * 0.8) # 显示数据框 sales.show()运行结果:
+----------+-----------+-------+------+ | date| category|revenue|profit| +----------+-----------+-------+------+ |2022-01-01|Electronics| 100| 80.0| |2022-01-02| Clothing| 50| 40.0| |2022-01-03|Electronics| 200| 160.0| +----------+-----------+-------+------+通过使用withColumn函数,我们已成功添加了一个新列“profit”,该列包含每个销售类别的利润,该利润通过计算公式相应计算得出。
- User
spark 使用select和 withColumn 的例子
- Assistant
假设我们有一个名为“students”的数据框,其中包含以下列:name(字符串类型),age(整数类型),gender(字符串类型)和grade(整数类型)。我们想要筛选出所有女性学生,并添加一个新列“passed”,该列在Student的得分等级grade是A或B时为True,否则为False。
我们将需要结合使用select函数和withColumn函数,如下所示:
from pyspark.sql.functions import col, when # 创建一个名为“students”的数据框 students = spark.createDataFrame([("Alice", 20, "Female", "A"), ("Bob", 18, "Male", "C"), ("Charlie", 22, "Male", "B"), ("David", 19, "Male", "B"), ("Eva", 21, "Female", "A")], ["name", "age", "gender", "grade"]) # 选择所有女性学生并添加一个新列“passed”,在得分等级为A或B时为True,否则为False female_students = students.select("name", "age", "gender", "grade").filter(col("gender") == "Female").withColumn("passed", when(col("grade").isin(["A", "B"]), True).otherwise(False)) # 显示数据框 female_students.show()运行结果:
+-----+---+------+-----+ | name|age|gender|grade| +-----+---+------+-----+ |Alice| 20|Female| A| | Eva| 21|Female| A| +-----+---+------+-----+通过使用select函数和withColumn函数相结合,我们已成功筛选出了所有女性学生,并添加了一个新列“passed”,该列在学生的得分等级为A或B时为True,否则为False。
- User
更多例子
- Assistant
以下是更多关于Spark中select函数和withColumn函数的例子:
- 使用select函数选择某些列:
from pyspark.sql.functions import col # 创建一个名为“students”的数据框 students = spark.createDataFrame([("Alice", 20, "Female", "A"), ("Bob", 18, "Male", "C"), ("Charlie", 22, "Male", "B"), ("David", 19, "Male", "B"), ("Eva", 21, "Female", "A")], ["name", "age", "gender", "grade"]) # 选择名字和年龄列 selected_columns = students.select(col("name"), col("age")) # 显示数据框 selected_columns.show()运行结果:
+-------+---+ | name|age| +-------+---+ | Alice| 20| | Bob| 18| |Charlie| 22| | David| 19| | Eva| 21| +-------+---+- 使用withColumnRenamed函数将某一列重命名:
# 创建一个名为“students”的数据框 students = spark.createDataFrame([("Alice", 20, "Female", "A"), ("Bob", 18, "Male", "C"), ("Charlie", 22, "Male", "B"), ("David", 19, "Male", "B"), ("Eva", 21, "Female", "A")], ["name", "age", "gender", "grade"]) # 将列名“name”改为“student_name” students = students.withColumnRenamed("name", "student_name") # 显示数据框 students.show()运行结果:
+-------------+---+------+-----+ | student_name|age|gender|grade| +-------------+---+------+-----+ | Alice| 20|Female| A| | Bob| 18| Male| C| | Charlie| 22| Male| B| | David| 19| Male| B| | Eva| 21|Female| A| +-------------+---+------+-----+- 使用withColumn函数将某一列值减去一个固定值:
from pyspark.sql.functions import lit # 创建一个名为“students”的数据框 students = spark.createDataFrame([("Alice", 20, "Female", "A"), ("Bob", 18, "Male", "C"), ("Charlie", 22, "Male", "B"), ("David", 19, "Male", "B"), ("Eva", 21, "Female", "A")], ["name", "age", "gender", "grade"]) # 将“age”列中的值减去5 students = students.withColumn("age", col("age") - lit(5)) # 显示数据框 students.show()运行结果:
+-------+---+------+-----+ | name|age|gender|grade| +-------+---+------+-----+ | Alice| 15|Female| A| | Bob| 13| Male| C| |Charlie| 17| Male| B| | David| 14| Male| B| | Eva| 16|Female| A| +-------+---+------+-----+这些例子展示了如何使用Spark中的select函数和withColumn函数。这两个函数非常有用,可以帮助我们轻松地对数据进行筛选和变换操作。
- User
spark 使用select和 withColumn 的例子
- Assistant
假设我们有一个名为“orders”的数据框,其中包含以下列:order_id(整数类型),customer_id(整数类型),order_status(字符串类型),order_date(日期时间类型)和order_total(整数类型)。我们想要选择与特定客户相关的订单,并添加一个新列“discounted_total”,该列将包含每个订单的总计减去10的结果。
我们可以结合使用select和withColumn函数,如下所示:
from pyspark.sql.functions import col # 创建一个名为“orders”的数据框 orders = spark.createDataFrame([(1, 101, "Pending", "2022-01-01 12:00:00", 100), (2, 102, "Shipped", "2022-01-02 12:00:00", 200), (3, 101, "Canceled", "2022-01-03 12:00:00", 150), (4, 103, "Pending", "2022-01-04 12:00:00", 300), (5, 101, "Completed", "2022-01-05 12:00:00", 250)], ["order_id", "customer_id", "order_status", "order_date", "order_total"]) # 选择与客户ID 101 相关的订单,并添加一个新列“discounted_total”,该列将包含每个订单的总计减去10的结果。 customer_orders = orders.select("order_id", "customer_id", "order_status", "order_date", col("order_total") - 10).filter(col("customer_id") == 101) # 将新列名称更改为“discounted_total” customer_orders = customer_orders.withColumnRenamed("(order_total - 10)", "discounted_total") # 显示数据框 customer_orders.show()运行结果:
+--------+-----------+------------+-------------------+----------------+ |order_id|customer_id|order_status| order_date|discounted_total| +--------+-----------+------------+-------------------+----------------+ | 1| 101| Pending|2022-01-01 12:00:00| 90| | 3| 101| Canceled|2022-01-03 12:00:00| 140| | 5| 101| Completed|2022-01-05 12:00:00| 240| +--------+-----------+------------+-------------------+----------------+通过使用select和withColumn函数结合使用,我们已成功选择与客户ID 101相关的订单,并添加了一个新列“discounted_total”,该列将包含每个订单的总计减去10的结果。