Secondary Sort: Introduction in map reduce in hadoop . give full code to implement…
- User
Secondary Sort: Introduction in map reduce in hadoop . give full code to implement secondary sort in haddop in java. it should take input as file and give output in file. take temparature example . also giver command to excute that. use 3 reducers here.
- Assistant
Implementing Secondary Sort in Hadoop MapReduce Using Java
Introduction
In Hadoop MapReduce, the secondary sort technique allows you to sort records by more than one key. By default, Hadoop sorts the data using the key from the mapper output. However, if you need to sort by a secondary key within the reducer, you need to implement additional logic.
A common example is sorting temperature readings by year (primary key) and then by temperature (secondary key). In this guide, we'll implement a secondary sort using Hadoop MapReduce in Java, where we'll sort temperature readings first by year and then by temperature.
Overview of the Solution
To achieve secondary sorting, we'll:
- Create a Composite Key: A custom key that includes both the primary and secondary keys.
- Implement Custom Partitioner: Ensures that all keys with the same primary key are sent to the same reducer.
- Implement Custom Sorting and Grouping Comparators: Defines how keys are compared during sorting and grouping.
- Write Mapper and Reducer Classes: Processes the input and output data.
- Configure and Run the Job: Set up the job configuration to use our custom classes and specify the number of reducers.
Prerequisites
- Hadoop installed and configured.
- Java Development Kit (JDK) installed.
- Basic understanding of Hadoop MapReduce programming.
Step 1: Define the Composite Key Class
Create a class that implements
WritableComparableto serve as our composite key.CompositeKey.java
import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import org.apache.hadoop.io.WritableComparable; public class CompositeKey implements WritableComparable<CompositeKey> { private int year; private int temperature; public CompositeKey() { } public CompositeKey(int year, int temperature) { this.year = year; this.temperature = temperature; } public void set(int year, int temperature) { this.year = year; this.temperature = temperature; } public int getYear() { return year; } public int getTemperature() { return temperature; } @Override public void write(DataOutput out) throws IOException { out.writeInt(year); out.writeInt(temperature); } @Override public void readFields(DataInput in) throws IOException { year = in.readInt(); temperature = in.readInt(); } @Override public int compareTo(CompositeKey o) { int cmp = Integer.compare(this.year, o.year); if (cmp != 0) { return cmp; } return Integer.compare(this.temperature, o.temperature); } @Override public int hashCode() { return year * 163 + temperature; } @Override public boolean equals(Object o) { if (o instanceof CompositeKey) { CompositeKey other = (CompositeKey) o; return year == other.year && temperature == other.temperature; } return false; } }
Step 2: Implement Custom Partitioner
Ensure that all records with the same year go to the same reducer.
CompositeKeyPartitioner.java
import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Partitioner; public class CompositeKeyPartitioner extends Partitioner<CompositeKey, Text> { @Override public int getPartition(CompositeKey key, Text value, int numPartitions) { return (Integer.hashCode(key.getYear()) & Integer.MAX_VALUE) % numPartitions; } }
Step 3: Implement Custom Sorting and Grouping Comparators
Sorting Comparator
Defines the sort order during the shuffle and sort phase.
CompositeKeyComparator.java
import org.apache.hadoop.io.WritableComparable; import org.apache.hadoop.io.WritableComparator; public class CompositeKeyComparator extends WritableComparator { protected CompositeKeyComparator() { super(CompositeKey.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { CompositeKey k1 = (CompositeKey) w1; CompositeKey k2 = (CompositeKey) w2; return k1.compareTo(k2); } }Grouping Comparator
Groups keys for the reducer.
CompositeKeyGroupingComparator.java
import org.apache.hadoop.io.WritableComparable; import org.apache.hadoop.io.WritableComparator; public class CompositeKeyGroupingComparator extends WritableComparator { protected CompositeKeyGroupingComparator() { super(CompositeKey.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { CompositeKey k1 = (CompositeKey) w1; CompositeKey k2 = (CompositeKey) w2; return Integer.compare(k1.getYear(), k2.getYear()); } }
Step 4: Write the Mapper and Reducer Classes
Mapper Class
Processes each line of input and outputs the composite key and temperature.
SecondarySortMapper.java
import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class SecondarySortMapper extends Mapper<LongWritable, Text, CompositeKey, Text> { private CompositeKey compositeKey = new CompositeKey(); private Text tempValue = new Text(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { // Input format: year temperature String[] tokens = value.toString().split("\\s+"); if (tokens.length == 2) { int year = Integer.parseInt(tokens[0]); int temperature = Integer.parseInt(tokens[1]); compositeKey.set(year, temperature); tempValue.set(tokens[1]); context.write(compositeKey, tempValue); } } }Reducer Class
Collects all temperatures for a year and outputs them in sorted order.
SecondarySortReducer.java
import java.io.IOException; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class SecondarySortReducer extends Reducer<CompositeKey, Text, Text, Text> { private Text yearKey = new Text(); private Text tempList = new Text(); @Override protected void reduce(CompositeKey key, Iterable<Text> values, Context context) throws IOException, InterruptedException { StringBuilder sb = new StringBuilder(); for (Text temp : values) { sb.append(temp.toString()).append(", "); } // Remove trailing comma and space if (sb.length() > 0) { sb.setLength(sb.length() - 2); } yearKey.set(Integer.toString(key.getYear())); tempList.set(sb.toString()); context.write(yearKey, tempList); } }
Step 5: Write the Driver Class
Configure the job to use our custom classes and set the number of reducers to 3.
SecondarySortDriver.java
import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class SecondarySortDriver { public static void main(String[] args) throws Exception { if (args.length != 2) { System.err.println("Usage: SecondarySortDriver <input path> <output path>"); System.exit(-1); } Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Secondary Sort Example"); job.setJarByClass(SecondarySortDriver.class); job.setMapperClass(SecondarySortMapper.class); job.setReducerClass(SecondarySortReducer.class); job.setMapOutputKeyClass(CompositeKey.class); job.setMapOutputValueClass(Text.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); // Set custom partitioner and comparators job.setPartitionerClass(CompositeKeyPartitioner.class); job.setSortComparatorClass(CompositeKeyComparator.class); job.setGroupingComparatorClass(CompositeKeyGroupingComparator.class); // Set number of reducers to 3 job.setNumReduceTasks(3); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); boolean success = job.waitForCompletion(true); System.exit(success ? 0 : 1); } }
Step 6: Prepare the Input Data
Create an input file
input.txtwith the following sample data:2020 30 2020 25 2021 32 2020 28 2021 31 2022 29 2021 30 2022 27Upload the input file to HDFS:
hdfs dfs -mkdir -p /user/yourusername/input hdfs dfs -put input.txt /user/yourusername/input/Replace
yourusernamewith your Hadoop username.
Step 7: Compile and Package the Code
Compile the Java Files
Create a directory structure and compile the code:
mkdir -p secondarysort/classes cd secondarysortCopy all the
.javafiles into thesecondarysortdirectory.Set the Hadoop classpath and compile:
export HADOOP_CLASSPATH=$(hadoop classpath) javac -classpath $HADOOP_CLASSPATH -d classes *.javaPackage into a JAR File
Create a JAR file from the compiled classes:
jar -cvf SecondarySort.jar -C classes/ .
Step 8: Run the MapReduce Job with 3 Reducers
Execute the job using the Hadoop
jarcommand:hadoop jar SecondarySort.jar SecondarySortDriver /user/yourusername/input /user/yourusername/outputAlternatively, you can specify the number of reducers from the command line:
hadoop jar SecondarySort.jar SecondarySortDriver -D mapreduce.job.reduces=3 /user/yourusername/input /user/yourusername/output
Step 9: View the Output
After the job completes, retrieve the output from HDFS:
hdfs dfs -cat /user/yourusername/output/part-*Expected Output:
2020 25, 28, 30 2021 30, 31, 32 2022 27, 29This output shows that temperatures within each year are sorted in ascending order.
Complete Commands to Execute
Here's a summary of all the commands:
# Prepare HDFS input directory and upload data hdfs dfs -mkdir -p /user/yourusername/input hdfs dfs -put input.txt /user/yourusername/input/ # Compile Java code export HADOOP_CLASSPATH=$(hadoop classpath) javac -classpath $HADOOP_CLASSPATH -d classes *.java # Create JAR file jar -cvf SecondarySort.jar -C classes/ . # Run MapReduce job with 3 reducers hadoop jar SecondarySort.jar SecondarySortDriver -D mapreduce.job.reduces=3 /user/yourusername/input /user/yourusername/output # View the output hdfs dfs -cat /user/yourusername/output/part-*
Conclusion
By customizing the key, partitioner, and comparators, we've successfully implemented a secondary sort in Hadoop MapReduce. This technique is essential when you need sorted data beyond the primary key in your reduce phase.
Key Takeaways:
- Composite Key: Combine multiple fields to form a composite key for sorting.
- Custom Partitioner: Controls which reducer processes a given key.
- Custom Comparators: Define sorting and grouping behavior during the shuffle and sort phase.
- Multiple Reducers: Distribute workload and improve performance.
Note: Ensure that your Hadoop environment is properly configured and that you have the necessary permissions to execute the commands and access HDFS directories.