Secondary Sort: Introduction in map reduce in hadoop . give full code to implement…
- User
Secondary Sort: Introduction in map reduce in hadoop . give full code to implement secondary sort in haddop in java. it should take input as file and give output in file. take temparature example . also giver command to excute that
- Assistant
Secondary Sort in Hadoop MapReduce
Introduction
In Hadoop MapReduce, the framework automatically sorts the outputs of the map tasks by key before they are input to the reducer. However, sometimes you need to sort the values associated with a key before they reach the reducer. This is where Secondary Sort comes into play.
Secondary Sort allows you to control the ordering of the values for each key, enabling more complex data processing patterns directly within the MapReduce framework without the need for additional processing steps.
Temperature Example
Consider a dataset containing temperature readings with the following format:
<Year> <Month> <Day> <Temperature>Our goal is to:
- Group the temperature readings by year.
- Sort the temperatures within each year in descending order.
This is a typical scenario where secondary sort is useful.
Implementing Secondary Sort in Hadoop (Java)
Below is the full Java code to implement secondary sort in Hadoop. The code includes:
- Custom writable classes.
- Custom partitioner and comparators.
- Mapper and reducer classes.
- Instructions to compile and execute the code.
1. CompositeKeyWritable Class
We create a custom key class that implements
WritableComparable.// CompositeKeyWritable.java import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import org.apache.hadoop.io.WritableComparable; public class CompositeKeyWritable implements WritableComparable<CompositeKeyWritable> { private int year; private int temperature; public CompositeKeyWritable() { } public void set(int year, int temperature) { this.year = year; this.temperature = temperature; } public int getYear() { return year; } public int getTemperature() { return temperature; } @Override public void write(DataOutput out) throws IOException { out.writeInt(year); out.writeInt(temperature); } @Override public void readFields(DataInput in) throws IOException { year = in.readInt(); temperature = in.readInt(); } @Override public int compareTo(CompositeKeyWritable o) { int result = Integer.compare(this.year, o.year); if (result == 0) { // Descending order for temperature result = -Integer.compare(this.temperature, o.temperature); } return result; } @Override public int hashCode() { return year * 163 + temperature; } @Override public boolean equals(Object o) { if (o instanceof CompositeKeyWritable) { CompositeKeyWritable other = (CompositeKeyWritable) o; return year == other.year && temperature == other.temperature; } return false; } }2. Mapper Class
// TemperatureMapper.java import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class TemperatureMapper extends Mapper<LongWritable, Text, CompositeKeyWritable, IntWritable> { private CompositeKeyWritable compositeKey = new CompositeKeyWritable(); private IntWritable temperatureValue = new IntWritable(); @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { // Expected input: year month day temperature String[] tokens = value.toString().split("\\s+"); if (tokens.length == 4) { int year = Integer.parseInt(tokens[0]); int temperature = Integer.parseInt(tokens[3]); compositeKey.set(year, temperature); temperatureValue.set(temperature); context.write(compositeKey, temperatureValue); } } }3. Reducer Class
// TemperatureReducer.java import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class TemperatureReducer extends Reducer<CompositeKeyWritable, IntWritable, Text, IntWritable> { private Text resultKey = new Text(); @Override protected void reduce(CompositeKeyWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { for (IntWritable val : values) { resultKey.set(Integer.toString(key.getYear())); context.write(resultKey, val); } } }4. Partitioner and Comparators
NaturalKeyPartitioner
Partitions records such that all records with the same year go to the same reducer.
// NaturalKeyPartitioner.java import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.mapreduce.Partitioner; public class NaturalKeyPartitioner extends Partitioner<CompositeKeyWritable, IntWritable> { @Override public int getPartition(CompositeKeyWritable key, IntWritable value, int numPartitions) { return Math.abs(Integer.hashCode(key.getYear())) % numPartitions; } }CompositeKeyComparator
Sorts keys first by year (ascending) and then by temperature (descending).
// CompositeKeyComparator.java import org.apache.hadoop.io.WritableComparable; import org.apache.hadoop.io.WritableComparator; public class CompositeKeyComparator extends WritableComparator { protected CompositeKeyComparator() { super(CompositeKeyWritable.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { CompositeKeyWritable key1 = (CompositeKeyWritable) w1; CompositeKeyWritable key2 = (CompositeKeyWritable) w2; return key1.compareTo(key2); } }NaturalKeyGroupingComparator
Groups keys by year for the reducer.
// NaturalKeyGroupingComparator.java import org.apache.hadoop.io.WritableComparable; import org.apache.hadoop.io.WritableComparator; public class NaturalKeyGroupingComparator extends WritableComparator { protected NaturalKeyGroupingComparator() { super(CompositeKeyWritable.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { CompositeKeyWritable key1 = (CompositeKeyWritable) w1; CompositeKeyWritable key2 = (CompositeKeyWritable) w2; return Integer.compare(key1.getYear(), key2.getYear()); } }5. Driver Code
// SecondarySortDriver.java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class SecondarySortDriver { public static void main(String[] args) throws Exception { if (args.length != 2) { System.err.println("Usage: SecondarySortDriver <input path> <output path>"); System.exit(-1); } Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Secondary Sort Example"); job.setJarByClass(SecondarySortDriver.class); job.setMapperClass(TemperatureMapper.class); job.setReducerClass(TemperatureReducer.class); job.setPartitionerClass(NaturalKeyPartitioner.class); job.setSortComparatorClass(CompositeKeyComparator.class); job.setGroupingComparatorClass(NaturalKeyGroupingComparator.class); job.setMapOutputKeyClass(CompositeKeyWritable.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }6. Compiling and Running the Code
Prerequisites
- Hadoop is installed and configured.
- The
HADOOP_HOMEandJAVA_HOMEenvironment variables are set. - The input data file is prepared.
Sample Input Data (
input.txt)2020 01 01 35 2020 01 02 32 2020 01 03 30 2021 01 01 28 2021 01 02 35 2021 01 03 31Steps
a. Create Project Structure
mkdir secondarysort cd secondarysortb. Save the Java Classes
-
Save each of the Java classes in separate
.javafiles within thesecondarysortdirectory:CompositeKeyWritable.javaTemperatureMapper.javaTemperatureReducer.javaNaturalKeyPartitioner.javaCompositeKeyComparator.javaNaturalKeyGroupingComparator.javaSecondarySortDriver.java
c. Compile the Code
javac -classpath $(hadoop classpath) -d . CompositeKeyWritable.java javac -classpath $(hadoop classpath) -d . TemperatureMapper.java javac -classpath $(hadoop classpath) -d . TemperatureReducer.java javac -classpath $(hadoop classpath) -d . NaturalKeyPartitioner.java javac -classpath $(hadoop classpath) -d . CompositeKeyComparator.java javac -classpath $(hadoop classpath) -d . NaturalKeyGroupingComparator.java javac -classpath $(hadoop classpath):. -d . SecondarySortDriver.javad. Create a JAR File
jar -cvf SecondarySort.jar *e. Prepare the Input Data in HDFS
Copy the
input.txtfile to HDFS:hdfs dfs -mkdir -p /user/hadoop/input hdfs dfs -copyFromLocal input.txt /user/hadoop/input/f. Run the MapReduce Job
hadoop jar SecondarySort.jar SecondarySortDriver /user/hadoop/input /user/hadoop/outputg. Check the Output
hdfs dfs -cat /user/hadoop/output/part-r-00000Expected Output:
2020 35 2020 32 2020 30 2021 35 2021 31 2021 28Explanation
-
Mapper Phase:
- Reads the input and emits a composite key consisting of
yearandtemperature. - The temperature is included in the key to influence the sorting order.
- Reads the input and emits a composite key consisting of
-
Partitioning Phase:
NaturalKeyPartitionerensures that all data for the sameyeargoes to the same reducer.
-
Sorting Phase:
CompositeKeyComparatorsorts the keys first byyear(ascending) and then bytemperature(descending).
-
Grouping Phase:
NaturalKeyGroupingComparatorgroups keys byyearso that the reducer receives all the temperatures for a year in order.
-
Reducer Phase:
- Since the values are already sorted, the reducer simply outputs them.
Conclusion
By implementing custom partitioners and comparators, we control the shuffle and sort phase of MapReduce to perform secondary sorting. This allows us to sort values associated with a key before they reach the reducer, expanding the capabilities of data processing tasks in Hadoop.
Note: Ensure that all paths and environment variables are correctly set according to your Hadoop installation.