Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Secondary Sort: Introduction in map reduce in hadoop . give full code to implement…

  1. User

    Secondary Sort: Introduction in map reduce in hadoop . give full code to implement secondary sort in haddop in java. it should take input as file and give output in file. take temparature example . also giver command to excute that

  2. Assistant

    Secondary Sort in Hadoop MapReduce

    Introduction

    In Hadoop MapReduce, the framework automatically sorts the outputs of the map tasks by key before they are input to the reducer. However, sometimes you need to sort the values associated with a key before they reach the reducer. This is where Secondary Sort comes into play.

    Secondary Sort allows you to control the ordering of the values for each key, enabling more complex data processing patterns directly within the MapReduce framework without the need for additional processing steps.

    Temperature Example

    Consider a dataset containing temperature readings with the following format:

    <Year> <Month> <Day> <Temperature>
    

    Our goal is to:

    • Group the temperature readings by year.
    • Sort the temperatures within each year in descending order.

    This is a typical scenario where secondary sort is useful.

    Implementing Secondary Sort in Hadoop (Java)

    Below is the full Java code to implement secondary sort in Hadoop. The code includes:

    • Custom writable classes.
    • Custom partitioner and comparators.
    • Mapper and reducer classes.
    • Instructions to compile and execute the code.

    1. CompositeKeyWritable Class

    We create a custom key class that implements WritableComparable.

    // CompositeKeyWritable.java
    
    import java.io.DataInput;
    import java.io.DataOutput;
    import java.io.IOException;
    import org.apache.hadoop.io.WritableComparable;
    
    public class CompositeKeyWritable implements WritableComparable<CompositeKeyWritable> {
        private int year;
        private int temperature;
    
        public CompositeKeyWritable() {
        }
    
        public void set(int year, int temperature) {
            this.year = year;
            this.temperature = temperature;
        }
    
        public int getYear() {
            return year;
        }
    
        public int getTemperature() {
            return temperature;
        }
    
        @Override
        public void write(DataOutput out) throws IOException {
            out.writeInt(year);
            out.writeInt(temperature);
        }
    
        @Override
        public void readFields(DataInput in) throws IOException {
            year = in.readInt();
            temperature = in.readInt();
        }
    
        @Override
        public int compareTo(CompositeKeyWritable o) {
            int result = Integer.compare(this.year, o.year);
            if (result == 0) {
                // Descending order for temperature
                result = -Integer.compare(this.temperature, o.temperature);
            }
            return result;
        }
    
        @Override
        public int hashCode() {
            return year * 163 + temperature;
        }
    
        @Override
        public boolean equals(Object o) {
            if (o instanceof CompositeKeyWritable) {
                CompositeKeyWritable other = (CompositeKeyWritable) o;
                return year == other.year && temperature == other.temperature;
            }
            return false;
        }
    }
    

    2. Mapper Class

    // TemperatureMapper.java
    
    import java.io.IOException;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Mapper;
    
    public class TemperatureMapper extends Mapper<LongWritable, Text, CompositeKeyWritable, IntWritable> {
        private CompositeKeyWritable compositeKey = new CompositeKeyWritable();
        private IntWritable temperatureValue = new IntWritable();
    
        @Override
        protected void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {
            // Expected input: year month day temperature
            String[] tokens = value.toString().split("\\s+");
            if (tokens.length == 4) {
                int year = Integer.parseInt(tokens[0]);
                int temperature = Integer.parseInt(tokens[3]);
                compositeKey.set(year, temperature);
                temperatureValue.set(temperature);
                context.write(compositeKey, temperatureValue);
            }
        }
    }
    

    3. Reducer Class

    // TemperatureReducer.java
    
    import java.io.IOException;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Reducer;
    
    public class TemperatureReducer extends Reducer<CompositeKeyWritable, IntWritable, Text, IntWritable> {
        private Text resultKey = new Text();
    
        @Override
        protected void reduce(CompositeKeyWritable key, Iterable<IntWritable> values, Context context)
                throws IOException, InterruptedException {
            for (IntWritable val : values) {
                resultKey.set(Integer.toString(key.getYear()));
                context.write(resultKey, val);
            }
        }
    }
    

    4. Partitioner and Comparators

    NaturalKeyPartitioner

    Partitions records such that all records with the same year go to the same reducer.

    // NaturalKeyPartitioner.java
    
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.mapreduce.Partitioner;
    
    public class NaturalKeyPartitioner extends Partitioner<CompositeKeyWritable, IntWritable> {
        @Override
        public int getPartition(CompositeKeyWritable key, IntWritable value, int numPartitions) {
            return Math.abs(Integer.hashCode(key.getYear())) % numPartitions;
        }
    }
    

    CompositeKeyComparator

    Sorts keys first by year (ascending) and then by temperature (descending).

    // CompositeKeyComparator.java
    
    import org.apache.hadoop.io.WritableComparable;
    import org.apache.hadoop.io.WritableComparator;
    
    public class CompositeKeyComparator extends WritableComparator {
        protected CompositeKeyComparator() {
            super(CompositeKeyWritable.class, true);
        }
    
        @Override
        public int compare(WritableComparable w1, WritableComparable w2) {
            CompositeKeyWritable key1 = (CompositeKeyWritable) w1;
            CompositeKeyWritable key2 = (CompositeKeyWritable) w2;
            return key1.compareTo(key2);
        }
    }
    

    NaturalKeyGroupingComparator

    Groups keys by year for the reducer.

    // NaturalKeyGroupingComparator.java
    
    import org.apache.hadoop.io.WritableComparable;
    import org.apache.hadoop.io.WritableComparator;
    
    public class NaturalKeyGroupingComparator extends WritableComparator {
        protected NaturalKeyGroupingComparator() {
            super(CompositeKeyWritable.class, true);
        }
    
        @Override
        public int compare(WritableComparable w1, WritableComparable w2) {
            CompositeKeyWritable key1 = (CompositeKeyWritable) w1;
            CompositeKeyWritable key2 = (CompositeKeyWritable) w2;
            return Integer.compare(key1.getYear(), key2.getYear());
        }
    }
    

    5. Driver Code

    // SecondarySortDriver.java
    
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    
    public class SecondarySortDriver {
        public static void main(String[] args) throws Exception {
            if (args.length != 2) {
                System.err.println("Usage: SecondarySortDriver <input path> <output path>");
                System.exit(-1);
            }
    
            Configuration conf = new Configuration();
            Job job = Job.getInstance(conf, "Secondary Sort Example");
            job.setJarByClass(SecondarySortDriver.class);
    
            job.setMapperClass(TemperatureMapper.class);
            job.setReducerClass(TemperatureReducer.class);
    
            job.setPartitionerClass(NaturalKeyPartitioner.class);
            job.setSortComparatorClass(CompositeKeyComparator.class);
            job.setGroupingComparatorClass(NaturalKeyGroupingComparator.class);
    
            job.setMapOutputKeyClass(CompositeKeyWritable.class);
            job.setMapOutputValueClass(IntWritable.class);
    
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
    
            FileInputFormat.addInputPath(job, new Path(args[0]));
            FileOutputFormat.setOutputPath(job, new Path(args[1]));
    
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    }
    

    6. Compiling and Running the Code

    Prerequisites

    • Hadoop is installed and configured.
    • The HADOOP_HOME and JAVA_HOME environment variables are set.
    • The input data file is prepared.

    Sample Input Data (input.txt)

    2020 01 01 35
    2020 01 02 32
    2020 01 03 30
    2021 01 01 28
    2021 01 02 35
    2021 01 03 31
    

    Steps

    a. Create Project Structure
    mkdir secondarysort
    cd secondarysort
    
    b. Save the Java Classes
    • Save each of the Java classes in separate .java files within the secondarysort directory:

      • CompositeKeyWritable.java
      • TemperatureMapper.java
      • TemperatureReducer.java
      • NaturalKeyPartitioner.java
      • CompositeKeyComparator.java
      • NaturalKeyGroupingComparator.java
      • SecondarySortDriver.java
    c. Compile the Code
    javac -classpath $(hadoop classpath) -d . CompositeKeyWritable.java
    javac -classpath $(hadoop classpath) -d . TemperatureMapper.java
    javac -classpath $(hadoop classpath) -d . TemperatureReducer.java
    javac -classpath $(hadoop classpath) -d . NaturalKeyPartitioner.java
    javac -classpath $(hadoop classpath) -d . CompositeKeyComparator.java
    javac -classpath $(hadoop classpath) -d . NaturalKeyGroupingComparator.java
    javac -classpath $(hadoop classpath):. -d . SecondarySortDriver.java
    
    d. Create a JAR File
    jar -cvf SecondarySort.jar *
    
    e. Prepare the Input Data in HDFS

    Copy the input.txt file to HDFS:

    hdfs dfs -mkdir -p /user/hadoop/input
    hdfs dfs -copyFromLocal input.txt /user/hadoop/input/
    
    f. Run the MapReduce Job
    hadoop jar SecondarySort.jar SecondarySortDriver /user/hadoop/input /user/hadoop/output
    
    g. Check the Output
    hdfs dfs -cat /user/hadoop/output/part-r-00000
    

    Expected Output:

    2020    35
    2020    32
    2020    30
    2021    35
    2021    31
    2021    28
    

    Explanation

    • Mapper Phase:

      • Reads the input and emits a composite key consisting of year and temperature.
      • The temperature is included in the key to influence the sorting order.
    • Partitioning Phase:

      • NaturalKeyPartitioner ensures that all data for the same year goes to the same reducer.
    • Sorting Phase:

      • CompositeKeyComparator sorts the keys first by year (ascending) and then by temperature (descending).
    • Grouping Phase:

      • NaturalKeyGroupingComparator groups keys by year so that the reducer receives all the temperatures for a year in order.
    • Reducer Phase:

      • Since the values are already sorted, the reducer simply outputs them.

    Conclusion

    By implementing custom partitioners and comparators, we control the shuffle and sort phase of MapReduce to perform secondary sorting. This allows us to sort values associated with a key before they reach the reducer, expanding the capabilities of data processing tasks in Hadoop.


    Note: Ensure that all paths and environment variables are correctly set according to your Hadoop installation.