Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

Error: java.io.IOException: cannot find dir = hdfs:/

  1. User

    Error: java.io.IOException: cannot find dir = hdfs:/

  2. Assistant

    That message is Hadoop / HDFS telling you it can’t resolve the target directory because the URI you passed in is incomplete.

    Notice the path it prints:

    hdfs:/
    

    You’re missing at least one slash (and usually a hostname + port).
    Valid HDFS URIs look like one of these:

    1. Absolute path, letting Hadoop pick up the NameNode from fs.defaultFS
      hdfs:///user/yourid/data ← three slashes after the colon

    2. Fully qualified URI with host (and optional port)
      hdfs://namenode.example.com:8020/user/yourid/data

    Why it matters
    Hadoop’s Path class parses the URI; with only one slash it thinks “hdfs:” is the scheme, but the authority (host:port) is empty and the path is /, so it literally tries to look for the directory named / on a filesystem it can’t resolve—hence “cannot find dir”.

    What to check / do

    1. Make sure you typed the path correctly
      • In shell:
      hdfs dfs -ls hdfs:///tmp            # good
      hdfs dfs -ls hdfs:/tmp              # bad (your error)
      
      • In Java / Spark:
      FileSystem fs = FileSystem.get(new URI("hdfs:///user/yourid/data"), conf);
      
    2. Verify core-site.xml
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://namenode.example.com:8020</value>
      </property>
      
      If that’s missing or wrong, Hadoop won’t know which NameNode to contact.
    3. If you really intended the root directory, use one of:
      • hdfs:/// (root dir on default NameNode)
      • hdfs://namenode:8020/ (root dir on explicit NameNode)
    4. Re-run your command / job after fixing the path.

    Still stuck?
    Drop the exact command or code snippet you’re running and the value of fs.defaultFS ( hdfs getconf -confKey fs.defaultFS ) and I can help debug further.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.