From Beocat
Kylehutson (talk | contribs) No edit summary |
(minor notes on best practices) |
||
| Line 1: | Line 1: | ||
== Hadoop == | == Hadoop == | ||
[http://hadoop.apache.org/ Hadoop] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB). | |||
Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus. | Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus. | ||
| Line 9: | Line 9: | ||
* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.''' | * Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.''' | ||
* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt> | * Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt> | ||
=== (Some) Best Practices === | |||
* Block size is set to 64MB on the HDFS filesystem | |||
* As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [http://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html HAR files] | |||
* Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs. | |||
* you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://theia.beocat:50030</tt> | |||
Revision as of 22:56, 28 August 2014
Hadoop
Hadoop is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).
Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.
To use Hadoop:
- Login to Beocat
- From there login to the Hadoop headnode, named 'theia'. ssh theia
- Copy files into or out of the Hadoop filesystem. Use hadoop fs put and hadoop fs get to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. Data which remains untouched may be deleted with no prior notice.
- Run your Hadoop job. hadoop -jar path/to/file.jar
(Some) Best Practices
- Block size is set to 64MB on the HDFS filesystem
* As such, please keep the files stored there at least that size. If you need smaller files, we recommend using HAR files
- Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs.
* you can check the status of the hadoop cluster from any beocat host with elinks http://theia.beocat:50030
