Hadoop - Revision history

Mozes at 20:50, 1 May 2018

2018-05-01T20:50:47Z

← Older revision		Revision as of 15:50, 1 May 2018
Line 2:		Line 2:
	[http://hadoop.apache.org/ Hadoop] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).		[http://hadoop.apache.org/ Hadoop] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).

	Our previous Hadoop cluster has died, and we are evaluating other options. One such option is [[https://github.com/LLNL/magpie/tree/master/doc magpie]]. We will let the list know if and when we have something functional.		Our previous Hadoop cluster has died, and we are evaluating other options. One such option is [https://github.com/LLNL/magpie/tree/master/doc magpie]. We will let the list know if and when we have something functional.

Mozes at 20:50, 1 May 2018

2018-05-01T20:50:24Z

← Older revision		Revision as of 15:50, 1 May 2018
Line 2:		Line 2:
	[http://hadoop.apache.org/ Hadoop] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).		[http://hadoop.apache.org/ Hadoop] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).

	~~Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera~~ Hadoop cluster ~~to accommodate the increased usage of Hadoop on campus.~~		Our previous Hadoop cluster has died, and we are evaluating other options. One such option is [[https://github.com/LLNL/magpie/tree/master/doc magpie]]. We will let the list know if and when we have something functional.

	~~To use Hadoop:~~
	* Login to Beocat
	* From there login to the Hadoop headnode, ~~named 'gremlin00'. <tt>ssh gremlin00</tt>~~
	* Copy files into or out of the Hadoop filesystem. Use <tt>hdfs dfs put</tt> and ~~<tt>hdfs dfs get</tt> to copy files. Note:~~
	*# the Hadoop filesystem is both smaller than the Beocat filesystem and is '''not backed up.'''
	*# Please copy data back out of Hadoop as soon as you are ~~done using it~~.
	*# Data which remains untouched may be '''deleted with no prior notice.'''
	*# We make no claims of the viability of data within HDFS nor for how long the data will be available.
	*# If you have data from your runs that you need to keep please copy it out of HDFS as soon as possible.
	*# The HDFS volume may disappear at any point in time and when it comes back your data may not.
	* Run your Hadoop job. <tt>yarn jar path/to/file.jar</tt>

	~~=== (Some) Best Practices ===~~
	* Block size is ~~set to 64MB on the HDFS filesystem~~
	** As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [https://~~hadoop.apache~~.~~org~~/~~docs~~/~~r2.6.4~~/~~hadoop-mapreduce-client~~/~~hadoop-mapreduce-client-core~~/~~HadoopArchives~~.~~html HAR files]~~
	* Multiple users running jobs at the ~~same time can be problematic, as they can slow each other down. If you can, try to run jobs~~ when ~~the cluster isn't already running someone else's jobs.~~
	** you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://gremlin00.~~beocat:8088</tt>~~

Mozes at 15:48, 11 April 2017

2017-04-11T15:48:05Z

← Older revision		Revision as of 10:48, 11 April 2017
Line 7:		Line 7:
	* Login to Beocat		* Login to Beocat
	* From there login to the Hadoop headnode, named 'gremlin00'. <tt>ssh gremlin00</tt>		* From there login to the Hadoop headnode, named 'gremlin00'. <tt>ssh gremlin00</tt>
	* Copy files into or out of the Hadoop filesystem. Use <tt>hdfs fs put</tt> and <tt>hdfs fs get</tt> to copy files. Note:		* Copy files into or out of the Hadoop filesystem. Use <tt>hdfs dfs put</tt> and <tt>hdfs dfs get</tt> to copy files. Note:
	*# the Hadoop filesystem is both smaller than the Beocat filesystem and is '''not backed up.'''		*# the Hadoop filesystem is both smaller than the Beocat filesystem and is '''not backed up.'''
	*# Please copy data back out of Hadoop as soon as you are done using it.		*# Please copy data back out of Hadoop as soon as you are done using it.
Line 14:		Line 14:
	*# If you have data from your runs that you need to keep please copy it out of HDFS as soon as possible.		*# If you have data from your runs that you need to keep please copy it out of HDFS as soon as possible.
	*# The HDFS volume may disappear at any point in time and when it comes back your data may not.		*# The HDFS volume may disappear at any point in time and when it comes back your data may not.
	* Run your Hadoop job. <tt>yarn -jar path/to/file.jar</tt>		* Run your Hadoop job. <tt>yarn jar path/to/file.jar</tt>

	=== (Some) Best Practices ===		=== (Some) Best Practices ===

Mozes at 20:36, 24 August 2016

2016-08-24T20:36:22Z

← Older revision		Revision as of 15:36, 24 August 2016
Line 6:		Line 6:
	To use Hadoop:		To use Hadoop:
	* Login to Beocat		* Login to Beocat
	* From there login to the Hadoop headnode, named '~~theia~~'. <tt>ssh ~~theia~~</tt>		* From there login to the Hadoop headnode, named 'gremlin00'. <tt>ssh gremlin00</tt>
	* Copy files into or out of the Hadoop filesystem. Use <tt>~~hadoop~~ fs put</tt> and <tt>~~hadoop~~ fs get</tt> to copy files. Note:		* Copy files into or out of the Hadoop filesystem. Use <tt>hdfs fs put</tt> and <tt>hdfs fs get</tt> to copy files. Note:
	*# the Hadoop filesystem is both smaller than the Beocat filesystem and is '''not backed up.'''		*# the Hadoop filesystem is both smaller than the Beocat filesystem and is '''not backed up.'''
	*# Please copy data back out of Hadoop as soon as you are done using it.		*# Please copy data back out of Hadoop as soon as you are done using it.
Line 14:		Line 14:
	*# If you have data from your runs that you need to keep please copy it out of HDFS as soon as possible.		*# If you have data from your runs that you need to keep please copy it out of HDFS as soon as possible.
	*# The HDFS volume may disappear at any point in time and when it comes back your data may not.		*# The HDFS volume may disappear at any point in time and when it comes back your data may not.
	* Run your Hadoop job. <tt>~~hadoop~~ -jar path/to/file.jar</tt>		* Run your Hadoop job. <tt>yarn -jar path/to/file.jar</tt>

	=== (Some) Best Practices ===		=== (Some) Best Practices ===
	* Block size is set to 64MB on the HDFS filesystem		* Block size is set to 64MB on the HDFS filesystem
	** As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [~~http~~://hadoop.apache.org/docs/r1.2.1/~~hadoop_archives~~.html HAR files]		** As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [https://hadoop.apache.org/docs/r2.6.4/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopArchives.html HAR files]
	* Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs.		* Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs.
	** you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://~~theia~~.beocat:~~50030~~</tt>		** you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://gremlin00.beocat:8088</tt>

Mozes: HDFS WARNINGS Can I state this any other way?

2014-09-03T00:36:36Z

HDFS WARNINGS Can I state this any other way?

← Older revision		Revision as of 19:36, 2 September 2014
Line 7:		Line 7:
	* Login to Beocat		* Login to Beocat
	* From there login to the Hadoop headnode, named 'theia'. <tt>ssh theia</tt>		* From there login to the Hadoop headnode, named 'theia'. <tt>ssh theia</tt>
	* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note ~~that~~ the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. ~~'''~~Data which remains untouched may be deleted with no prior notice.'''		* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note:
			*# the Hadoop filesystem is both smaller than the Beocat filesystem and is '''not backed up.'''
			*# Please copy data back out of Hadoop as soon as you are done using it.
			*# Data which remains untouched may be '''deleted with no prior notice.'''
			*# We make no claims of the viability of data within HDFS nor for how long the data will be available.
			*# If you have data from your runs that you need to keep please copy it out of HDFS as soon as possible.
			*# The HDFS volume may disappear at any point in time and when it comes back your data may not.
	* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>		* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>

Mozes at 04:56, 29 August 2014

2014-08-29T04:56:46Z

← Older revision		Revision as of 23:56, 28 August 2014
Line 12:		Line 12:
	=== (Some) Best Practices ===		=== (Some) Best Practices ===
	* Block size is set to 64MB on the HDFS filesystem		* Block size is set to 64MB on the HDFS filesystem
	* As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [http://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html HAR files]		** As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [http://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html HAR files]
	* Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs.		* Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs.
	* you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://theia.beocat:50030</tt>		** you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://theia.beocat:50030</tt>

Mozes: minor notes on best practices

2014-08-29T04:56:09Z

minor notes on best practices

← Older revision		Revision as of 23:56, 28 August 2014
Line 1:		Line 1:
	== Hadoop ==		== Hadoop ==
	[[http://hadoop.apache.org/ Hadoop]] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).		[http://hadoop.apache.org/ Hadoop] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).

	Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.		Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.
Line 9:		Line 9:
	* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.'''		* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.'''
	* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>		* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>

			=== (Some) Best Practices ===
			* Block size is set to 64MB on the HDFS filesystem
			* As such, please keep the files stored there at least that size. If you need smaller files, we recommend using [http://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html HAR files]
			* Multiple users running jobs at the same time can be problematic, as they can slow each other down. If you can, try to run jobs when the cluster isn't already running someone else's jobs.
			* you can check the status of the hadoop cluster from any beocat host with <tt>elinks http://theia.beocat:50030</tt>

Kylehutson at 15:47, 9 July 2014

2014-07-09T15:47:02Z

← Older revision		Revision as of 10:47, 9 July 2014
Line 1:		Line 1:
	== Hadoop ==		== Hadoop ==
			[[http://hadoop.apache.org/ Hadoop]] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).

	Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.		Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.

Kylehutson: Created page with "== Hadoop == Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to acc..."

2014-07-09T15:44:44Z

Created page with "== Hadoop == Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to acc..."

New page

== Hadoop ==
Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.

To use Hadoop:
* Login to Beocat
* From there login to the Hadoop headnode, named 'theia'. <tt>ssh theia</tt>
* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.'''
* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>