목록hadoop (4)

hadoop fs -df의 disk usage와 df의 disk usage가 다를 때

기본 replication factor가 3이므로 3배수의 차이가 발생하므로 hadoop fs -df는 여유가 있어도 df로 local disk usage를 보면 그렇지 않은 경우가 있다. replication factor 보기; hdfs fsck 명령을 사용 # sudo -u hdfs hdfs fsck /... .................................................................................................... ............Status: HEALTHY Total size: 1151862045171 B Total dirs: 148 Total files: 16312 Total blocks (validated): 1742..
Programming/Hadoop 2013. 9. 8. 14:00
failed to restart hbase because of time synchronization

Missing time synchronization, scm console cannot start properly. Because of time sync, I installed ntpdate like below; 2011/10/07 - [Programming] - CentOS: time syncronization in local network / setting timezone However time sync is broken after 2 months. So, to prohibit this problem, I wrote 'crontab' on every server. 0 0 * * * /sbin/ntpdate [ntp server] > /dev/null 2>&1
Programming/Hadoop 2011. 12. 9. 07:42
Installation CDH3 on Cent OS

Cent OS during installation select only 'server' package manual setting for IP address manual setting for DNS after installation # yum -y update # /etc/init.d/ssh start set service which need to start automatically http://frucyan.tistory.com/entry/%EC%84%9C%EB%B9%84%EC%8A%A4-%EC%9E%90%EB%8F%99-%EC%8B%9C%EC%9E%91 http://apollo89.com/blog/329 # ntsysv CDH Prerequisite add user for servers http://o..
Programming 2011. 8. 25. 04:47
Cascading on Ubuntu 10.04

Prerequisite hadoop cascading library & wordcount example from http://www.cascading.org/downloads.html $ wget http://files.cascading.org/cascading/1.2/cascading-1.2.4-hadoop-0.19.2%2B.tgz $ wget http://files.cascading.org/samples/wordcount-20101201.tgz $ tar xfz cascading-1.2.4-hadoop-0.19.2+.tgz $ tar xfz wordcount-20101201.tgz $ cd wordcount/ /wordcount$ ant -Dhadoop.home=/usr/lib/hadoop -Dcas..
Programming 2011. 8. 13. 05:19
Prev 1 Next