2013/09/13 - [Programming/Hadoop] - read json from pig http://pig.apache.org/docs/r0.8.1/udf.html#Load+Functions http://gethue.tumblr.com/post/60376973455/hadoop-tutorials-ii-1-prepare-the-data-for-analysis http://opensource.xhaus.com/projects/jyson/wiki/JysonFaq https://github.com/romainr/yelp-data-analysis http://stackoverflow.com/questions/16705259/parsing-text-file-of-one-line-json-objects-u..

어떤 size 단위로 hdfs에 file write을 하는 경우 FSDataOutputStream 사용(http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample) FileSystem#append를 사용하는 건 아직 불안정하다고 함. Xmx option을 사용해도 조금 더 오래 동작하다가 오류 발생 http://stackoverflow.com/questions/15609909/error-java-heap-space StringBuffer의 append로 String을 모아 일정 size를 넘으면 한 번 file write을 하고, 기존의 object를 재활용하는 방식을 했더니 StringBuffer에서 heap OutOfMemory가 발생sb.length(0)를 ..

$ hadoop versionHadoop 2.0.0-cdh4.3.0Subversion file:///data/1/jenkins/workspace/generic-package-centos64-5-5/topdir/BUILD/hadoop-2.0.0-cdh4.3.0/src/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155aCompiled by jenkins on Mon May 27 19:45:28 PDT 2013From source with checksum a4218d77f9b12df4e3e49ef96f9d357dThis command was run using /usr/lib/hadoop/hadoop-common-2.0...

# sudo -u hdfs hdfs fsck /.../user/oozie/share/lib/sqoop/libthrift-0.9.0.jar: MISSING 1 blocks of total size 347531 B../user/oozie/share/lib/sqoop/metrics-core-2.1.2.jar: CORRUPT blockpool BP-766882569-10.15.86.206-1376438928219 block blk_-3018521587264545106 /user/oozie/share/lib/sqoop/metrics-core-2.1.2.jar: MISSING 1 blocks of total size 82445 B.../user/oozie/share/lib/sqoop/oozie-sharelib-sq..

기본 replication factor가 3이므로 3배수의 차이가 발생하므로 hadoop fs -df는 여유가 있어도 df로 local disk usage를 보면 그렇지 않은 경우가 있다. replication factor 보기; hdfs fsck 명령을 사용 # sudo -u hdfs hdfs fsck /... .................................................................................................... ............Status: HEALTHY Total size: 1151862045171 B Total dirs: 148 Total files: 16312 Total blocks (validated): 1742..

jobread hdfs files line by line(=json string) -> extract some key - value.pig + python error messagewhich: no hbase in (:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin)2013-09-06 10:26:43,316 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.0-cdh4.3.0 (rexported) compiled May 27 2013, 20:40:512013-09-06 10:26:43,316 [main] INFO org.apache.pig.Main - Logging error messages to: /data1/han..

http://freevps.us/thread-4311.htmlCentOS 5 32Bit: Code: rpm -Uvh http://pkgs.repoforge.org/tmux/tmux-1.6-1.el5.rf.i386.rpmCentOS 5 64Bit: Code: rpm -Uvh http://pkgs.repoforge.org/tmux/tmux-1.6-1.el5.rf.x86_64.rpmCentOS 6 32Bit: Code: rpm -Uvh http://pkgs.repoforge.org/tmux/tmux-1.6-1.el6.rf.i686.rpmCentOS 6 64Bit: Code: rpm -Uvh http://pkgs.repoforge.org/tmux/tmux-1.6-1.el6.rf.x86_64.rpm

mysqladmin create [target database name] -h [host] -u [user] -p[passwd]; mysqldump -h [host] -u [user] -p[passwd] [source database name] | mysql -h [host] -u [user] -p[passwd] [target database name]

http://mysql-python.blogspot.kr/https://pypi.python.org/pypi/piphttp://blog.eflow.org/archives/54 prerequisitepython installedpip installedinstall# pip install MySQL-pythonerror# yum install mysql-devel.x86_64 (from here)

Java for OS X 2013-002 (1.6.0_43) http://support.apple.com/kb/DL1572 Java for OS X 2013-002: How to re-enable the Apple-provided Java SE 6 web plug-in and Web Start funcionality http://support.apple.com/kb/HT5559 http://reviews.cnet.com/8301-13727_7-57414869-263/how-to-use-java-applets-in-os-x-after-java-2012-003-update/