read json using pig + jython udf

2013/09/13 - [Programming/Hadoop] - read json from pig

http://pig.apache.org/docs/r0.8.1/udf.html#Load+Functions

http://gethue.tumblr.com/post/60376973455/hadoop-tutorials-ii-1-prepare-the-data-for-analysis

http://opensource.xhaus.com/projects/jyson/wiki/JysonFaq

https://github.com/romainr/yelp-data-analysis

http://stackoverflow.com/questions/16705259/parsing-text-file-of-one-line-json-objects-using-rjsonio

http://ragrawal.wordpress.com/2013/02/24/on-writing-python-udf-for-pig-a-perspective/

http://hadooptips.wordpress.com/2013/08/13/nosuchmethoderror-while-using-joda-time-2-2-jar-in-pig/

http://blog.cloudera.com/blog/2009/06/analyzing-apache-logs-with-pig/

download and unzip jyson-1.0.2.zip from http://downloads.xhaus.com/jyson/1.0.2/

$ hadoop fs -cat yelp_academic_dataset_business.json

{"business_id": "rncjoVoEFUJGCUoC1JgnUA", "full_address": "8466 W Peoria Ave\nSte 6\nPeoria, AZ 85345", "open": true, "categories": ["Accountants", "Professional Services", "Tax Services", "Financial Services"], "city": "Peoria", "review_count": 3, "name": "Peoria Income Tax Service", "neighborhoods": [], "longitude": -112.241596, "state": "AZ", "stars": 5.0, "latitude": 33.581867000000003, "type": "business"}
{"business_id": "0FNFSzCFP_rGUoJx8W7tJg", "full_address": "2149 W Wood Dr\nPhoenix, AZ 85029", "open": true, "categories": ["Sporting Goods", "Bikes", "Shopping"], "city": "Phoenix", "review_count": 5, "name": "Bike Doctor", "neighborhoods": [], "longitude": -112.10593299999999, "state": "AZ", "stars": 5.0, "latitude": 33.604053999999998, "type": "business"}

$ cat converter.py

from com.xhaus.jyson import JysonCodec as json

@outputSchema("business:chararray")
def tsvify(line):
        business_json = json.loads(line)
        business = map(unicode, business_json.values())
        return '\t'.join(business).replace('\n', ' ').encode('utf-8')

$ cat converter.pig

-- REGISTER directory; LOCAL

REGISTER '/some/home/directory/jyson-1.0.2/lib/jyson-1.0.2.jar';
REGISTER '/some/home/directory/converter.py' USING jython AS converter;

-- PIG SCRIPT directory; HDFS
reviews = LOAD '/user/hanadmin/yelp_academic_dataset_business.json' AS (line:CHARARRAY);

tsv = FOREACH reviews GENERATE converter.tsvify(line);

STORE tsv INTO '/user/hanadmin/yelp_academic_dataset_business.tsv';

$ ls jyson-1.0.2/lib/jyson-1.0.2.jar
jyson-1.0.2/lib/jyson-1.0.2.jar

$ pig -f converter.pig

...

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
2.0.0-cdh4.3.0  0.11.0-cdh4.3.0 hanadmin        2013-09-14 14:33:14     2013-09-14 14:33:42     UNKNOWN

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime      MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime   MedianReducetime    Alias   Feature Outputs
job_201309061431_0112   1       0       9       9       9       9       0       0       0       0       reviews,tsv     MAP_ONLY        /user/hanadmin/yelp_academic_dataset_business.tsv,

Input(s):
Successfully read 2 records (1187 bytes) from: "/user/hanadmin/yelp_academic_dataset_business.json"

Output(s):
Successfully stored 2 records (395 bytes) in: "/user/hanadmin/yelp_academic_dataset_business.tsv"

Counters:
Total records written : 2
Total bytes written : 395
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201309061431_0112


2013-09-14 14:33:42,047 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

$ hadoop fs -cat yelp_academic_dataset_business.tsv/part-m-00000

5.0     []      AZ      33.581867       True    8466 W Peoria Ave Ste 6 Peoria, AZ 85345        -112.241596     3       rncjoVoEFUJGCUoC1JgnUA      Peoria  [u'Accountants', u'Professional Services', u'Tax Services', u'Financial Services']      business        Peoria Income Tax Service
5.0     []      AZ      33.604054       True    2149 W Wood Dr Phoenix, AZ 85029        -112.105933     5       0FNFSzCFP_rGUoJx8W7tJg  Phoenix     [u'Sporting Goods', u'Bikes', u'Shopping']      business        Bike Doctor