{"published":"2018-09-23 01:09:27","description":"\u6e96\u5099 EMR\u30af\u30e9\u30b9\u30bf\u30fc\u3092\u4f5c\u6210\u3059\u308b\u3002 EMR \u306e\u30bb\u30ad\u30e5\u30ea\u30c6\u30a3\u30b0\u30eb\u30fc\u30d7\u3067 ssh \u3067\u306e\u30a2\u30af\u30bb\u30b9\u3092\u8a31\u53ef\u3059\u308b\u3002 \u30de\u30b9\u30bf\u30fc\u30ce\u30fc\u30c9\u306b ssh \u3067\u30ed\u30b0\u30a4\u30f3\u3059\u308b\u3002 $ ssh -i ~/us-east-1.pem hadoop@ec2-**-***-**-**.compute-1.amazonaws.com HDFS \u306b\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u3092\u4f5c\u6210\u3057\u3066 S3 \u304b\u3089\u30c7\u30fc\u30bf\u3092\u30b3\u30d4\u30fc\u3059\u308b\u3002 $ hadoop fs -mkdir /amazon-reviews-pds-az/ $ s3-dist-cp --src s3://amazon-reviews-pds/ --dest /amazon-reviews-pds-az/ \u30b3\u2026","provider_url":"https://hatena.blog","title":"PySpark \u304b\u3089 Parquet \u30d5\u30a1\u30a4\u30eb on HFDS \u306b\u30af\u30a8\u30ea\u3092\u5b9f\u884c\u3057\u3066\u307f\u308b","height":"190","blog_url":"https://yohei-a.hatenablog.jp/","url":"https://yohei-a.hatenablog.jp/entry/20180923/1537718967","author_name":"yohei-a","categories":["AWS"],"html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fyohei-a.hatenablog.jp%2Fentry%2F20180923%2F1537718967\" title=\"PySpark \u304b\u3089 Parquet \u30d5\u30a1\u30a4\u30eb on HFDS \u306b\u30af\u30a8\u30ea\u3092\u5b9f\u884c\u3057\u3066\u307f\u308b - ablog\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","provider_name":"Hatena Blog","type":"rich","version":"1.0","blog_title":"ablog","width":"100%","author_url":"https://blog.hatena.ne.jp/yohei-a/","image_url":"https://cdn-ak.f.st-hatena.com/images/fotolife/y/yohei-a/20180924/20180924023501.png"}