{"description":"PySpark \u3067\u306f\u3001\u3054\u304f\u6700\u8fd1\u307e\u3067 UDAF (User Defined Aggregate Function: \u30e6\u30fc\u30b6\u5b9a\u7fa9\u96c6\u8a08\u95a2\u6570) \u304c\u30b5\u30dd\u30fc\u30c8\u3055\u308c\u3066\u3044\u306a\u304b\u3063\u305f\u3002 Apache Spark 2.3 \u4ee5\u964d\u3067\u306f Pandas UDF \u3092\u4f7f\u3046\u3053\u3068\u3067 UDAF \u306b\u76f8\u5f53\u3059\u308b\u51e6\u7406\u3092\u66f8\u304f\u3053\u3068\u304c\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u3063\u3066\u3044\u308b\u3002 \u4eca\u56de\u306f\u3001\u305d\u308c\u4ee5\u524d\u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u3092\u4f7f\u3063\u3066\u3044\u308b\u3068\u304d\u306b\u3001\u540c\u7b49\u306e\u51e6\u7406\u3092\u66f8\u304f\u305f\u3081\u306e\u56de\u907f\u7b56\u306b\u3064\u3044\u3066\u66f8\u3044\u3066\u307f\u308b\u3002 \u4f7f\u3063\u305f\u74b0\u5883\u306f\u6b21\u306e\u3068\u304a\u308a\u3002 $ cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) $ uname -r 3.10.0-106\u2026","author_name":"momijiame","url":"https://blog.amedama.jp/entry/pyspark-udaf-workaround","blog_title":"CUBE SUGAR CONTAINER","provider_url":"https://hatena.blog","author_url":"https://blog.hatena.ne.jp/momijiame/","version":"1.0","image_url":"https://m.media-amazon.com/images/I/51AW5aaN7PL._SL160_.jpg","published":"2020-05-03 01:01:52","height":"190","width":"100%","categories":["Python","PySpark","Apache Spark","CentOS7"],"provider_name":"Hatena Blog","type":"rich","blog_url":"https://blog.amedama.jp/","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fblog.amedama.jp%2Fentry%2Fpyspark-udaf-workaround\" title=\"Python: PySpark \u3067 UDAF \u304c\u4f5c\u308c\u306a\u3044\u5834\u5408\u306e\u56de\u907f\u7b56\u306b\u3064\u3044\u3066 - CUBE SUGAR CONTAINER\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","title":"Python: PySpark \u3067 UDAF \u304c\u4f5c\u308c\u306a\u3044\u5834\u5408\u306e\u56de\u907f\u7b56\u306b\u3064\u3044\u3066"}