ApacheCon North America 2019
Apache Hivemall is a scalable machine learning library for Apache Hive, Spark, and Pig. Hivemall allows us to apply a wealth of machine learning techniques to massive data stored in distributed storage by just writing a series of SQL-like queries. It provides classification, regression, recommendation, anomaly detection, and topic modeling functionalities in a scalable manner, along with a variety of auxiliary functions for data preprocessing and feature engineering.
This talk demonstrates the Hivemall library with a special emphasis on its new features merged after the first Apache Incubator release. Hivemall v0.5.2-incubating, the latest version as of April 2019, has introduced a state-of-the-art generalized factor model named Field-Aware Factorization Machines and many useful UDFs (e.g., data sketching) originated from the Brickhouse Hive UDF package.
We also show the roadmap of this incubating project. Open issues and pull requests include Apache Spark 2.4 support, implementation of new algorithms such as word2vec and multi-nominal logistic regression, as well as integration with widely-used tools like XGBoost and LightGBM.
Author: Takuya Kitazawa
Takuya Kitazawa is a freelance software developer, previously working at a Big Tech and Silicon Valley-based start-up company where he wore multiple hats as a full-stack software developer, machine learning engineer, data scientist, and product manager. At the intersection of technological and social aspects of data-driven applications, he is passionate about promoting the ethical use of information technologies through his mentoring, business consultation, and public engagement activities. See CV for more information.