Apache Parquet
| Stable release |
2.0[1]
|
|---|---|
| Development status | Active |
| Operating system | Cross-platform |
| Type | Database management system |
| License | Apache License 2.0 |
| Website |
parquet |
Apache Parquet is a free and open source column-oriented data store of the Apache Hadoop ecosystem. It is similar to the other columnar storage file formats available in Hadoop namely RCFile and Optimized RCFile. It is compatible with most of the data processing frameworks in the Hadoop environment. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk.
The open source project to build Apache Parquet began as a joint effort between Twitter[2] and Cloudera.[3] The first version Apache Parquet 1.0 was released in July 2013.
Features
Apache Parquet is implemented using the record shredding and assembly algorithm taking into account the complex data structures that can be used to store the data.[4] Apache Parquet stores data where the values in each column are physically stored in contiguous memory locations. It is similar to the data storage format of the RCFile. Due to the columnar storage, Apache Parquet provides the following benefits:[5]
- Column-wise compression is efficient and saves storage space
- Compression techniques specific to a type can be applied as the column values tend to be of the same type
- Queries that fetch specific column values need not read the entire row data thus improving performance
- Different encoding techniques can be applied to different columns
- Apache Parquet is implemented using the Apache Thrift framework which increases its flexibility to work with a number of programming languages like C++, Java, Python, PHP, etc.[6]
As of August 2015,[7] Parquet supports the big data processing frameworks including Apache Hive, Apache Drill, Cloudera Impala, Apache Crunch, Apache Pig, Cascading and Apache Spark.
Compression and encoding
In Parquet, compression is performed column by column hence enabling different encoding schemes to be used for text and integer data. In addition this strategy also keeps the door open for newer and better encoding schemes to be implemented as they are invented.
Dictionary encoding
Parquet has an automatic dictionary encoding enabled dynamically for data with a small number of unique values ( < 10^5 ) that aids in significant compression and boosts processing speed.[8]
Bit packing
Storage of integers is usually done with a dedicated 32 or 64 bits per integer. For small integers packing multiple integers into the same space makes storage more efficient.[8]
Run-length encoding ( RLE )
To optimize storage of multiple occurrences of the same value, the value is stored only once along with the number of occurrences.[8]
Parquet implements a hybrid of bit packing and RLE where the encoding switches based on the which produces the best compression results. This strategy works well for certain types of integer data and combines well with dictionary encoding.[8]
Comparison
Apache Parquet can be compared with RCFile and Optimized RCFile (ORC) file formats as all the three fall under the category of columnar data storage within the Hadoop ecosystem. They all have better compression and encoding with improved read performance at the cost of slower writes. In addition to these features, Apache Parquet supports limited schema evolution where the schema can be modified according to the changes in the data. It also provides the ability to add new columns at the end of the file structure. As of now, only Apache Hive and Cloudera Impala are able to query such newly added columns and the other frameworks like Apache Pig are working it.
See also
References
- ↑ "Downloads".
- ↑ "Release Date".
- ↑ "Introducing Parquet: Efficient Columnar Storage for Apache Hadoop - Cloudera Engineering Blog". 2013-03-13. Retrieved 2016-09-14.
- ↑ "Apache Parquet Documentation".
- ↑ "Apache Parquet Cloudera".
- ↑ "Apache Thrift".
- ↑ "Supported Frameworks".
- 1 2 3 4 "Announcing Parquet 1.0: Columnar Storage for Hadoop | Twitter Blogs". blog.twitter.com. Retrieved 2016-09-14.