【入门】what is apache orc？提高hive存储效率？怎么和hive搭配使用？

文章目录一. 什么是orc二. hive集成orc三. hive表属性一. 什么是orc官网https://orc.apache.org/docs/1.ORC files目标为了提高hive的存储效率以及减少文件大小。Back in January 2013, we created ORC files as part of the initiative to massively speed up Apache Hive and improve the storage efficiency of data stored in Apache Hadoop. The focus was on enabling high speed processing and reducing file sizes.2.ORC是一种为Hadoop工作负载设计的自描述type感知式(ing) 的列格式存储文件。ORC is a self-describing type-aware columnar file format designed for Hadoop workloads. It is optimized for large streaming reads, but with integrated support for finding required rows quickly. Storing data in a columnar format lets the reader read, decompress, and process only the values that are required for the current query.3.提高大型流数据的读效率在写入文件时ORC这边会构建一个内部索引。其中当谓词下推使用这些索引来确定需要为特定查询读取文件中的哪些stripes而行索引可以将搜索范围缩小到10,000行的特定集合。Because ORC files are type-aware, the writer chooses the most appropriate encoding for the type and builds an internal index as the file is written.Predicate pushdown uses those indexes to determine which stripes in a file need to be read for a particular query and the row indexes can narrow the search to a particular set of 10,000 rows.4.ORC支持Hive中的完整类型集包括复杂类型:结构、列表、映射和联合。ORC supports the complete set of types in Hive, including the complex types: structs, lists, maps, and unions.5. stripes的概念ORC文件被划分为条默认情况下大约为64MB。文件中的条彼此独立形成分布式工作的自然单元。在每个条带中列之间是分开的因此writer只能读取所需的列。ORC files are divided in to stripes that are roughly 64MB by default. The stripes in a file are independent of each other and form the natural unit of distributed work. Within each stripe, the columns are separated from each other so the reader can read just the columns that are required.二. hive集成orcif the table is ORC. Using show create table, you get this:ORC is well integrated into Hive, so storing your istari table as ORC is done by adding “STORED AS ORC”.CREATETABLEistari(name STRING,color STRING)STOREDASORC;具体的实现类STOREDASINPUTFORMAT ‘org.apache.hadoop.hive.ql.io.orc.OrcInputFormat’ OUTPUTFORMAT ‘org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat’To modify a table so that new partitions of the istari table are stored as ORC files:ALTERTABLEistariSETFILEFORMATORC;三. hive表属性Tables stored as ORC files use table properties to control their behavior. By using table properties, the table owner ensures that all clients store data with the same options.所有的客户端在储存数据时都会有按照相同的属性操作。官网hive-config表中添加参数CREATETABLEistari(name STRING,color STRING)STOREDASORC TBLPROPERTIES(orc.compressNONE);