Hive cannot use INSERT OVERWRITE DIRECTORY command

Background

In some environment,  we cannot use INSERT OVERWRITE DIRECTORY command to get file in hadoop, so we need find another way to do this.

Two ways

There are two ways we can use to get hive query result as local file.

First way

Use INSERT OVERWRITE TABLE

The old command is 

But now we cannot use this, so we need create an table to receive our query result first

And then we use command hadoop fs -ls “/user/brric.cdr/test” to check if the file is create.

then, we can continue to exec our other script.

In this way, we need have an db like brric_cdr_db, otherwise we cannot use this way.

Second way

We can simply redirect the results of the query to a local file

In this way, we can easily get output as local file, but the defect is it only can get thousands records at one second, so when data is very big, it will take a long time.In production environment, it takes more than 5 hours to finish it.

发表评论

电子邮件地址不会被公开。 必填项已用*标注