å®æsqoopçå®è£
åï¼å¯ä»¥è¿æ ·æµè¯æ¯å¦å¯ä»¥è¿æ¥å°mysql(注æï¼mysqlçjarå
è¦æ¾å° SQOOP_HOME/lib ä¸)ï¼
sqoop list-databases --connect jdbc:mysql://192.168.1.109:3306/ --username root --password 19891231
ç»æå¦ä¸
å³è¯´æsqoopå·²ç»å¯ä»¥æ£å¸¸ä½¿ç¨äºã
ä¸é¢ï¼è¦å°mysqlä¸çæ°æ®å¯¼å
¥å°hadoopä¸ã
æåå¤çæ¯ä¸ä¸ª300ä¸æ¡æ°æ®ç身份è¯æ°æ®è¡¨ï¼
å
å¯å¨hiveï¼ä½¿ç¨å½ä»¤è¡ï¼hive å³å¯å¯å¨ï¼
ç¶å使ç¨sqoop导å
¥æ°æ®å°hiveï¼
sqoop import --connect jdbc:mysql://192.168.1.109:3306/hadoop --username root --password 19891231 --table test_sfz --hive-import
sqoop ä¼å¯å¨jobæ¥å®æ导å
¥å·¥ä½ã
å®æ导å
¥ç¨äº2å20ç§ï¼è¿æ¯ä¸éçã
å¨hiveä¸å¯ä»¥çå°åå导å
¥çæ°æ®è¡¨ï¼
æ们æ¥ä¸å¥sqlæµè¯ä¸ä¸æ°æ®ï¼
select * from test_sfz where id < 10;
å¯ä»¥çå°ï¼hiveå®æè¿ä¸ªä»»å¡ç¨äºå°è¿25ç§ï¼ç¡®å®æ¯æºæ
¢çï¼å¨mysqlä¸å ä¹æ¯ä¸è´¹æ¶é´ï¼ï¼ä½æ¯è¦èèå°hiveæ¯å建äºjobå¨hadoopä¸è·ï¼æ¶é´å½ç¶å¤ã
æ¥ä¸æ¥ï¼æ们ä¼å¯¹è¿äºæ°æ®è¿è¡å¤ææ¥è¯¢çæµè¯ï¼
ææºåçé
ç½®å¦ä¸ï¼
hadoop æ¯è¿è¡å¨èææºä¸ç伪åå¸å¼ï¼èææºOSæ¯ubuntu12.04 64ä½ï¼é
ç½®å¦ä¸ï¼
TEST 1 计ç®å¹³åå¹´é¾
æµè¯æ°æ®ï¼300.8 W
1. 计ç®å¹¿ä¸çå¹³åå¹´é¾
mysqlï¼select (sum(year(NOW()) - SUBSTRING(borth,1,4))/count(*)) as ageAvge from test_sfz where address like '广ä¸%';
ç¨æ¶ï¼ 0.877s
hiveï¼select (sum(year('2014-10-01') - SUBSTRING(borth,1,4))/count(*)) as ageAvge from test_sfz where address like '广ä¸%';
ç¨æ¶ï¼25.012s
2. 对æ¯ä¸ªåå¸ççå¹³åå¹´é¾è¿è¡ä»é«å°ä½çæåº
mysqlï¼select
address, (sum(year(NOW()) - SUBSTRING(borth,1,4))/count(*)) as ageAvge
from test_sfz GROUP BY address order by ageAvge desc;
ç¨æ¶ï¼2.949s
hiveï¼select
address, (sum(year('2014-10-01') - SUBSTRING(borth,1,4))/count(*)) as
ageAvge from test_sfz GROUP BY address order by ageAvge desc;
ç¨æ¶ï¼51.29s
å¯ä»¥çå°ï¼å¨èæ¶ä¸é¢ï¼hiveçå¢é¿é度è¾mysqlæ
¢ã
TEST 2
æµè¯æ°æ®ï¼1200W
mysql å¼æï¼ MyISAMï¼ä¸ºäºå å¿«æ¥è¯¢é度ï¼
导å
¥å°hiveï¼
1. 计ç®å¹¿ä¸çå¹³åå¹´é¾
mysqlï¼select (sum(year(NOW()) - SUBSTRING(borth,1,4))/count(*)) as ageAvge from test_sfz2 where address like '广ä¸%';
ç¨æ¶ï¼ 5.642s
hiveï¼select (sum(year('2014-10-01') - SUBSTRING(borth,1,4))/count(*)) as ageAvge from test_sfz2 where address like '广ä¸%';
ç¨æ¶ï¼168.259s
2. 对æ¯ä¸ªåå¸ççå¹³åå¹´é¾è¿è¡ä»é«å°ä½çæåº
mysqlï¼select
address, (sum(year(NOW()) - SUBSTRING(borth,1,4))/count(*)) as ageAvge
from test_sfz2 GROUP BY address order by ageAvge desc;
ç¨æ¶ï¼11.964s
hiveï¼select
address, (sum(year('2014-10-01') - SUBSTRING(borth,1,4))/count(*)) as
ageAvge from test_sfz2 GROUP BY address order by ageAvge desc;
ç¨æ¶ï¼311.714s
温馨提示:答案为网友推荐,仅供参考