微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

impala jdbc驱动执行impala sql的一个坑不支持多行sql

架构使用spark streaming 消费kafka的数据,并通过impala来插入到kudu中,但是通过对比发现落地到kudu表中的数据比kafka消息数要少,通过后台日志发现,偶发性的出现java.sql.sqlException: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, sql state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Syntax error in line 1  原因是调用过程中使用了数据库连接池,会合并多行sql执行,实现如下:  

import com.alibaba.druid.pool.DruidDataSource;
import com.alibaba.druid.pool.DruidDataSourceFactory;
import com.alibaba.druid.pool.DruidPooledConnection;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import java.sql.sqlException;
import java.util.Properties;


public class ImapalConnPool {

    private static Log logger = LogFactory.getLog(ImapalConnPool.class);

    private static ImapalConnPool imapalConnPool = null;
    private static DruidDataSource druidDataSource = null;

    static {
        Properties properties = new Properties();
        properties.setProperty("driverClassName","com.cloudera.impala.jdbc41.Driver");
        properties.setProperty("url","jdbc:impala://127.0.0.1:21050");  
        properties.setProperty("username","");
        properties.setProperty("password","");
        properties.setProperty("initialSize","50");
        properties.setProperty("maxActive","100");
        properties.setProperty("maxWait","60000");
        properties.setProperty("timeBetweenevictionRunsMillis","60000");
        properties.setProperty("minevictableIdleTimeMillis","300000");
        properties.setProperty("validationQuery","SELECT 1");
        properties.setProperty("testWhileIdle","true");
        properties.setProperty("testOnBorrow","false");
        properties.setProperty("testOnReturn","false");
        properties.setProperty("poolPreparedStatements","false");
        //当该值大于0时,启用pool,poolPreparedStatements为true
        properties.setProperty("maxPoolPreparedStatementPerConnectionSize","-1");

        try {
            druidDataSource = (DruidDataSource) DruidDataSourceFactory.createDataSource(properties); //DruidDataSrouce工厂模式
        } catch (Exception e) {
            logger.error(e);
        }
    }

    public static ImapalConnPool getInstance(){
        if (null == imapalConnPool){
            synchronized(ImapalConnPool.class) {
                if(null == imapalConnPool) {
                    imapalConnPool = new ImapalConnPool();
                }
            }
        }
        return imapalConnPool;
    }

    public DruidPooledConnection getConnection() throws sqlException {
        return druidDataSource.getConnection();
    }

}

 

调整上述参数poolPreparedStatements和maxPoolPreparedStatementPerConnectionSize任然不能解决一个connection合并多行sql的问题   后去掉连接池,改为jdbc直连问题解决。有解决过上面数据库连接池问题的麻烦告知我一下
Connection connectionn = null;
Statement statement = null;
try {
    Class.forName("com.cloudera.impala.jdbc41.Driver");
    connectionn = DriverManager.getConnection("jdbc:impala://127.0.0.1:21050");
    statement = connectionn.createStatement();
    for (String item : execsql) {
        statement.execute(item);
    }
    processResult = true;
}catch (Exception ex){
    logger.error(this,ex);
}finally {
    if(null != statement){
        try {
            statement.close();
        }catch (Exception ex){
            logger.error(this,ex);
        }
    }
    if(null != connectionn) {
        try {
            connectionn.close();
        }catch (Exception ex){
            logger.error(this,ex);
        }
    }
}

 

 

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 [email protected] 举报,一经查实,本站将立刻删除。

相关推荐