mysql – 在oracle中插入pandas数据帧非常慢

我正在研究pandas数据帧.我的应用程序的目标是对csv文件执行某些分析,一旦完成,将此csv文件插入到oracle数据库中.

要插入到oracle数据库中,我使用了pandas库的to_sql命令.但要插入300,000行,我的代码大约需要2小时10分钟.但是,当我使用MySQL 数据库进行相同的分析和相同大小的输入数据时,只用了90秒.

我在lubuntu VM中进行所有操作.作为参考,您可以在下面找到相关代码.

data_frame.to_sql(name='RSA_DATA', con=get_engine(), if_exists='append',
                           index=False, chunksize=config.CHUNK_SIZE)
input_data_list = get_rsa_object_list(data_frame)

我对两种情况都使用了相同的CHUNK_SIZE(设置为500).

有人可以建议这可能是什么原因,我该如何解决这个问题？我无法将我的数据库系统从oracle更改为任何其他数据库.我使用MysqL只是为了确保问题不在于我的代码.

提前致谢.

以下是Oracle数据库的CREATE TABLE命令：

  CREATE TABLE "CRISDEV"."RSA_DATA" 
   (    "GENE_SYMBOL" VARCHAR2(50 BYTE), 
    "GENEID" NUMBER, 
    "CRISPR" VARCHAR2(75 BYTE), 
    "RSAscore" NUMBER, 
    "LOGP" NUMBER, 
    "BEST_ACTIVITY" NUMBER, 
    "RSA_HIT" NUMBER, 
    "HIT_PER_GENE" NUMBER, 
    "TOTAL_CRISPRS" NUMBER, 
    "RSA_RANK" NUMBER, 
    "SAMPLE_NAME" VARCHAR2(10 BYTE), 
    "FOLD_CHANGE" NUMBER, 
    "EXPERIMENT_NUM" NUMBER, 
    "CELL_LINE" VARCHAR2(75 BYTE), 
    "CRISPR_LIB" VARCHAR2(75 BYTE), 
    "CRISPR_LOT" VARCHAR2(25 BYTE), 
    "CAS9_vendOR" VARCHAR2(50 BYTE), 
    "CAS9_LOT" VARCHAR2(25 BYTE), 
    "CRISPR_INFECT_DATE" DATE, 
    "CAS9_INFECT_DATE" DATE, 
    "CAS9_MOI" NUMBER, 
    "FTE" VARCHAR2(50 BYTE), 
    "SAMPLE_NAME_LONG" VARCHAR2(150 BYTE), 
    "INDICATION" VARCHAR2(100 BYTE), 
    "LOW_CRISPR_COUNT" NUMBER DEFAULT 0, 
    "MODIFICATION" VARCHAR2(20 BYTE), 
    "TIMEPOINT" VARCHAR2(10 BYTE)
   ) SEGMENT CREATION IMMEDIATE 
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_DATA" ;

  CREATE INDEX "CRISDEV"."C20150312_RSA_CELLLINE" ON "CRISDEV"."RSA_DATA" ("CELL_LINE") 
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COmpuTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_IDX" ;

  CREATE INDEX "CRISDEV"."C20150312_RSA_CRISPRLIB" ON "CRISDEV"."RSA_DATA" ("CRISPR_LIB") 
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COmpuTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_IDX" ;

  CREATE INDEX "CRISDEV"."C20150312_RSA_GENEID" ON "CRISDEV"."RSA_DATA" ("GENEID") 
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COmpuTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_IDX" ;

  CREATE INDEX "CRISDEV"."C20150312_RSA_GEnesYMBOL" ON "CRISDEV"."RSA_DATA" ("GENE_SYMBOL") 
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COmpuTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_IDX" ;

  CREATE INDEX "CRISDEV"."C20150312_RSA_SAMPLELONG" ON "CRISDEV"."RSA_DATA" ("SAMPLE_NAME_LONG") 
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COmpuTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_IDX" ;

  CREATE INDEX "CRISDEV"."C20150312_RSA_SAMPLENAME" ON "CRISDEV"."RSA_DATA" ("SAMPLE_NAME") 
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COmpuTE STATISTICS 
  STORAGE(INITIAL 65536 NEXT 1048576 mineXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "CRISDEV_IDX" ;

MysqL的CREATE TABLE命令如下：

CREATE TABLE `RSA_DATA` (
  `GENE_SYMBOL` varchar(50) DEFAULT NULL,
  `GENEID` int(11) DEFAULT NULL,
  `CRISPR` varchar(75) DEFAULT NULL,
  `RSAscore` float DEFAULT NULL,
  `LOGP` float DEFAULT NULL,
  `BEST_ACTIVITY` float DEFAULT NULL,
  `RSA_HIT` int(11) DEFAULT NULL,
  `HIT_PER_GENE` int(11) DEFAULT NULL,
  `TOTAL_CRISPRS` int(11) DEFAULT NULL,
  `RSA_RANK` int(11) DEFAULT NULL,
  `SAMPLE_NAME` varchar(10) DEFAULT NULL,
  `FOLD_CHANGE` int(11) DEFAULT NULL,
  `EXPERIMENT_NUM` int(11) DEFAULT NULL,
  `CELL_LINE` varchar(75) DEFAULT NULL,
  `CRISPR_LIB` varchar(75) DEFAULT NULL,
  `CRISPR_LOT` varchar(25) DEFAULT NULL,
  `CAS9_vendOR` varchar(50) DEFAULT NULL,
  `CAS9_LOT` varchar(25) DEFAULT NULL,
  `CRISPR_INFECT_DATE` date DEFAULT NULL,
  `CAS9_INFECT_DATE` date DEFAULT NULL,
  `CAS9_MOI` int(11) DEFAULT NULL,
  `FTE` varchar(50) DEFAULT NULL,
  `SAMPLE_NAME_LONG` varchar(150) DEFAULT NULL,
  `INDICATION` varchar(100) DEFAULT NULL,
  `LOW_CRISPR_COUNT` int(11) DEFAULT NULL,
  `MODIFICATION` varchar(20) DEFAULT NULL,
  `TIMEPOINT` varchar(10) DEFAULT NULL
);

解决方法:

对你的问题的一些评论：

>你可能想玩不同的chunksize’s
> Oracle / ExaData端有6个索引,这当然会让它变慢
>在此加载操作期间,检查您的ExaData是否忙碌/过载
>您可以尝试将数据加载到临时表中,然后将其复制到目标表中.如果您没有[数据保护/待机/流/黄金门] – 您可能还想使用直接加载(nologging)：insert / * APPEND * / into …
>我会在此负载期间分析等待事件(在Oracle端),以便查看您最多花费时间的位置

mysql – 在oracle中插入pandas数据帧非常慢

相关推荐