Python：快速子集化和循环数据帧

我有下面的最小代码太慢了.对于我需要的1000行,大约需要2分钟.我需要它跑得更快.

import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.randint(0,1000,size=(1000, 4)), columns=list('ABCD'))
start_algorithm = time.time()
myunique = df['D'].unique()
for i in myunique:
    itemp = df[df['D'] == i]
    for j in myunique:
        jtemp = df[df['D'] == j]

我知道numpy可以让它运行得更快,但请记住,我想保留原始数据帧(或numpy中的数组)的一部分,以获得列’D’的特定值.如何改善其性能？

解决方法:

避免多次计运算符数据帧df [df [‘D’] == i].原始代码计算此len(myunique)** 2次.相反,您可以为每个i计算一次(即总共len(myunique)次),存储结果,然后将它们配对.例如,

    groups = [grp for di, grp in df.groupby('D')]
    for itemp, jtemp in IT.product(groups, repeat=2):
        pass

import pandas as pd
import itertools as IT
df = pd.DataFrame(np.random.randint(0,1000,size=(1000, 4)), columns=list('ABCD'))

def using_orig():
    myunique = df['D'].unique()
    for i in myunique:
        itemp = df[df['D'] == i]
        for j in myunique:
            jtemp = df[df['D'] == j]

def using_groupby():
    groups = [grp for di, grp in df.groupby('D')]
    for itemp, jtemp in IT.product(groups, repeat=2):
        pass

In [28]: %timeit using_groupby()
10 loops, best of 3: 63.8 ms per loop
In [31]: %timeit using_orig()
1 loop, best of 3: 2min 22s per loop

关于评论：

I can easily replace itemp and jtemp with a=1 or print “Hello” so ignore that

上面的答案解决了如何更有效地计算itemp和jtemp.如果itemp和jtemp不是你真实计算的核心,那么我们需要更好地理解你真正想要计算的内容,以便建议(如果可能的话)更快地计算它的方法.