c# – Cassandra System.OutOfMemoryException,它是一个Thrift错误吗？

我正在使用Cassandra 0.8.7,Aquiles作为C#客户端和Thrift 0.7,我试图从具有以下定义的SuperColumnFamily中获取大量数据：

create column family SCF with column_type=Super and comparator=TimeUUIDType and subcomparator=AsciiType;

我想将从Cassandra获取的数据插入到DataTable中,这样我就可以过滤行并根据它生成一些报告,但我总是得到一个OutOfMemoryException.

[OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.]
   Thrift.Transport.TFramedTransport.ReadFrame() +191
   Thrift.Transport.TFramedTransport.Read(Byte[] buf,Int32 off,Int32 len) +101
   Thrift.Transport.TTransport.ReadAll(Byte[] buf,Int32 len) +76
   Thrift.Protocol.TBinaryProtocol.ReadAll(Byte[] buf,Int32 len) +66
   Thrift.Protocol.TBinaryProtocol.ReadI32() +47
   Thrift.Protocol.TBinaryProtocol.ReadMessageBegin() +75
   Apache.Cassandra.Client.recv_multiget_slice() in D:\apache-cassandra-0.8.0-beta2\interface\gen-csharp\Apache\Cassandra\Cassandra.cs:304
   Apache.Cassandra.Client.multiget_slice(List`1 keys,ColumnParent column_parent,SlicePredicate predicate,ConsistencyLevel consistency_level) in D:\apache-cassandra-0.8.0-beta2\interface\gen-csharp\Apache\Cassandra\Cassandra.cs:286

我尝试了几种方法来优化我的代码,我的最终版本是分割时间段(如果它们超过了前缀数字,那么键的数量)我用来切割较小范围内的SuperColumn但是没有,最终我总是得到同样的例外.

它可能是Thrift库的错误吗？当我得到异常时,它总是指向Thrift.Transport.TFramedTransport中的代码的以下部分：

private void ReadFrame()
        {
            byte[] i32rd = new byte[header_size];
            transport.ReadAll(i32rd,header_size);
            int size =
                ((i32rd[0] & 0xff) << 24) |
                ((i32rd[1] & 0xff) << 16) |
                ((i32rd[2] & 0xff) <<  8) |
                ((i32rd[3] & 0xff));

            byte[] buff = new byte[size]; //Here the exception is thrown
            transport.ReadAll(buff,size);
            readBuffer = new MemoryStream(buff);
        }

以下是我试图运行的代码：

string columnFamily = "SCF";
    ICluster cluster = AquilesHelper.RetrieveCluster(ConfigurationManager.AppSettings["CLUSTERNAME"].ToString());
    ColumnParent columnParent = new ColumnParent()
        {
            Column_family = columnFamily
        };
    List<byte[]> keys = //Function that return the list of the key i want to query

    SlicePredicate predicate = new SlicePredicate();
    foreach (DateTime[] dates in dateList)
    {
       from = GuidGenerator.GenerateTimeBasedGuid(dates[0]);
       to = GuidGenerator.GenerateTimeBasedGuid(dates[1]);
       predicate = new SlicePredicate()
       {
          Slice_range = new SliceRange()
          {


     Count = int.MaxValue,Reversed = false,Start = Aquiles.Helpers.Encoders.ByteEncoderHelper.UUIDEnconder.ToByteArray(from),Finish = Aquiles.Helpers.Encoders.ByteEncoderHelper.UUIDEnconder.ToByteArray(to)
      },};
   cluster.Execute(new ExecutionBlock(delegate(CassandraClient client)
   {
      int maxKeys = Convert.ToInt32(ConfigurationManager.AppSettings["maxKeys"]);
      CassandraMethods.TableCreator(ref dt,columnParent,predicate,keys,client,maxKeys);
      return null;
   }),ConfigurationManager.AppSettings["KEYSPACE"].ToString());
}

这是应该将cassandra中的数据插入DataTable的函数：

public static DataTable TableCreator(ref DataTable dt,ColumnParent columnParent,List<byte[]> keys,CassandraClient client,int maxKeys)
{
   int keyCount = keys.Count;
   if (keyCount < maxKeys)
      CassandraMethods.CassandratoDataTable(ref dt,client.multiget_slice(keys,ConsistencyLevel.ONE));
   else
   {
      int counter = 0;
      while (counter < keyCount)
      {
         if (counter + maxKeys <= keyCount)
            CassandraMethods.CassandratoDataTable(ref dt,client.multiget_slice(keys.GetRange(counter,maxKeys),ConsistencyLevel.ONE));
         else
            CassandraMethods.CassandratoDataTable(ref dt,keyCount - counter),ConsistencyLevel.ONE));
         counter += maxKeys;
      }
   }
   return dt;
}

我错过了什么吗？我究竟做错了什么？

更新1：我也试过Cassandra 1.0,Aquiles 1.0,版本0.6和0.7的Thrift,但没有,仍然是相同的例外.

更新2：问题解决了,请阅读下面的答案

解决方法

问题解决了：)
我玩了内存使用和垃圾收集器,我解决了问题.

发生的事情是,每当我的应用程序达到1.5 GB的Ram时,由于visual studio将其编译为32位应用程序,因此抛出异常.

编译和运行x64解决了发布,为了确保不使用太多内存现在我在每次Cassandra multiget_slice调用之前添加了以下3行代码.

GC.Collect(GC.MaxGeneration,GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
GC.Collect(GC.MaxGeneration,GCCollectionMode.Forced);

谢谢,N.

c# – Cassandra System.OutOfMemoryException,它是一个Thrift错误吗？

解决方法

相关推荐