java – Spark是否在其集群中维护哈希函数？

hashCode的general contract说

This integer need not remain consistent from one execution of an application to another execution of the same application.

因此对于类似Spark的东西,每个执行器都有单独的JVM,它是否可以确保散列代码在整个群集中保持一致？

根据我的经验,我使用具有确定性哈希的东西,所以它不是一个问题.

解决方法:

In my experience I use things with deterministic hashes so it hasn’t been a problem.

这确实是要走的路,Spark无法克服具有非确定性哈希码的对象的使用.

Java Enums的使用是一个特别臭名昭着的例子,说明如何出错,请参阅：http://dev.bizo.com/2014/02/beware-enums-in-spark.html.引用该帖子：

… the hashCode method on Java’s enum type is based on the memory address of the object. So while yes, we’re guaranteed that the same enum value have a stable hashCode inside a particular JVM (since the enum will be a static object) – we don’t have this guarantee when you try to compare hashCodes of Java enums with identical values living in different JVMs