Reducing Cassandra heap usage

Reducing Cassandra heap usage

Using Cassandra 1.1.5, have been battling slow write performance, JVM GC
lockups, ... in our logs, we see this rather frequently:
WARN [ScheduledTasks:1] 2013-08-28 09:28:51,983 GCInspector.java (line
145) Heap is 0.8589157615524839 full. You may need to reduce memtable
and/or cache sizes. Cassandra will now flush up to the two largest
memtables to free up memory. Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
The largest memtable in our system (as observed through JConsole) runs up
to about 20,000,000 data size (which I assume is ~20MB, if those are
bytes).
If it matters, this column family has almost 1B rows in it.
flush_largest_memtables_at is set to .75, but it seems we hit that almost
continuously. The pattern for this table is heavy writes, and very little
reads. (essentially a clustered log)
Row cache is disabled, key cache is set to 40MB. We have 8GB of heap
associated to the JVM (24GB of physical).
Heap usage is between 6.5 and 7.5GB, mostly.
Advice on what to look at to reduce heap usage here? Surely it's not a
factor of how much data we have in the cluster, is it? (We have gobs of
disk available across this cluster)