Cassandra节点出现故障

高拉夫·古普塔（Gaurav Gupta）

昨晚，我的node.js API抛出错误

All host(s) tried for query failed.First host tried, 127.0.0.1:Host considered as DOWN.

正如错误所建议的那样，似乎cassandra节点已关闭，因此我深入研究了cassandra日志，发现它们是cassandra引发的异常，但我不知道何时可以引发此错误。

卡桑德拉原木

INFO  [MemtableFlushWriter:73] 2015-04-02 19:12:58,844 Memtable.java:370 - Completed flushing; nothing needed to be retained.  Commitlog position was ReplayPosition(segmentId=1427954898588, position=3473692)
INFO  [CompactionExecutor:60] 2015-04-02 19:18:28,208 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:61] 2015-04-02 19:28:28,209 CompactionManager.java:521 - No files to compact for user defined compaction
ERROR [CompactionExecutor:62] 2015-04-02 19:38:17,782 CassandraDaemon.java:153 - Exception in thread Thread[CompactionExecutor:62,1,main]
java.lang.NullPointerException: null
    at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:236) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1089) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
INFO  [ScheduledTasks:1] 2015-04-02 19:38:22,991 ColumnFamilyStore.java:856 - Enqueuing flush of sstable_activity: 65753 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:74] 2015-04-02 19:38:22,992 Memtable.java:324 - Writing Memtable-sstable_activity@166239084(6642 serialized bytes, 2952 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:74] 2015-04-02 19:38:23,105 Memtable.java:363 - Completed flushing /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-208-Data.db (3897 bytes) for commitlog position ReplayPosition(segmentId=1427954898588, position=3589687)
INFO  [CompactionExecutor:63] 2015-04-02 19:38:23,106 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-205-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-207-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-208-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-206-Data.db')]
INFO  [CompactionExecutor:63] 2015-04-02 19:38:23,221 CompactionTask.java:252 - Compacted 4 sstables to [/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-209,].  15,482 bytes to 3,897 (~25% of original) in 113ms = 0.032889MB/s.  328 total partitions merged to 82.  Partition merge counts were {4:82, }
INFO  [CompactionExecutor:62] 2015-04-02 19:38:28,210 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:64] 2015-04-02 19:48:28,213 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:65] 2015-04-02 19:58:28,214 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:66] 2015-04-02 20:08:28,215 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:67] 2015-04-02 20:18:28,216 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [BatchlogTasks:1] 2015-04-02 20:19:58,913 ColumnFamilyStore.java:856 - Enqueuing flush of batchlog: 5848 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:75] 2015-04-02 20:19:58,914 Memtable.java:324 - Writing Memtable-batchlog@1542233641(4538 serialized bytes, 10 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:75] 2015-04-02 20:19:58,915 Memtable.java:370 - Completed flushing; nothing needed to be retained.  Commitlog position was ReplayPosition(segmentId=1427954898588, position=3789073)
INFO  [CompactionExecutor:68] 2015-04-02 20:28:28,217 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [ScheduledTasks:1] 2015-04-02 20:38:22,989 ColumnFamilyStore.java:856 - Enqueuing flush of compaction_history: 1214 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:76] 2015-04-02 20:38:22,990 Memtable.java:324 - Writing Memtable-compaction_history@1912511539(240 serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
INFO  [ScheduledTasks:1] 2015-04-02 20:38:22,992 ColumnFamilyStore.java:856 - Enqueuing flush of sstable_activity: 65753 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:77] 2015-04-02 20:38:22,993 Memtable.java:324 - Writing Memtable-sstable_activity@1348219988(6642 serialized bytes, 2952 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:76] 2015-04-02 20:38:23,049 Memtable.java:363 - Completed flushing /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-107-Data.db (250 bytes) for commitlog position ReplayPosition(segmentId=1427954898588, position=3841853)
INFO  [MemtableFlushWriter:77] 2015-04-02 20:38:23,053 Memtable.java:363 - Completed flushing /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-210-Data.db (3916 bytes) for commitlog position ReplayPosition(segmentId=1427954898588, position=3841853)
INFO  [CompactionExecutor:69] 2015-04-02 20:38:28,217 CompactionManager.java:521 - No files to compact for user defined compaction