Debug hanging / freezing process

Last updated: Oct 25, 2022

If a ClickHouse node is unresponsive and you don't know what is happening, you can easily check the stacktraces of all the working threads by running:

SQL

SELECT
 arrayStringConcat(arrayMap(x -> demangle(addressToSymbol(x)), trace), '\n') AS trace_functions,
 count()
FROM system.stack_trace
GROUP BY trace_functions
ORDER BY count()
DESC
SETTINGS allow_introspection_functions=1
FORMAT Vertical;

If you can't run any query but you have access to the node running the process, you can execute the following command to send a TSTP signal to every thread of the process:

Terminal

for i in $(ls -1 /proc/$(pidof clickhouse-server)/task/); do kill -TSTP $i; done

Stack traces will be printed in the ClickHouse logs.

Questions?

Was this page useful?

Horizontal scaling (sharding & replication)

Sharding is a horizontal cluster scaling strategy that puts parts of one ClickHouse database on different shards. This can help you to: Improve fault tolerance. Sharding lets you isolate individual host or replica set malfunctions. If you don't use sharding, then when one host or a set of replicas fails, the entire data they contain may become inaccessible. But if 1 shard out of 5 fails, for example, then 80% of the table data is still available. Improve the query performance. Requests…