Stack & HeapIntermediate

Segfault at exactly the same input size, only in worker threads

Symptoms

  • The parser service crashes with SIGSEGV on some documents; the same documents parse fine in the unit tests.
  • The crash is deterministic: any input nested deeper than ~1,300 levels dies, shallower ones never do.
  • Core dumps show a very long backtrace of the same three functions repeating.
  • It only happens in the worker threads, never on the main thread.
$ dmesg -T | grep parser | tail -1                                   # Linux
[Wed 10:41:02] parser[4242]: segfault at 7f3a1c3fef88 ip 000055d1c0a2e1c4 sp 00007f3a1c3fef80 error 6 in parser[55d1c0a00000+80000]

$ gdb -batch -ex bt ./parser core | head -6
#0  parse_value (p=0x7f3a1c4a0010, depth=1361) at json.cc:112
#1  parse_array (p=0x7f3a1c4a0010, depth=1360) at json.cc:171
#2  parse_value (p=0x7f3a1c4a0010, depth=1360) at json.cc:130
#3  parse_array (p=0x7f3a1c4a0010, depth=1359) at json.cc:171
(More stack frames follow...)                  # 2,722 frames in total

$ grep -A1 7f3a1c3ff000 /proc/4242/maps         # from a live reproduction
7f3a1c3ff000-7f3a1c47f000 rw-p 00000000 00:00 0      # 512 KB worker stack
                                                     # fault address 7f3a1c3fef88 is 120 bytes BELOW it: the guard page
$ grep -n stacksize src/worker.cc
41:  pthread_attr_setstacksize(&attr, 512 * 1024);   // "workers don't need much"

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Heap corruption
Input validation
The faulting address versus the memory map
Recursion depth of the parser
Memory leak / RSS
Compiler optimisation level