← PracticeExpertState

Two Singletons, and One With Nothing In It

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Two rare startup problems that we assumed were unrelated until this week. First: some pods show 80 database connections when the pool is configured for 40, and the DBA sees two distinct application_name sessions from one pod. Second: about one request in ten million, very early in a pod's life, segfaults dereferencing a member of our config singleton that the constructor definitely sets. Both got roughly eight times more common after we migrated to Graviton instances.
1Config* instance = nullptr;      // plain pointer: not atomic, no barrier
2std::mutex init_mtx;
3 
4Config* get() {
5  if (instance == nullptr) {                       // 1. unsynchronized read
6    std::lock_guard<std::mutex> g(init_mtx);
7    if (instance == nullptr) {                     // 2. checked again under the lock
8      instance = new Config();                     // 3. allocate, construct, publish
9    }
10  }
11  return instance;                                 // 4. may be a pointer to a half-built object
12}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?