← PracticeAdvancedMemory Model

The Quote With Yesterday's Rate

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Our pricing service holds a rate table in memory and republishes it every thirty seconds from the upstream feed. About once every few hours a quote goes out with the new currency list and the previous set of rates — a EUR line priced at yesterday's USD number. It has cost us real money twice. We added a reader-side assertion that the two arrays are the same length and it fires roughly once in forty million quotes. When we added logging around the publisher to catch it, it stopped happening for two days, then came back.
1struct RateTable {
2  std::vector<Currency> currencies;
3  std::vector<Rate>     rates;      // parallel to currencies
4  uint32_t              version;
5};
6 
7RateTable table;    // plain object, no lock, no atomic
8 
9void publish(const Feed& f) {          // publisher thread, every 30s
10  table.currencies = f.currencies();   // store 1
11  table.rates      = f.rates();        // store 2
12  table.version    = f.version();      // store 3
13}
14 
15Quote quote(const Key& k) {            // 64 reader threads
16  auto idx = indexOf(table.currencies, k);
17  return { table.currencies[idx], table.rates[idx], table.version };
18}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?