The platforms team gets involved sometimes and really digs in-iirc in one bug they mentioned sending a CPU back to the vendor to examine with an electron microscope. It's more rare for them to really take the time to really understand the problem in detail like "core 34 sometimes returns incorrect results on this computation", although there's nothing in particular stopping them from doing so (other than lack of expertise and a long list of other things to do). It's not unusual for application teams to suspect a machine and basically vote it off the island (I don't want my jobs running here anymore, I cast a vote for it to be repaired / Office Spaced). * There's a convenient frontend for looking up a bunch of diagnostic info for the machine, including failures of borg tasks (were other people's tasks crashing at the same time mine did? what was their crash message?), syslog-level stuff, other machine diagnostics like ECC / MCE errors, and repair history (swapped this DIMM, next attempt will swap this CPU).
(Not as easy to cross this layer and get to the relevant D servers actually storing the data but I think it's all checksummed here anyway.) If your team has your own partition, you can see tabletserver/spanserver debug logs yourself also. * Similarly, even as a client of Bigtable or Spanner, you can find out the specific tabletservers/spanservers operating on a portion of your database and what machines they're running on. You can query live state, or you can use Dremel to look up history. In fact, not just your own borg job but anyone's. * You can easily find out what machine a given borg task was running on. Google's internal production stack is much more amenable to that kind of digging than public cloud products: Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here.
0 kommentar(er)
