The Global Chubby Planned Outage
Written by Marc Alvidrez
Chubby [Bur06] is Google’s lock service for loosely coupled distributed systems. In the global case, we distribute Chubby instances such that each replica is in a different geographical region. Over time, we found that the failures of the global instance of Chubby consistently generated service outages, many of which were visible to end users. As it turns out, true global Chubby outages are so infrequent that service owners began to add dependencies to Chubby assuming that it would never go down. Its high reliability provided a false sense of security because the services could not function appropriately when Chubby was unavailable, however rarely that occurred.
The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. In this way, we are able to flush out unreasonable dependencies on Chubby shortly after they are added. Doing so forces service owners to reckon with the reality of distributed systems sooner rather than later.
Written by Marc Alvidrez
Chubby [Bur06] is Google’s lock service for loosely coupled distributed systems. In the global case, we distribute Chubby instances such that each replica is in a different geographical region. Over time, we found that the failures of the global instance of Chubby consistently generated service outages, many of which were visible to end users. As it turns out, true global Chubby outages are so infrequent that service owners began to add dependencies to Chubby assuming that it would never go down. Its high reliability provided a false sense of security because the services could not function appropriately when Chubby was unavailable, however rarely that occurred.
The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. In this way, we are able to flush out unreasonable dependencies on Chubby shortly after they are added. Doing so forces service owners to reckon with the reality of distributed systems sooner rather than later.
Сидишь такой, чай пьешь, конфиг правишь, а тебе 10+ алёртов за полминуты, и все с разных хостов, и все с намеком, что вот-вот забьется ин-мем очередь перед Эластиком и машины лягут с ООМ.
А потом тимлид говорит, что ты не понял алёрт и проверь, прежде чем ответственной тиме Эластика жаловаться. И ведь реально, в логах у тебя проблем нет, только в Графане видно изменение.
Вот только Эластик перешёл в желтое состояние (неок, некрит), и таки тима подтверждает проблему.
Айти это круто!
А потом тимлид говорит, что ты не понял алёрт и проверь, прежде чем ответственной тиме Эластика жаловаться. И ведь реально, в логах у тебя проблем нет, только в Графане видно изменение.
Вот только Эластик перешёл в желтое состояние (неок, некрит), и таки тима подтверждает проблему.
Айти это круто!
У вас есть проблемы с отправкой сообщ в телеге? Красный значок как на скрине выше
Anonymous Poll
7%
Да
87%
Не
1%
У друзей
4%
Уже прошло
Вышел LLVM 9.0 (http://releases.llvm.org/9.0.0/docs/ReleaseNotes.html)
И оказывается существует проект Mull - LLVM-based mutation testing. Получается любой компилятор с LLVM можно облепить мутационными тестами, почти бесплатно.
Репа проекта https://github.com/mull-project/mull
И оказывается существует проект Mull - LLVM-based mutation testing. Получается любой компилятор с LLVM можно облепить мутационными тестами, почти бесплатно.
Репа проекта https://github.com/mull-project/mull
Еще один кеш.......................дграф могут хоть что-то написать кроме ЕЩЕ ОДНОГО КЕША ?
Пойду читать их репо и громко бухтеть на твич-стриме "олег опять бухтит на чужой код"
https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/
https://github.com/dgraph-io/ristretto
Пойду читать их репо и громко бухтеть на твич-стриме "олег опять бухтит на чужой код"
https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/
https://github.com/dgraph-io/ristretto
Indeed, Google is almost only interested in global scale products.
Довольно интересный текст о гугле от одного из инженеров (6 лет отмотал). Из понравившегося:
Hadoop has become a billion dollar industry sector, while in reality, similar systems inside Google
A global engineering organization cannot rely on third party providers. For these technologies are too challenging to build and too valuable to share. Google has built a complete suite of developer tools for engineers to do their best work.
Google pioneered SRE. Unlike the traditional operational personell, SRE is mandated to spend at least 50% of their time on writing software that automates the daily operations.
https://docs.google.com/document/d/10wylnn8qHD4clDyrEIhYyP1FWxzEkf89hfwbmoyS-yI/edit
Довольно интересный текст о гугле от одного из инженеров (6 лет отмотал). Из понравившегося:
Hadoop has become a billion dollar industry sector, while in reality, similar systems inside Google
A global engineering organization cannot rely on third party providers. For these technologies are too challenging to build and too valuable to share. Google has built a complete suite of developer tools for engineers to do their best work.
Google pioneered SRE. Unlike the traditional operational personell, SRE is mandated to spend at least 50% of their time on writing software that automates the daily operations.
https://docs.google.com/document/d/10wylnn8qHD4clDyrEIhYyP1FWxzEkf89hfwbmoyS-yI/edit
Полез в спецификацию LDAP. Узнал расшифровку: Lightweight Directory Access Protocol.
Мне казалось Lightweight это когда меньше 68 страниц в спецификации..
А там внутри еще и свой(?) формат сериализации BER (Basic Encoding Rules), в общем все обмазано весельем.
Кстати, можете что-то хорошее-полезное про лдап рассказать? а может и нехорошее.
https://tools.ietf.org/html/rfc4511
Мне казалось Lightweight это когда меньше 68 страниц в спецификации..
А там внутри еще и свой(?) формат сериализации BER (Basic Encoding Rules), в общем все обмазано весельем.
Кстати, можете что-то хорошее-полезное про лдап рассказать? а может и нехорошее.
https://tools.ietf.org/html/rfc4511
Forwarded from HN Best Comments
Re: Tired of Stack Overflow
Children, SO is a blessing for humanity.
Their creators should be canonized. You don't remember the pre SO times. You couldn't find anything to fix your problems and must read the source code or pay zillions for some sh*tty support. In the pre-internet times, you just would hit a wall and have to quit. I really love Jeff and Joel.
neves, 4 hours ago
Children, SO is a blessing for humanity.
Their creators should be canonized. You don't remember the pre SO times. You couldn't find anything to fix your problems and must read the source code or pay zillions for some sh*tty support. In the pre-internet times, you just would hit a wall and have to quit. I really love Jeff and Joel.
neves, 4 hours ago
oleg_log
Я прям не удивлен результатом. Мяу https://impurepics.com/quiz/
1й вопрос самым сложным был
Довольно интересный тред на ночь:
What’s your container-less deployment process ?
В треде доминирует NixOS :)
https://lobste.rs/s/xov7nz/what_s_your_container_less_deployment
(А еще узнал про https://nixos.org/nixops/ )
What’s your container-less deployment process ?
В треде доминирует NixOS :)
https://lobste.rs/s/xov7nz/what_s_your_container_less_deployment
(А еще узнал про https://nixos.org/nixops/ )
lobste.rs
What’s your container-less deployment process?
69 comments