Sensu 0.12 調査メモ No Items found.
盛大に失敗した履歴です。
sensu-serverとclientの環境を整えたはずなのに、
sensu-dashboardからclientが表示されない問題を抱えています。
sensu-server.logを眺めているとsensu-serverがrabbitmqからデータを取得していないように見えます。
さて以下より記録です。
sensu-server用ノード
- OS: CentOS6.5
- 役割:sensu-server/sensu-api/sensu-dashboard/rabbitmq/redis
- IP: 10.0.0.150
- sensuのバージョン:sensu-0.12.3-1.x86_64
sensu-client用ノード
- OS: CentOS6.5
- 役割: sensu-client
- IP: 10.0.0.163
- sensuのバージョン:sensu-0.12.3-1.x86_64
APIコール結果
infoは取得できるがclientsはぬるぽ
$ curl http://localhost:4567/info {"sensu":{"version":"0.12.3"},"rabbitmq":{"keepalives":{"messages":0,"consumers":1},"results":{"messages":0,"consumers":1},"connected":true},"redis":{"connected":true}} # curl http://localhost:4567/clients []←これが原因。なぜ空なのか
DashBoardから確認すると, itemが0に...
なぜsensu-clientがsensu-serverに登録されていないように見えるのかを調査してみました。
Sensuのアーキテクチャ
sens-clientはrabbitmqと会話します。
- sensu-clientがrabbitmqへアクセスできない?
- エンドポイント名称は正しいか
- rabbitmqポート(デフォルト5672)は開いてるか
rabbitmqのWebUI
rabbitmqはCLIやWebUIがプラグインで提供されています。
rabbitmq_managementというプラグインを有効にしてキューの状態を確認してみます。
- rabbitmq_managementの有効化
# rabbitmq-plugins list [e] amqp_client 3.1.5 [ ] cowboy 0.5.0-rmq3.1.5-git4b93c2d [ ] eldap 3.1.5-gite309de4 [e] mochiweb 2.7.0-rmq3.1.5-git680dba8 [ ] rabbitmq_amqp1_0 3.1.5 [ ] rabbitmq_auth_backend_ldap 3.1.5 [ ] rabbitmq_auth_mechanism_ssl 3.1.5 [ ] rabbitmq_consistent_hash_exchange 3.1.5 [ ] rabbitmq_federation 3.1.5 [ ] rabbitmq_federation_management 3.1.5 [ ] rabbitmq_jsonrpc 3.1.5 [ ] rabbitmq_jsonrpc_channel 3.1.5 [ ] rabbitmq_jsonrpc_channel_examples 3.1.5 [E] rabbitmq_management 3.1.5 [e] rabbitmq_management_agent 3.1.5 [E] rabbitmq_management_visualiser 3.1.5 [ ] rabbitmq_mqtt 3.1.5 [ ] rabbitmq_shovel 3.1.5 [ ] rabbitmq_shovel_management 3.1.5 [ ] rabbitmq_stomp 3.1.5 [ ] rabbitmq_tracing 3.1.5 [e] rabbitmq_web_dispatch 3.1.5 [ ] rabbitmq_web_stomp 3.1.5 [ ] rabbitmq_web_stomp_examples 3.1.5 [ ] rfc4627_jsonrpc 3.1.5-git5e67120 [ ] sockjs 0.3.4-rmq3.1.5-git3132eb9 [e] webmachine 1.10.3-rmq3.1.5-gite9359c7 # rabbitmq-plugins enable rabbitmq_mamagement
- rabbitmq_managementへアクセス
guestでログインします(本番運用するときは消しましょう)
rabbitmqの状態を確認できます。
rabbitmqに接続できているノード一覧を確認します。
どうやらrabbitmqにsensu-clientのノードは疎通できているみたいです。
sensu-dashboardが怪しいのかな?
そうだsensu-dashboardを捨てて他のコンソールを使ってみようヽ(´ー`)ノ
sensu-adminを導入します。
sensu-adminのインストール
sensu-adminのcookbooksは公開されていますがdebianプラットフォーム向けのため手動でインストールします。
- rbenvとruby-buildのインストール
Amazon Linuxなら標準でRuby 2.0がインストールされているけど今回はCentOSで全部入りsensu-serverを稼働させてしまったので('A`)
rbenv+ruby-buildを導入します。
$ git clone https://github.com/sstephenson/rbenv.git ~/.rbenv $ git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build $ rbenv install -l |grep 2.1 $ rbenv install 2.1.1 $ gem install bundler --no-ri --no-rdoc Successfully installed bundler-1.6.2 1 gem installed
- sensu-adminの導入
sensu-adminを導入し、簡易Webサーバモードで起動してみます。
$ sudo yum install -y sqlite3-devel $ git clone https://github.com/sensu/sensu-admin.git Initialized empty Git repository in /opt/sensu-admin/.git/ remote: Reusing existing pack: 2754, done. remote: Total 2754 (delta 0), reused 0 (delta 0) Receiving objects: 100% (2754/2754), 876.71 KiB | 314 KiB/s, done. Resolving deltas: 100% (1143/1143), done. $ bundle exec rake db:migrate $ bundle exec rake db:seed $ bundle exec rails server $ bundle exec rails server => Booting Thin => Rails 3.2.14 application starting in development on http://0.0.0.0:3000 => Call with -d to detach => Ctrl-C to shutdown server >> Thin web server (v1.5.1 codename Straight Razor) >> Maximum connections set to 1024 >> Listening on 0.0.0.0:3000, CTRL+C to stop
ブラウザからhttp://<sensu-admin>:3000へアクセスすると、ログイン画面が表示されています。
だめだったorz.......
以下コンフィグとログ
共通
- /etc/sensu/config.json
{ "rabbitmq": { "host": "10.0.0.150", "port": 5672, "vhost": "/sensu", "user": "sensu", "password": "oresensu" }, "redis": { "host": "10.0.0.150", "port": 6379 }, "api": { "host": "10.0.0.150", "bind": "0.0.0.0", "port": 4567 }, "dashboard": { "bind": "0.0.0.0", "port": 8080, "user": "admin", "password": "ikemen" } }
- /etc/sensu/sensu-server.log
とり忘れてた
client側(10.0.0.163)
- /etc/sensu/conf.d/client.json
{ "client": { "name": "orenoec2", "address": "10.0.0.163", "subscriptions": [ "all" ] } }
- /var/log/sensu/sensu-client.log
起動時のログ
{"timestamp":"2014-04-25T22:29:13.529970+0900","level":"debug","message":"loading config file","config_file":"/etc/sensu/config.json"} {"timestamp":"2014-04-25T22:29:13.530580+0900","level":"debug","message":"loadin g config file","config_file":"/etc/sensu/conf.d/client.json"} {"timestamp":"2014-04-25T22:29:13.531098+0900","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"orenoec2","address":"10.0.0.163","subscriptions":["all"]} ]}} {"timestamp":"2014-04-25T22:29:13.531264+0900","level":"debug","message":"loadin g config file","config_file":"/etc/sensu/conf.d/checks/cpu.json"} {"timestamp":"2014-04-25T22:29:13.531616+0900","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/checks/cpu.json","changes ":{"checks":{"cpu":[null,{"command":"check-cpu.rb -w 80 -c 90","standalone":true ,"handlers":["mailer-ses"],"interval":30,"occurrences":3}]}}} {"timestamp":"2014-04-25T22:29:13.531885+0900","level":"debug","message":"valida ting settings"} {"timestamp":"2014-04-25T22:29:13.532255+0900","level":"debug","message":"settin gs are valid"} {"timestamp":"2014-04-25T22:29:13.545915+0900","level":"info","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns ch eck output"} {"timestamp":"2014-04-25T22:29:13.550142+0900","level":"info","message":"loaded extension","type":"handler","name":"debug","description":"outputs json event dat a"} {"timestamp":"2014-04-25T22:29:13.587469+0900","level":"debug","message":"connec ting to rabbitmq","settings":{"host":"10.0.0.150","port":5672,"vhost":"/sensu"," user":"sensu","password":"oresensu"}} {"timestamp":"2014-04-25T22:29:13.612744+0900","level":"debug","message":"schedu ling keepalives"} {"timestamp":"2014-04-25T22:29:13.616990+0900","level":"debug","message":"publis hing keepalive","payload":{"name":"orenoec2","address":"10.0.0.163","subscriptio ns":["all"],"timestamp":1398432553}} {"timestamp":"2014-04-25T22:29:13.617499+0900","level":"debug","message":"subscr ibing to client subscriptions"} {"timestamp":"2014-04-25T22:29:13.617727+0900","level":"debug","message":"schedu ling standalone checks"}