{"componentChunkName":"component---node-modules-gatsby-theme-medium-to-own-blog-src-templates-blog-post-js","path":"/riemann-learnings/","result":{"data":{"site":{"siteMetadata":{"siteUrl":"https://javame.netlify.app","githubUrl":"https://github.com/aterreno/blog"}},"mdx":{"fields":{"slug":"/riemann-learnings/"},"excerpt":"We’ve been using and troubleshooting  Riemann  quite extensively in the last few months, here’s a little write up of the main learnings…","timeToRead":3,"frontmatter":{"title":"Riemann Learnings","description":"","categories":[],"date":"January 27, 2014","canonical_link":"https://javame.netlify.app//riemann-learnings-29466513eb5f"},"body":"function _extends() { _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; }; return _extends.apply(this, arguments); }\n\nfunction _objectWithoutProperties(source, excluded) { if (source == null) return {}; var target = _objectWithoutPropertiesLoose(source, excluded); var key, i; if (Object.getOwnPropertySymbols) { var sourceSymbolKeys = Object.getOwnPropertySymbols(source); for (i = 0; i < sourceSymbolKeys.length; i++) { key = sourceSymbolKeys[i]; if (excluded.indexOf(key) >= 0) continue; if (!Object.prototype.propertyIsEnumerable.call(source, key)) continue; target[key] = source[key]; } } return target; }\n\nfunction _objectWithoutPropertiesLoose(source, excluded) { if (source == null) return {}; var target = {}; var sourceKeys = Object.keys(source); var key, i; for (i = 0; i < sourceKeys.length; i++) { key = sourceKeys[i]; if (excluded.indexOf(key) >= 0) continue; target[key] = source[key]; } return target; }\n\n/* @jsxRuntime classic */\n\n/* @jsx mdx */\nvar _frontmatter = {\n  \"title\": \"Riemann Learnings\",\n  \"description\": \"\",\n  \"date\": \"2014-01-27T12:25:01.000Z\",\n  \"categories\": [],\n  \"published\": true,\n  \"canonical_link\": \"https://javame.netlify.app//riemann-learnings-29466513eb5f\",\n  \"redirect_from\": [\"/riemann-learnings-29466513eb5f\"]\n};\nvar layoutProps = {\n  _frontmatter: _frontmatter\n};\nvar MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  var components = _ref.components,\n      props = _objectWithoutProperties(_ref, [\"components\"]);\n\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(\"p\", null, \"We\\u2019ve been using and troubleshooting \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"http://riemann.io/\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"Riemann\"), \" quite extensively in the last few months, here\\u2019s a little write up of the main learnings, with the hope that other users might find it useful and don\\u2019t repeat some of the mistakes we have done.\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"A praise for Riemann\"), mdx(\"br\", {\n    parentName: \"p\"\n  }), \"\\n\", \"To start with, Riemann is a great monitoring tool, its code and the code of the clients I\\u2019ve been using is clean, readable and all our issues have been caused by misconfiguration or misunderstanding the tool. The support of Riemann itself has been great, \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://twitter.com/aphyr\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"aphyr\"), \" helped us in more than one occasion with quick, prompt suggestions.\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Why Riemann\"), mdx(\"br\", {\n    parentName: \"p\"\n  }), \"\\n\", \"Why would you spend time in using, configuring and learning Riemann, which comes with a Clojure DSL for configuration?\"), mdx(\"p\", null, \"It does add value, tools like \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"http://newrelic.com/\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"New Relic\"), \" can be run together with Riemann, but they solve a subset of what can Riemann can provide.\"), mdx(\"p\", null, \"If New Relic is a tool for monitoring and alerting at a high level the performance of an application, Riemann can be configured to monitor business logic in the code and can be finely grained configured to alert. Obviously new relic is a simple tool, drop its jar, configure your java agent and you are done, you have your monitoring/alerting solution up and running.\"), mdx(\"p\", null, \"Riemann providing more needs more, you\\u2019ll need to read its docs, understand its goals, limitations and configure it adequately depending on your needs.\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"A typical setup\"), mdx(\"br\", {\n    parentName: \"p\"\n  }), \"\\n\", \"We have a Riemann server running in production, few production applications instances (about 12 right now) send events which gets enriched, filtered and forwarded to \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"http://graphite.wikidot.com/\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"graphite\"), \", to \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://github.com/guardian/alerta\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"alerta\"), \" and eventually forwarded for email reporting to an smtp server.\"), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Deploy and \\u2018release\\u2019\"), mdx(\"br\", {\n    parentName: \"p\"\n  }), \"\\n\", \"Our process at the beginning was poor, a spike and learn approach, hacking the files on the server, day by day our configuration has grown and now we have a separate github project for the configuration, we copy the files over via Scp and we respect the Riemann file structure that the project has on \", mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://github.com/aphyr/riemann\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"github\"), \", this allow us to change the bash script to start the process, update and track a new version of the jar and keep the configs in the etc folder.\"), mdx(\"p\", null, \"We test new changes locally by starting it with the exact same startup script, often we just copy the files over and reload the configuration by connecting to the Riemann REPL.\"), mdx(\"pre\", null, mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-bash\"\n  }), \"lein repl\\xA0:connect 127.0.0.1:5557  \\n\")), mdx(\"pre\", null, mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-clojure\"\n  }), \"(riemann.bin/reload!)  \\n\")), mdx(\"p\", null, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Riemann \\u2018good practices\\u2019\")), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Split the configuration in multiple files  \", mdx(\"pre\", {\n    parentName: \"li\"\n  }, mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-clojure\"\n  }), \"(include \\\"util.clj\\\")  \\n(include \\\"mailer.clj\\\")  \\n(include \\\"graphite.clj\\\")  \\n(include \\\"alerta.clj\\\")  \\n(include \\\"molads.clj\\\")\\n\"))), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"The configuration grows and we found beneficial to split in different files, each with a clear name and responsibility. We name them as Clojure files making our life easier within emacs.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Carefully define your functions and variables: a team on day changed a variable used by the mailer, it took quite some time to troubleshoot the problem and realize that being Riemann configuration written in Clojure everything can be redefined.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Leverage use of \", mdx(\"a\", _extends({\n    parentName: \"li\"\n  }, {\n    \"href\": \"http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"LBQ\"), \" We are currently using LBQ both on the Riemann to graphite/Alerta side, using the function \")), mdx(\"pre\", null, mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-clojure\"\n  }), \"   (async-queue!)\\n    (let index (default\\xA0:ttl 300 (update-index (index)))  \\n    alert (async-queue!\\xA0:alerta {:queue-size 1000} alerta)  \\n    graph (async-queue!\\xA0:graphite {:queue-size 1000} graph)  \\n\")), mdx(\"p\", null, \"and also on the Clojure side, wrapping the event sending within this function:\"), mdx(\"pre\", null, mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-clojure\"\n  }), \"(ns clj-components.utils.bounded-executor  \\n    \\\"See: [https://github.com/aphyr/riemann-clojure-client/issues/9#issuecomment-32624706](https://github.com/aphyr/riemann-clojure-client/issues/9#issuecomment-32624706) to understand what\\u2019s going on here\\\"  \\n    (:import (java.util.concurrent ThreadPoolExecutor TimeUnit  LinkedBlockingQueue RejectedExecutionHandler)))\\n(def reject-handler  \\n    \\\"Handles a rejection on the bounded executor. i.e. when the LBQ is full.\\\"  \\n    (proxy [RejectedExecutionHandler] []  \\n    (rejectedExecution [runnable executor])))\\n(def bounded-executor  \\n    \\\"Bounded Execution, current settings are calcuated thinking on the current volumes of Riemann In Production\\\"  \\n    (let [cores (.availableProcessors (Runtime/getRuntime))]  \\n    (ThreadPoolExecutor. 1 cores 5 TimeUnit/SECONDS (LinkedBlockingQueue. 250) reject-handler)))\\n(defn run-bounded [f]  \\n    \\\"Exectutes f in a bounded executor\\\"  \\n    (let [executor bounded-executor]  \\n    (.execute executor (Thread. f))))\\n\")), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"For a deep explaination on why using LBQ follow the discussion on this issue on \", mdx(\"a\", _extends({\n    parentName: \"li\"\n  }, {\n    \"href\": \"https://github.com/aphyr/riemann-clojure-client/issues/9#issuecomment-32624706\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"github\")), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"We had problems with TCP without LBQ, we switched to UDP but then noticed that the java client used by the Riemann client to send events swallows exceptions when the connection is closed, until this bug will be fixed, I\\u2019d recommend using TCP with LBQs as there\\u2019s no way to recognise a disconnected UDP client. More on this on the same github issue, just down on the same \", mdx(\"a\", _extends({\n    parentName: \"li\"\n  }, {\n    \"href\": \"https://github.com/aphyr/riemann-clojure-client/issues/9#issuecomment-32971989\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"thread\"), \".\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Don\\u2019t use it as a datastore: a silly mistake: keeping data too long on the index will lead to poor performance, Riemann is designed to process events which have a short life and little state, keeping your events for more than few minutes will lead to obvious performance problems. Make sure you set a reasonable default ttl on the events in the index.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Don\\u2019t query the index too often This is the last finding: we were trying to set a flag for maintenance mode when deploying applications, to stop events propagation, however we were querying the index all the times to check if Riemann was set in maintenance mode for a certain application, this was growing somehow the heap allocation day by day. As it\\u2019s a recent finding/topic you might find some interesting comments in this github \", mdx(\"a\", _extends({\n    parentName: \"li\"\n  }, {\n    \"href\": \"https://github.com/aphyr/riemann/issues/312\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"issue\"), \".\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Leverage the java toolset: we configured Riemann to run with NewRelic, Jmx and Yourkit, without these tools it would have been really hard to find out where the problems where, the java command would be enriched with something like:  \", mdx(\"pre\", {\n    parentName: \"li\"\n  }, mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-bash\"\n  }), \"JMX\\\\_OPTS=\\\" -Djava.rmi.server.hostname=10.250.76.85 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=9100 -Dcom.sun.management.jmxremote\\\"  \\nAGENT\\\\_OPTS=\\\"-javaagent:/opt/molsfw/newrelic/newrelic.jar -Dnewrelic.environment=production\\\"  \\nPROFILER\\\\_OPTS=\\\"-agentpath:/opt/molsfw/yourkit/libyjpagent.so\\\"  \\n\"))), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"In conclusion\"), \" I\\u2019ve been very positevely impressed by the tool and by the support and received, the DSL might be a little hard to remember if you don\\u2019t play with the code too often but the power of the tool is impressive. Reading the Guardian\\u2019s \", mdx(\"a\", _extends({\n    parentName: \"li\"\n  }, {\n    \"href\": \"https://github.com/guardian/riemann-config\",\n    \"target\": \"_blank\",\n    \"rel\": \"nofollow noopener noreferrer\"\n  }), \"Riemann config\"), \" has been a great source of inspiration and I encourage whoever can to share their Riemann configuration.\")));\n}\n;\nMDXContent.isMDXComponent = true;"},"allWebMentionEntry":{"nodes":[]}},"pageContext":{"id":"a58039db-1536-5e9a-ab4c-1c000fa130e9","previous":{"id":"84cea2fd-6244-5084-9168-42231fbff1b9","fields":{"slug":"/bash-learnings/","published":true},"frontmatter":{"redirect_from":["/bash-learnings-7728c7169141"],"redirect_to":null,"title":"Bash Leanings"}},"next":{"id":"cc0ac217-f20e-54c8-ad01-dfa1b990e7b5","fields":{"slug":"/introducing-setlista/","published":true},"frontmatter":{"redirect_from":["/introducing-setlista-eb9bf2a357e4"],"redirect_to":null,"title":"Introducing setlista"}},"permalink":"https://javame.netlify.app/riemann-learnings/","themeOptions":{"plugins":[],"config":{"title":"Antonio Terreno","description":"Antonio Terreno's blog archive","shortBio":"","bio":"Principal Consultant at Equal Experts","author":"Antonio Terreno","githubUrl":"https://github.com/aterreno/blog","siteUrl":"https://javame.netlify.app/","social":{"twitter":"javame","medium":"","facebook":"","github":"aterreno","linkedin":"antonioterreno","instagram":"tritonitri"},"goatCounterCode":null}}}},"staticQueryHashes":["4131332129","645483741"]}