{"id":410,"date":"2017-12-05T20:07:38","date_gmt":"2017-12-06T04:07:38","guid":{"rendered":"http:\/\/www.ossintegrators.com\/blog\/?p=410"},"modified":"2017-12-13T13:42:18","modified_gmt":"2017-12-13T21:42:18","slug":"configuring-and-optimizing-the-f5-analytics-apps-kpi-generation-system","status":"publish","type":"post","link":"http:\/\/www.ossintegrators.com\/blog\/configuring-and-optimizing-the-f5-analytics-apps-kpi-generation-system\/","title":{"rendered":"Configuring and Optimizing the F5 Analytics App\u2019s KPI Generation System"},"content":{"rendered":"<div style=\"padding-bottom:20px; padding-top:10px;\" class=\"hupso-share-buttons\"><!-- Hupso Share Buttons - http:\/\/www.hupso.com\/share\/ --><a class=\"hupso_toolbar\" href=\"http:\/\/www.hupso.com\/share\/\"><img decoding=\"async\" src=\"http:\/\/static.hupso.com\/share\/buttons\/share-small.png\" border=\"0\" style=\"padding-top:5px; float:left;\" alt=\"Share\"\/><\/a><script type=\"text\/javascript\">var hupso_services_t=new Array(\"Twitter\",\"Facebook\",\"Google Plus\",\"Linkedin\",\"Digg\",\"Reddit\");var hupso_toolbar_size_t=\"small\";var hupso_counters_lang = \"en_US\";var hupso_url_t=\"\";var hupso_title_t=\"Configuring and Optimizing the F5 Analytics App\u2019s KPI Generation System\";<\/script><script type=\"text\/javascript\" src=\"http:\/\/static.hupso.com\/share\/js\/share_toolbar.js\"><\/script><!-- Hupso Share Buttons --><\/div><p><span style=\"font-size: 12pt;\">As mentioned in my <a href=\"http:\/\/www.ossintegrators.com\/blog\/which-f5-app-should-i-use-with-splunk\/\">previous post<\/a>, one of the key features of the <a href=\"https:\/\/splunkbase.splunk.com\/app\/3161\/\"><span style=\"color: #0563c1; text-decoration: underline;\">F5 Networks &#8211; Analytics (new)<\/span><\/a>\u00a0App is it&#8217;s KPI generation subsystem. Unfortunately, when I developed it I ran out of time to do much documentation on how to properly set it up. This post will clear up that oversight \ud83d\ude09<br \/>\n<\/span><\/p>\n<h1>KPI System Overview<\/h1>\n<p><span style=\"font-size: 12pt;\">The purpose of the KPI generation system is to allow many sub-KPIs to be rolled-up into overall KPIs and then be written to a\u00a0summary\u00a0index for super-fast searching and reporting. Without it, the KPI searches would be <em>extremely<\/em> slow \u2013 for example, the top-level device KPI search is <em>183 lines<\/em> of SPL (after macro expansion)!<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">The KPI generation system consists of the following parts:<br \/>\n<\/span><\/p>\n<ul>\n<li><span style=\"font-size: 12pt;\">A set of ~28 macros named beginning with &#8220;t_&#8221; that contain the default threshold values. For example, &#8220;t_kpi_cpu_violation&#8221; defaults to &#8220;65&#8221;, which means that your CPU health will take a hit if it&#8217;s consistently over 65%.<br \/>\n<\/span><\/li>\n<li><span style=\"font-size: 12pt;\">A set of ~70 macros that build upon each other to calculate the sub-KPI values, culminating in a set of top-level macros that generate overall device and application health.<br \/>\n<\/span><\/li>\n<li><span style=\"font-size: 12pt;\">A python-based modular input to generate the actual KPI data and write it out to a summary index. I could do an entire blog post on how it works and the logic behind making sure it doesn&#8217;t destroy your Search Head!<br \/>\n<\/span><\/li>\n<\/ul>\n<p><span style=\"font-size: 12pt;\">A quick note on the need for the modular input vs regular scheduled searches with Summary Indexing enabled: using regular searches was not possible due to the index-related RBAC built into the App itself. This RBAC capability is <em>crucial<\/em> in that it allows an admin to \u2013 for example \u2013 only allow the Sharepoint Admin to see their\u00a0data and not the data from the IIS application. Using a modular input allowed for dynamically determining which Summary Index to use.<br \/>\n<\/span><\/p>\n<p><!--more--><\/p>\n<h1>Enabling KPI Generation<\/h1>\n<p><span style=\"font-size: 12pt;\">The App installs with both modular inputs disabled. To enable them, navigate to Settings-&gt;Inputs and click on the &#8220;F5 Health KPI Summary Generator&#8221; link:<br \/>\n<\/span><\/p>\n<p><img decoding=\"async\" src=\"http:\/\/www.ossintegrators.com\/blog\/wp-content\/uploads\/2017\/12\/120517_1841_Configuring1.png\" alt=\"\" \/><span style=\"font-size: 12pt;\"><br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">Clicking on &#8220;app_kpi_gen&#8221; (&#8220;host_kpi_gen&#8221; has similar settings) yields the following configuration screen:<br \/>\n<\/span><\/p>\n<p><img decoding=\"async\" src=\"http:\/\/www.ossintegrators.com\/blog\/wp-content\/uploads\/2017\/12\/120517_1841_Configuring2.png\" alt=\"\" \/><span style=\"font-size: 12pt;\"><br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">The setting descriptions are straightforward \u2013 just be sure to not set the Frequency too low or you will start skipping searches!<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">Once you are satisfied with the changes, click the &#8220;Enable&#8221; link here:<br \/>\n<\/span><\/p>\n<p><img decoding=\"async\" src=\"http:\/\/www.ossintegrators.com\/blog\/wp-content\/uploads\/2017\/12\/120517_1841_Configuring3.png\" alt=\"\" \/><span style=\"font-size: 12pt;\"><br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">Note that you cannot change the Index setting from the GUI since it&#8217;s determined &#8220;on-the-fly&#8221;.<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">Once the KPIs are running, you can use the Administration-&gt;Stats on Stats dashboard to monitor KPI generation:<br \/>\n<\/span><\/p>\n<p><img decoding=\"async\" src=\"http:\/\/www.ossintegrators.com\/blog\/wp-content\/uploads\/2017\/12\/120517_1841_Configuring4.png\" alt=\"\" \/><span style=\"font-size: 12pt;\"><br \/>\n<\/span><\/p>\n<p>You will also want to use your Splunk Monitoring Console to monitor the <a href=\"http:\/\/docs.splunk.com\/Documentation\/Splunk\/7.0.1\/DMC\/Scheduleractivity\">Skip Ratio<\/a>\u00a0(see below for some recommended configuration changes).<\/p>\n<p>There are also two dashboards under Administration (Device Health and Application Health) that can help you understand how the KPI numbers are being calculated since most of the KPI&#8217;s are composed from sub-KPI&#8217;s. The screen shot below illustrates this for the Overall Memory Health KPI \u2013 it is calculated by combining a predictive\/outlier KPI (Memory Prediction), overall memory usage (Memory Health), and a KPI that calculates how often a threshold is violated (Memory Violations).<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/www.ossintegrators.com\/blog\/wp-content\/uploads\/2017\/12\/120517_1841_Configuring5.png\" alt=\"\" \/><span style=\"font-size: 12pt;\"><br \/>\n<\/span><\/p>\n<h1>Important &#8211; SHC Support<\/h1>\n<p><span style=\"font-size: 12pt;\">The KPI Generation system does <em>not<\/em> support Search Head Clustering! Do not under <em>any<\/em> circumstances enable the modular inputs on a SHC member since doing so will heavily impact the entire Cluster. The modular input is not SHC-aware (and does not use the Splunk scheduler), so enabling the inputs will cause the KPI generation searches to run on all SHC members (which is not good).<br \/>\n<\/span><\/p>\n<h1>Recommended Splunk Configuration Changes<\/h1>\n<p><span style=\"font-size: 12pt;\">Since enabling KPI generation adds a fair number of scheduled searches and the App also has 52 (!) Accelerated Data Models, there are two changes you will want to make in limits.conf on the Search Head to avoid having a terrible skip ratio (e.g. &gt; 95%).<br \/>\n<\/span><\/p>\n<hr \/>\n<p><span style=\"color: #333333; font-family: Consolas; font-size: 10pt;\">[scheduler]<br \/>\n<\/span><span style=\"color: #333333; font-family: Consolas; font-size: 10pt;\">auto_summary_perc =\u00a0100<\/span><\/p>\n<p><span style=\"color: #333333; font-family: Consolas; font-size: 10pt;\">max_searches_perc =\u00a075<br \/>\n<\/span><\/p>\n<hr \/>\n<p><span style=\"font-size: 12pt;\">These changes tell Splunk that up to 75% of available searches can be scheduled (the default is 50%) and that up to 100% of those scheduled searches can be data model acceleration searches (the default is 50%).<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">Why do you need these settings? Let&#8217;s take the example of a 16 core SH with default settings:<br \/>\n<\/span><\/p>\n<ul>\n<li><span style=\"font-size: 12pt;\">Max allowed concurrent searches (system wide): 22<br \/>\n<\/span><\/li>\n<li><span style=\"font-size: 12pt;\">Max concurrent scheduled searches: 11<br \/>\n<\/span><\/li>\n<li><span style=\"font-size: 12pt;\">Max concurrent datamodel acceleration searches: 5<br \/>\n<\/span><\/li>\n<\/ul>\n<p><span style=\"font-size: 12pt;\">With only 5 search slots but 52 searches needing to be run every 5 minutes, most datamodel acceleration searches will be skipped and the datamodels will fall behind \u2013 especially if there are actual users on the system!<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12pt;\">Note that these settings are also the current recommended best practice settings for Enterprise Security \u2013 another heavy datamodel-dependent App.<br \/>\n<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<div style=\"padding-bottom:20px; padding-top:10px;\" class=\"hupso-share-buttons\"><!-- Hupso Share Buttons - http:\/\/www.hupso.com\/share\/ --><a class=\"hupso_toolbar\" href=\"http:\/\/www.hupso.com\/share\/\"><img src=\"http:\/\/static.hupso.com\/share\/buttons\/share-small.png\" border=\"0\" style=\"padding-top:5px; float:left;\" alt=\"Share\"\/><\/a><script type=\"text\/javascript\">var hupso_services_t=new Array(\"Twitter\",\"Facebook\",\"Google Plus\",\"Linkedin\",\"Digg\",\"Reddit\");var hupso_toolbar_size_t=\"small\";var hupso_counters_lang = \"en_US\";var hupso_url_t=\"\";var hupso_title_t=\"Configuring and Optimizing the F5 Analytics App\u2019s KPI Generation System\";<\/script><script type=\"text\/javascript\" src=\"http:\/\/static.hupso.com\/share\/js\/share_toolbar.js\"><\/script><!-- Hupso Share Buttons --><\/div><p>As mentioned in my previous post, one of the key features of the F5 Networks &#8211; Analytics (new)\u00a0App is it&#8217;s KPI generation subsystem. Unfortunately, when I developed it I ran out of time to do much documentation on how to properly set it up. This post will clear up that oversight \ud83d\ude09 KPI System Overview [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[6],"tags":[57,50,53],"class_list":["post-410","post","type-post","status-publish","format-standard","hentry","category-splunk","tag-f5","tag-shc","tag-splunk"],"_links":{"self":[{"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/posts\/410","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/comments?post=410"}],"version-history":[{"count":8,"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/posts\/410\/revisions"}],"predecessor-version":[{"id":440,"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/posts\/410\/revisions\/440"}],"wp:attachment":[{"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/media?parent=410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/categories?post=410"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.ossintegrators.com\/blog\/wp-json\/wp\/v2\/tags?post=410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}