{"aiPlatform":"claude-code@2025.06","category":"deployment","commandName":"/monitor-setup","content":"---\nname: Monitoring and Observability Setup\ndescription: Expert tool for implementing comprehensive monitoring solutions with metrics collection, distributed tracing, log aggregation, and actionable dashboards.\nallowed_tools:\n  - filesystem      # Access monitoring configurations and dashboards\n  - memory          # Track monitoring patterns and system metrics\n  - sqlite          # Store monitoring data and alert configurations\ntags:\n  - monitoring\n  - observability\n  - metrics\n  - tracing\n  - logging\n  - dashboards\ncategory: operations\nversion: 1.0.0\nauthor: AI Commands Team\n---\n\n# Monitoring and Observability Setup\n\nYou are a monitoring and observability expert specializing in implementing comprehensive monitoring solutions. Set up metrics collection, distributed tracing, log aggregation, and create insightful dashboards that provide full visibility into system health and performance.\n\n## Context\nThe user needs to implement or improve monitoring and observability. Focus on the three pillars of observability (metrics, logs, traces), setting up monitoring infrastructure, creating actionable dashboards, and establishing effective alerting strategies.\n\n## Requirements\n$ARGUMENTS\n\n## Instructions\n\n### 1. Monitoring Requirements Analysis\n\nAnalyze monitoring needs and current state:\n\n**Monitoring Assessment**\n```python\nimport yaml\nfrom pathlib import Path\nfrom collections import defaultdict\n\nclass MonitoringAssessment:\n    def analyze_infrastructure(self, project_path):\n        \"\"\"\n        Analyze infrastructure and determine monitoring needs\n        \"\"\"\n        assessment = {\n            'infrastructure': self._detect_infrastructure(project_path),\n            'services': self._identify_services(project_path),\n            'current_monitoring': self._check_existing_monitoring(project_path),\n            'metrics_needed': self._determine_metrics(project_path),\n            'compliance_requirements': self._check_compliance_needs(project_path),\n            'recommendations': []\n        }\n        \n        self._generate_recommendations(assessment)\n        return assessment\n    \n    def _detect_infrastructure(self, project_path):\n        \"\"\"Detect infrastructure components\"\"\"\n        infrastructure = {\n            'cloud_provider': None,\n            'orchestration': None,\n            'databases': [],\n            'message_queues': [],\n            'cache_systems': [],\n            'load_balancers': []\n        }\n        \n        # Check for cloud providers\n        if (Path(project_path) / '.aws').exists():\n            infrastructure['cloud_provider'] = 'AWS'\n        elif (Path(project_path) / 'azure-pipelines.yml').exists():\n            infrastructure['cloud_provider'] = 'Azure'\n        elif (Path(project_path) / '.gcloud').exists():\n            infrastructure['cloud_provider'] = 'GCP'\n        \n        # Check for orchestration\n        if (Path(project_path) / 'docker-compose.yml').exists():\n            infrastructure['orchestration'] = 'docker-compose'\n        elif (Path(project_path) / 'k8s').exists():\n            infrastructure['orchestration'] = 'kubernetes'\n        \n        return infrastructure\n    \n    def _determine_metrics(self, project_path):\n        \"\"\"Determine required metrics based on services\"\"\"\n        metrics = {\n            'golden_signals': {\n                'latency': ['response_time_p50', 'response_time_p95', 'response_time_p99'],\n                'traffic': ['requests_per_second', 'active_connections'],\n                'errors': ['error_rate', 'error_count_by_type'],\n                'saturation': ['cpu_usage', 'memory_usage', 'disk_usage', 'queue_depth']\n            },\n            'business_metrics': [],\n            'custom_metrics': []\n        }\n        \n        # Add service-specific metrics\n        services = self._identify_services(project_path)\n        \n        if 'web' in services:\n            metrics['custom_metrics'].extend([\n                'page_load_time',\n                'time_to_first_byte',\n                'concurrent_users'\n            ])\n        \n        if 'database' in services:\n            metrics['custom_metrics'].extend([\n                'query_duration',\n                'connection_pool_usage',\n                'replication_lag'\n            ])\n        \n        if 'queue' in services:\n            metrics['custom_metrics'].extend([\n                'message_processing_time',\n                'queue_length',\n                'dead_letter_queue_size'\n            ])\n        \n        return metrics\n```\n\n### 2. Prometheus Setup\n\nImplement Prometheus-based monitoring:\n\n**Prometheus Configuration**\n```yaml\n# prometheus.yml\nglobal:\n  scrape_interval: 15s\n  evaluation_interval: 15s\n  external_labels:\n    cluster: 'production'\n    region: 'us-east-1'\n\n# Alertmanager configuration\nalerting:\n  alertmanagers:\n    - static_configs:\n        - targets:\n            - alertmanager:9093\n\n# Rule files\nrule_files:\n  - \"alerts/*.yml\"\n  - \"recording_rules/*.yml\"\n\n# Scrape configurations\nscrape_configs:\n  # Prometheus self-monitoring\n  - job_name: 'prometheus'\n    static_configs:\n      - targets: ['localhost:9090']\n\n  # Node exporter for system metrics\n  - job_name: 'node'\n    static_configs:\n      - targets: \n          - 'node-exporter:9100'\n    relabel_configs:\n      - source_labels: [__address__]\n        regex: '([^:]+)(?::\\d+)?'\n        target_label: instance\n        replacement: '${1}'\n\n  # Application metrics\n  - job_name: 'application'\n    kubernetes_sd_configs:\n      - role: pod\n    relabel_configs:\n      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n        action: keep\n        regex: true\n      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]\n        action: replace\n        target_label: __metrics_path__\n        regex: (.+)\n      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]\n        action: replace\n        regex: ([^:]+)(?::\\d+)?;(\\d+)\n        replacement: $1:$2\n        target_label: __address__\n      - action: labelmap\n        regex: __meta_kubernetes_pod_label_(.+)\n      - source_labels: [__meta_kubernetes_namespace]\n        action: replace\n        target_label: kubernetes_namespace\n      - source_labels: [__meta_kubernetes_pod_name]\n        action: replace\n        target_label: kubernetes_pod_name\n\n  # Database monitoring\n  - job_name: 'postgres'\n    static_configs:\n      - targets: ['postgres-exporter:9187']\n    params:\n      query: ['pg_stat_database', 'pg_stat_replication']\n\n  # Redis monitoring\n  - job_name: 'redis'\n    static_configs:\n      - targets: ['redis-exporter:9121']\n\n  # Custom service discovery\n  - job_name: 'custom-services'\n    consul_sd_configs:\n      - server: 'consul:8500'\n        services: []\n    relabel_configs:\n      - source_labels: [__meta_consul_service]\n        target_label: service_name\n      - source_labels: [__meta_consul_tags]\n        regex: '.*,metrics,.*'\n        action: keep\n```\n\n**Custom Metrics Implementation**\n```typescript\n// metrics.ts\nimport { Counter, Histogram, Gauge, Registry } from 'prom-client';\n\nexport class MetricsCollector {\n    private registry: Registry;\n    \n    // HTTP metrics\n    private httpRequestDuration: Histogram<string>;\n    private httpRequestTotal: Counter<string>;\n    private httpRequestsInFlight: Gauge<string>;\n    \n    // Business metrics\n    private userRegistrations: Counter<string>;\n    private activeUsers: Gauge<string>;\n    private revenue: Counter<string>;\n    \n    // System metrics\n    private queueDepth: Gauge<string>;\n    private cacheHitRatio: Gauge<string>;\n    \n    constructor() {\n        this.registry = new Registry();\n        this.initializeMetrics();\n    }\n    \n    private initializeMetrics() {\n        // HTTP metrics\n        this.httpRequestDuration = new Histogram({\n            name: 'http_request_duration_seconds',\n            help: 'Duration of HTTP requests in seconds',\n            labelNames: ['method', 'route', 'status_code'],\n            buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 5]\n        });\n        \n        this.httpRequestTotal = new Counter({\n            name: 'http_requests_total',\n            help: 'Total number of HTTP requests',\n            labelNames: ['method', 'route', 'status_code']\n        });\n        \n        this.httpRequestsInFlight = new Gauge({\n            name: 'http_requests_in_flight',\n            help: 'Number of HTTP requests currently being processed',\n            labelNames: ['method', 'route']\n        });\n        \n        // Business metrics\n        this.userRegistrations = new Counter({\n            name: 'user_registrations_total',\n            help: 'Total number of user registrations',\n            labelNames: ['source', 'plan']\n        });\n        \n        this.activeUsers = new Gauge({\n            name: 'active_users',\n            help: 'Number of active users',\n            labelNames: ['timeframe']\n        });\n        \n        this.revenue = new Counter({\n            name: 'revenue_total_cents',\n            help: 'Total revenue in cents',\n            labelNames: ['product', 'currency']\n        });\n        \n        // Register all metrics\n        this.registry.registerMetric(this.httpRequestDuration);\n        this.registry.registerMetric(this.httpRequestTotal);\n        this.registry.registerMetric(this.httpRequestsInFlight);\n        this.registry.registerMetric(this.userRegistrations);\n        this.registry.registerMetric(this.activeUsers);\n        this.registry.registerMetric(this.revenue);\n    }\n    \n    // Middleware for Express\n    httpMetricsMiddleware() {\n        return (req: Request, res: Response, next: NextFunction) => {\n            const start = Date.now();\n            const route = req.route?.path || req.path;\n            \n            // Increment in-flight gauge\n            this.httpRequestsInFlight.inc({ method: req.method, route });\n            \n            res.on('finish', () => {\n                const duration = (Date.now() - start) / 1000;\n                const labels = {\n                    method: req.method,\n                    route,\n                    status_code: res.statusCode.toString()\n                };\n                \n                // Record metrics\n                this.httpRequestDuration.observe(labels, duration);\n                this.httpRequestTotal.inc(labels);\n                this.httpRequestsInFlight.dec({ method: req.method, route });\n            });\n            \n            next();\n        };\n    }\n    \n    // Business metric helpers\n    recordUserRegistration(source: string, plan: string) {\n        this.userRegistrations.inc({ source, plan });\n    }\n    \n    updateActiveUsers(timeframe: string, count: number) {\n        this.activeUsers.set({ timeframe }, count);\n    }\n    \n    recordRevenue(product: string, currency: string, amountCents: number) {\n        this.revenue.inc({ product, currency }, amountCents);\n    }\n    \n    // Export metrics endpoint\n    async getMetrics(): Promise<string> {\n        return this.registry.metrics();\n    }\n}\n\n// Recording rules for Prometheus\nexport const recordingRules = `\ngroups:\n  - name: aggregations\n    interval: 30s\n    rules:\n      # Request rate\n      - record: http_request_rate_5m\n        expr: rate(http_requests_total[5m])\n      \n      # Error rate\n      - record: http_error_rate_5m\n        expr: |\n          sum(rate(http_requests_total{status_code=~\"5..\"}[5m]))\n          /\n          sum(rate(http_requests_total[5m]))\n      \n      # P95 latency\n      - record: http_request_duration_p95_5m\n        expr: |\n          histogram_quantile(0.95,\n            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)\n          )\n      \n      # Business metrics\n      - record: user_registration_rate_1h\n        expr: rate(user_registrations_total[1h])\n      \n      - record: revenue_rate_1d\n        expr: rate(revenue_total_cents[1d]) / 100\n`;\n```\n\n### 3. Grafana Dashboard Setup\n\nCreate comprehensive dashboards:\n\n**Dashboard Configuration**\n```json\n{\n  \"dashboard\": {\n    \"title\": \"Application Overview\",\n    \"tags\": [\"production\", \"overview\"],\n    \"timezone\": \"browser\",\n    \"panels\": [\n      {\n        \"title\": \"Request Rate\",\n        \"type\": \"graph\",\n        \"gridPos\": { \"x\": 0, \"y\": 0, \"w\": 12, \"h\": 8 },\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(http_requests_total[5m])) by (method)\",\n            \"legendFormat\": \"{{method}}\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Error Rate\",\n        \"type\": \"graph\",\n        \"gridPos\": { \"x\": 12, \"y\": 0, \"w\": 12, \"h\": 8 },\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(http_requests_total{status_code=~\\\"5..\\\"}[5m])) / sum(rate(http_requests_total[5m]))\",\n            \"legendFormat\": \"Error Rate\"\n          }\n        ],\n        \"alert\": {\n          \"conditions\": [\n            {\n              \"evaluator\": { \"params\": [0.05], \"type\": \"gt\" },\n              \"query\": { \"params\": [\"A\", \"5m\", \"now\"] },\n              \"reducer\": { \"type\": \"avg\" },\n              \"type\": \"query\"\n            }\n          ],\n          \"name\": \"High Error Rate\"\n        }\n      },\n      {\n        \"title\": \"Response Time\",\n        \"type\": \"graph\",\n        \"gridPos\": { \"x\": 0, \"y\": 8, \"w\": 12, \"h\": 8 },\n        \"targets\": [\n          {\n            \"expr\": \"histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))\",\n            \"legendFormat\": \"p95\"\n          },\n          {\n            \"expr\": \"histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))\",\n            \"legendFormat\": \"p99\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Active Users\",\n        \"type\": \"stat\",\n        \"gridPos\": { \"x\": 12, \"y\": 8, \"w\": 6, \"h\": 4 },\n        \"targets\": [\n          {\n            \"expr\": \"active_users{timeframe=\\\"realtime\\\"}\"\n          }\n        ]\n      }\n    ]\n  }\n}\n```\n\n**Dashboard as Code**\n```typescript\n// dashboards/service-dashboard.ts\nimport { Dashboard, Panel, Target } from '@grafana/toolkit';\n\nexport const createServiceDashboard = (serviceName: string): Dashboard => {\n    return new Dashboard({\n        title: `${serviceName} Service Dashboard`,\n        uid: `${serviceName}-overview`,\n        tags: ['service', serviceName],\n        time: { from: 'now-6h', to: 'now' },\n        refresh: '30s',\n        \n        panels: [\n            // Row 1: Golden Signals\n            new Panel.Graph({\n                title: 'Request Rate',\n                gridPos: { x: 0, y: 0, w: 6, h: 8 },\n                targets: [\n                    new Target({\n                        expr: `sum(rate(http_requests_total{service=\"${serviceName}\"}[5m])) by (method)`,\n                        legendFormat: '{{method}}'\n                    })\n                ]\n            }),\n            \n            new Panel.Graph({\n                title: 'Error Rate',\n                gridPos: { x: 6, y: 0, w: 6, h: 8 },\n                targets: [\n                    new Target({\n                        expr: `sum(rate(http_requests_total{service=\"${serviceName}\",status_code=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"${serviceName}\"}[5m]))`,\n                        legendFormat: 'Error %'\n                    })\n                ],\n                yaxes: [{ format: 'percentunit' }]\n            }),\n            \n            new Panel.Graph({\n                title: 'Latency Percentiles',\n                gridPos: { x: 12, y: 0, w: 12, h: 8 },\n                targets: [\n                    new Target({\n                        expr: `histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=\"${serviceName}\"}[5m])) by (le))`,\n                        legendFormat: 'p50'\n                    }),\n                    new Target({\n                        expr: `histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=\"${serviceName}\"}[5m])) by (le))`,\n                        legendFormat: 'p95'\n                    }),\n                    new Target({\n                        expr: `histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"${serviceName}\"}[5m])) by (le))`,\n                        legendFormat: 'p99'\n                    })\n                ],\n                yaxes: [{ format: 's' }]\n            }),\n            \n            // Row 2: Resource Usage\n            new Panel.Graph({\n                title: 'CPU Usage',\n                gridPos: { x: 0, y: 8, w: 8, h: 8 },\n                targets: [\n                    new Target({\n                        expr: `avg(rate(container_cpu_usage_seconds_total{pod=~\"${serviceName}-.*\"}[5m])) by (pod)`,\n                        legendFormat: '{{pod}}'\n                    })\n                ],\n                yaxes: [{ format: 'percentunit' }]\n            }),\n            \n            new Panel.Graph({\n                title: 'Memory Usage',\n                gridPos: { x: 8, y: 8, w: 8, h: 8 },\n                targets: [\n                    new Target({\n                        expr: `avg(container_memory_working_set_bytes{pod=~\"${serviceName}-.*\"}) by (pod)`,\n                        legendFormat: '{{pod}}'\n                    })\n                ],\n                yaxes: [{ format: 'bytes' }]\n            }),\n            \n            new Panel.Graph({\n                title: 'Network I/O',\n                gridPos: { x: 16, y: 8, w: 8, h: 8 },\n                targets: [\n                    new Target({\n                        expr: `sum(rate(container_network_receive_bytes_total{pod=~\"${serviceName}-.*\"}[5m])) by (pod)`,\n                        legendFormat: '{{pod}} RX'\n                    }),\n                    new Target({\n                        expr: `sum(rate(container_network_transmit_bytes_total{pod=~\"${serviceName}-.*\"}[5m])) by (pod)`,\n                        legendFormat: '{{pod}} TX'\n                    })\n                ],\n                yaxes: [{ format: 'Bps' }]\n            })\n        ]\n    });\n};\n```\n\n### 4. Distributed Tracing Setup\n\nImplement OpenTelemetry-based tracing:\n\n**OpenTelemetry Configuration**\n```typescript\n// tracing.ts\nimport { NodeSDK } from '@opentelemetry/sdk-node';\nimport { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';\nimport { Resource } from '@opentelemetry/resources';\nimport { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';\nimport { JaegerExporter } from '@opentelemetry/exporter-jaeger';\nimport { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';\nimport { PrometheusExporter } from '@opentelemetry/exporter-prometheus';\n\nexport class TracingSetup {\n    private sdk: NodeSDK;\n    \n    constructor(serviceName: string, environment: string) {\n        const jaegerExporter = new JaegerExporter({\n            endpoint: process.env.JAEGER_ENDPOINT || 'http://localhost:14268/api/traces',\n        });\n        \n        const prometheusExporter = new PrometheusExporter({\n            port: 9464,\n            endpoint: '/metrics',\n        }, () => {\n            console.log('Prometheus metrics server started on port 9464');\n        });\n        \n        this.sdk = new NodeSDK({\n            resource: new Resource({\n                [SemanticResourceAttributes.SERVICE_NAME]: serviceName,\n                [SemanticResourceAttributes.SERVICE_VERSION]: process.env.SERVICE_VERSION || '1.0.0',\n                [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: environment,\n            }),\n            \n            traceExporter: jaegerExporter,\n            spanProcessor: new BatchSpanProcessor(jaegerExporter, {\n                maxQueueSize: 2048,\n                maxExportBatchSize: 512,\n                scheduledDelayMillis: 5000,\n                exportTimeoutMillis: 30000,\n            }),\n            \n            metricExporter: prometheusExporter,\n            \n            instrumentations: [\n                getNodeAutoInstrumentations({\n                    '@opentelemetry/instrumentation-fs': {\n                        enabled: false,\n                    },\n                }),\n            ],\n        });\n    }\n    \n    start() {\n        this.sdk.start()\n            .then(() => console.log('Tracing initialized'))\n            .catch((error) => console.error('Error initializing tracing', error));\n    }\n    \n    shutdown() {\n        return this.sdk.shutdown()\n            .then(() => console.log('Tracing terminated'))\n            .catch((error) => console.error('Error terminating tracing', error));\n    }\n}\n\n// Custom span creation\nimport { trace, context, SpanStatusCode, SpanKind } from '@opentelemetry/api';\n\nexport class CustomTracer {\n    private tracer = trace.getTracer('custom-tracer', '1.0.0');\n    \n    async traceOperation<T>(\n        operationName: string,\n        operation: () => Promise<T>,\n        attributes?: Record<string, any>\n    ): Promise<T> {\n        const span = this.tracer.startSpan(operationName, {\n            kind: SpanKind.INTERNAL,\n            attributes,\n        });\n        \n        return context.with(trace.setSpan(context.active(), span), async () => {\n            try {\n                const result = await operation();\n                span.setStatus({ code: SpanStatusCode.OK });\n                return result;\n            } catch (error) {\n                span.recordException(error as Error);\n                span.setStatus({\n                    code: SpanStatusCode.ERROR,\n                    message: error.message,\n                });\n                throw error;\n            } finally {\n                span.end();\n            }\n        });\n    }\n    \n    // Database query tracing\n    async traceQuery<T>(\n        queryName: string,\n        query: () => Promise<T>,\n        sql?: string\n    ): Promise<T> {\n        return this.traceOperation(\n            `db.query.${queryName}`,\n            query,\n            {\n                'db.system': 'postgresql',\n                'db.operation': queryName,\n                'db.statement': sql,\n            }\n        );\n    }\n    \n    // HTTP request tracing\n    async traceHttpRequest<T>(\n        method: string,\n        url: string,\n        request: () => Promise<T>\n    ): Promise<T> {\n        return this.traceOperation(\n            `http.request`,\n            request,\n            {\n                'http.method': method,\n                'http.url': url,\n                'http.target': new URL(url).pathname,\n            }\n        );\n    }\n}\n```\n\n### 5. Log Aggregation Setup\n\nImplement centralized logging:\n\n**Fluentd Configuration**\n```yaml\n# fluent.conf\n<source>\n  @type tail\n  path /var/log/containers/*.log\n  pos_file /var/log/fluentd-containers.log.pos\n  tag kubernetes.*\n  <parse>\n    @type json\n    time_format %Y-%m-%dT%H:%M:%S.%NZ\n  </parse>\n</source>\n\n# Add Kubernetes metadata\n<filter kubernetes.**>\n  @type kubernetes_metadata\n  @id filter_kube_metadata\n  kubernetes_url \"#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}\"\n  verify_ssl \"#{ENV['KUBERNETES_VERIFY_SSL'] || true}\"\n</filter>\n\n# Parse application logs\n<filter kubernetes.**>\n  @type parser\n  key_name log\n  reserve_data true\n  remove_key_name_field true\n  <parse>\n    @type multi_format\n    <pattern>\n      format json\n    </pattern>\n    <pattern>\n      format regexp\n      expression /^(?<severity>\\w+)\\s+\\[(?<timestamp>[^\\]]+)\\]\\s+(?<message>.*)$/\n      time_format %Y-%m-%d %H:%M:%S\n    </pattern>\n  </parse>\n</filter>\n\n# Add fields\n<filter kubernetes.**>\n  @type record_transformer\n  enable_ruby true\n  <record>\n    cluster_name ${ENV['CLUSTER_NAME']}\n    environment ${ENV['ENVIRONMENT']}\n    @timestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}\n  </record>\n</filter>\n\n# Output to Elasticsearch\n<match kubernetes.**>\n  @type elasticsearch\n  @id out_es\n  @log_level info\n  include_tag_key true\n  host \"#{ENV['FLUENT_ELASTICSEARCH_HOST']}\"\n  port \"#{ENV['FLUENT_ELASTICSEARCH_PORT']}\"\n  path \"#{ENV['FLUENT_ELASTICSEARCH_PATH']}\"\n  scheme \"#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}\"\n  ssl_verify \"#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}\"\n  ssl_version \"#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}\"\n  user \"#{ENV['FLUENT_ELASTICSEARCH_USER']}\"\n  password \"#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}\"\n  index_name logstash\n  logstash_format true\n  logstash_prefix \"#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}\"\n  <buffer>\n    @type file\n    path /var/log/fluentd-buffers/kubernetes.system.buffer\n    flush_mode interval\n    retry_type exponential_backoff\n    flush_interval 5s\n    retry_max_interval 30\n    chunk_limit_size 2M\n    queue_limit_length 8\n    overflow_action block\n  </buffer>\n</match>\n```\n\n**Structured Logging Library**\n```python\n# structured_logging.py\nimport json\nimport logging\nimport traceback\nfrom datetime import datetime\nfrom typing import Any, Dict, Optional\n\nclass StructuredLogger:\n    def __init__(self, name: str, service: str, version: str):\n        self.logger = logging.getLogger(name)\n        self.service = service\n        self.version = version\n        self.default_context = {\n            'service': service,\n            'version': version,\n            'environment': os.getenv('ENVIRONMENT', 'development')\n        }\n    \n    def _format_log(self, level: str, message: str, context: Dict[str, Any]) -> str:\n        log_entry = {\n            '@timestamp': datetime.utcnow().isoformat() + 'Z',\n            'level': level,\n            'message': message,\n            **self.default_context,\n            **context\n        }\n        \n        # Add trace context if available\n        trace_context = self._get_trace_context()\n        if trace_context:\n            log_entry['trace'] = trace_context\n        \n        return json.dumps(log_entry)\n    \n    def _get_trace_context(self) -> Optional[Dict[str, str]]:\n        \"\"\"Extract trace context from OpenTelemetry\"\"\"\n        from opentelemetry import trace\n        \n        span = trace.get_current_span()\n        if span and span.is_recording():\n            span_context = span.get_span_context()\n            return {\n                'trace_id': format(span_context.trace_id, '032x'),\n                'span_id': format(span_context.span_id, '016x'),\n            }\n        return None\n    \n    def info(self, message: str, **context):\n        log_msg = self._format_log('INFO', message, context)\n        self.logger.info(log_msg)\n    \n    def error(self, message: str, error: Optional[Exception] = None, **context):\n        if error:\n            context['error'] = {\n                'type': type(error).__name__,\n                'message': str(error),\n                'stacktrace': traceback.format_exc()\n            }\n        \n        log_msg = self._format_log('ERROR', message, context)\n        self.logger.error(log_msg)\n    \n    def warning(self, message: str, **context):\n        log_msg = self._format_log('WARNING', message, context)\n        self.logger.warning(log_msg)\n    \n    def debug(self, message: str, **context):\n        log_msg = self._format_log('DEBUG', message, context)\n        self.logger.debug(log_msg)\n    \n    def audit(self, action: str, user_id: str, details: Dict[str, Any]):\n        \"\"\"Special method for audit logging\"\"\"\n        self.info(\n            f\"Audit: {action}\",\n            audit=True,\n            user_id=user_id,\n            action=action,\n            details=details\n        )\n\n# Log correlation middleware\nfrom flask import Flask, request, g\nimport uuid\n\ndef setup_request_logging(app: Flask, logger: StructuredLogger):\n    @app.before_request\n    def before_request():\n        g.request_id = request.headers.get('X-Request-ID', str(uuid.uuid4()))\n        g.request_start = datetime.utcnow()\n        \n        logger.info(\n            \"Request started\",\n            request_id=g.request_id,\n            method=request.method,\n            path=request.path,\n            remote_addr=request.remote_addr,\n            user_agent=request.headers.get('User-Agent')\n        )\n    \n    @app.after_request\n    def after_request(response):\n        duration = (datetime.utcnow() - g.request_start).total_seconds()\n        \n        logger.info(\n            \"Request completed\",\n            request_id=g.request_id,\n            method=request.method,\n            path=request.path,\n            status_code=response.status_code,\n            duration=duration\n        )\n        \n        response.headers['X-Request-ID'] = g.request_id\n        return response\n```\n\n### 6. Alert Configuration\n\nSet up intelligent alerting:\n\n**Alert Rules**\n```yaml\n# alerts/application.yml\ngroups:\n  - name: application\n    interval: 30s\n    rules:\n      # High error rate\n      - alert: HighErrorRate\n        expr: |\n          sum(rate(http_requests_total{status_code=~\"5..\"}[5m])) by (service)\n          /\n          sum(rate(http_requests_total[5m])) by (service)\n          > 0.05\n        for: 5m\n        labels:\n          severity: critical\n          team: backend\n        annotations:\n          summary: \"High error rate on {{ $labels.service }}\"\n          description: \"Error rate is {{ $value | humanizePercentage }} for {{ $labels.service }}\"\n          runbook_url: \"https://wiki.company.com/runbooks/high-error-rate\"\n      \n      # Slow response time\n      - alert: SlowResponseTime\n        expr: |\n          histogram_quantile(0.95,\n            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)\n          ) > 1\n        for: 10m\n        labels:\n          severity: warning\n          team: backend\n        annotations:\n          summary: \"Slow response time on {{ $labels.service }}\"\n          description: \"95th percentile response time is {{ $value }}s\"\n      \n      # Pod restart\n      - alert: PodRestarting\n        expr: |\n          increase(kube_pod_container_status_restarts_total[1h]) > 5\n        labels:\n          severity: warning\n          team: platform\n        annotations:\n          summary: \"Pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting\"\n          description: \"Pod has restarted {{ $value }} times in the last hour\"\n\n  - name: infrastructure\n    interval: 30s\n    rules:\n      # High CPU usage\n      - alert: HighCPUUsage\n        expr: |\n          avg(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)\n          > 0.8\n        for: 15m\n        labels:\n          severity: warning\n          team: platform\n        annotations:\n          summary: \"High CPU usage on {{ $labels.pod }}\"\n          description: \"CPU usage is {{ $value | humanizePercentage }}\"\n      \n      # Memory pressure\n      - alert: HighMemoryUsage\n        expr: |\n          container_memory_working_set_bytes\n          / container_spec_memory_limit_bytes\n          > 0.9\n        for: 10m\n        labels:\n          severity: critical\n          team: platform\n        annotations:\n          summary: \"High memory usage on {{ $labels.pod }}\"\n          description: \"Memory usage is {{ $value | humanizePercentage }} of limit\"\n      \n      # Disk space\n      - alert: DiskSpaceLow\n        expr: |\n          node_filesystem_avail_bytes{mountpoint=\"/\"}\n          / node_filesystem_size_bytes{mountpoint=\"/\"}\n          < 0.1\n        for: 5m\n        labels:\n          severity: critical\n          team: platform\n        annotations:\n          summary: \"Low disk space on {{ $labels.instance }}\"\n          description: \"Only {{ $value | humanizePercentage }} disk space remaining\"\n```\n\n**Alertmanager Configuration**\n```yaml\n# alertmanager.yml\nglobal:\n  resolve_timeout: 5m\n  slack_api_url: '$SLACK_API_URL'\n  pagerduty_url: '$PAGERDUTY_URL'\n\nroute:\n  group_by: ['alertname', 'cluster', 'service']\n  group_wait: 10s\n  group_interval: 10s\n  repeat_interval: 12h\n  receiver: 'default'\n  \n  routes:\n    # Critical alerts go to PagerDuty\n    - match:\n        severity: critical\n      receiver: pagerduty\n      continue: true\n    \n    # All alerts go to Slack\n    - match_re:\n        severity: critical|warning\n      receiver: slack\n    \n    # Database alerts to DBA team\n    - match:\n        service: database\n      receiver: dba-team\n\nreceivers:\n  - name: 'default'\n    \n  - name: 'slack'\n    slack_configs:\n      - channel: '#alerts'\n        title: '{{ .GroupLabels.alertname }}'\n        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'\n        send_resolved: true\n        actions:\n          - type: button\n            text: 'Runbook'\n            url: '{{ .Annotations.runbook_url }}'\n          - type: button\n            text: 'Dashboard'\n            url: 'https://grafana.company.com/d/{{ .Labels.service }}'\n  \n  - name: 'pagerduty'\n    pagerduty_configs:\n      - service_key: '$PAGERDUTY_SERVICE_KEY'\n        description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'\n        details:\n          firing: '{{ .Alerts.Firing | len }}'\n          resolved: '{{ .Alerts.Resolved | len }}'\n          alerts: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'\n\ninhibit_rules:\n  # Inhibit warning alerts if critical alert is firing\n  - source_match:\n      severity: 'critical'\n    target_match:\n      severity: 'warning'\n    equal: ['alertname', 'service']\n```\n\n### 7. SLO Implementation\n\nDefine and monitor Service Level Objectives:\n\n**SLO Configuration**\n```typescript\n// slo-manager.ts\ninterface SLO {\n    name: string;\n    description: string;\n    sli: {\n        metric: string;\n        threshold: number;\n        comparison: 'lt' | 'gt' | 'eq';\n    };\n    target: number; // e.g., 99.9\n    window: string; // e.g., '30d'\n    burnRates: BurnRate[];\n}\n\ninterface BurnRate {\n    window: string;\n    threshold: number;\n    severity: 'warning' | 'critical';\n}\n\nexport class SLOManager {\n    private slos: SLO[] = [\n        {\n            name: 'API Availability',\n            description: 'Percentage of successful requests',\n            sli: {\n                metric: 'http_requests_total{status_code!~\"5..\"}',\n                threshold: 0,\n                comparison: 'gt'\n            },\n            target: 99.9,\n            window: '30d',\n            burnRates: [\n                { window: '1h', threshold: 14.4, severity: 'critical' },\n                { window: '6h', threshold: 6, severity: 'critical' },\n                { window: '1d', threshold: 3, severity: 'warning' },\n                { window: '3d', threshold: 1, severity: 'warning' }\n            ]\n        },\n        {\n            name: 'API Latency',\n            description: '95th percentile response time under 500ms',\n            sli: {\n                metric: 'http_request_duration_seconds',\n                threshold: 0.5,\n                comparison: 'lt'\n            },\n            target: 99,\n            window: '30d',\n            burnRates: [\n                { window: '1h', threshold: 36, severity: 'critical' },\n                { window: '6h', threshold: 12, severity: 'warning' }\n            ]\n        }\n    ];\n    \n    generateSLOQueries(): string {\n        return this.slos.map(slo => this.generateSLOQuery(slo)).join('\\n\\n');\n    }\n    \n    private generateSLOQuery(slo: SLO): string {\n        const errorBudget = 1 - (slo.target / 100);\n        \n        return `\n# ${slo.name} SLO\n- record: slo:${this.sanitizeName(slo.name)}:error_budget\n  expr: ${errorBudget}\n\n- record: slo:${this.sanitizeName(slo.name)}:consumed_error_budget\n  expr: |\n    1 - (\n      sum(rate(${slo.sli.metric}[${slo.window}]))\n      /\n      sum(rate(http_requests_total[${slo.window}]))\n    )\n\n${slo.burnRates.map(burnRate => `\n- alert: ${this.sanitizeName(slo.name)}BurnRate${burnRate.window}\n  expr: |\n    slo:${this.sanitizeName(slo.name)}:consumed_error_budget\n    > ${burnRate.threshold} * slo:${this.sanitizeName(slo.name)}:error_budget\n  labels:\n    severity: ${burnRate.severity}\n    slo: ${slo.name}\n  annotations:\n    summary: \"${slo.name} SLO burn rate too high\"\n    description: \"Burning through error budget ${burnRate.threshold}x faster than sustainable\"\n`).join('\\n')}\n        `;\n    }\n    \n    private sanitizeName(name: string): string {\n        return name.toLowerCase().replace(/\\s+/g, '_').replace(/[^a-z0-9_]/g, '');\n    }\n}\n```\n\n### 8. Monitoring Infrastructure as Code\n\nDeploy monitoring stack with Terraform:\n\n**Terraform Configuration**\n```hcl\n# monitoring.tf\nmodule \"prometheus\" {\n  source = \"./modules/prometheus\"\n  \n  namespace = \"monitoring\"\n  storage_size = \"100Gi\"\n  retention_days = 30\n  \n  external_labels = {\n    cluster = var.cluster_name\n    region  = var.region\n  }\n  \n  scrape_configs = [\n    {\n      job_name = \"kubernetes-pods\"\n      kubernetes_sd_configs = [{\n        role = \"pod\"\n      }]\n    }\n  ]\n  \n  alerting_rules = file(\"${path.module}/alerts/*.yml\")\n}\n\nmodule \"grafana\" {\n  source = \"./modules/grafana\"\n  \n  namespace = \"monitoring\"\n  \n  admin_password = var.grafana_admin_password\n  \n  datasources = [\n    {\n      name = \"Prometheus\"\n      type = \"prometheus\"\n      url  = \"http://prometheus:9090\"\n    },\n    {\n      name = \"Loki\"\n      type = \"loki\"\n      url  = \"http://loki:3100\"\n    },\n    {\n      name = \"Jaeger\"\n      type = \"jaeger\"\n      url  = \"http://jaeger-query:16686\"\n    }\n  ]\n  \n  dashboard_configs = [\n    {\n      name = \"default\"\n      folder = \"General\"\n      type = \"file\"\n      options = {\n        path = \"/var/lib/grafana/dashboards\"\n      }\n    }\n  ]\n}\n\nmodule \"loki\" {\n  source = \"./modules/loki\"\n  \n  namespace = \"monitoring\"\n  storage_size = \"50Gi\"\n  \n  ingester_config = {\n    chunk_idle_period = \"15m\"\n    chunk_retain_period = \"30s\"\n    max_chunk_age = \"1h\"\n  }\n}\n\nmodule \"alertmanager\" {\n  source = \"./modules/alertmanager\"\n  \n  namespace = \"monitoring\"\n  \n  config = templatefile(\"${path.module}/alertmanager.yml\", {\n    slack_webhook = var.slack_webhook\n    pagerduty_key = var.pagerduty_service_key\n  })\n}\n```\n\n## Output Format\n\n1. **Infrastructure Assessment**: Current monitoring capabilities analysis\n2. **Monitoring Architecture**: Complete monitoring stack design\n3. **Implementation Plan**: Step-by-step deployment guide\n4. **Metric Definitions**: Comprehensive metrics catalog\n5. **Dashboard Templates**: Ready-to-use Grafana dashboards\n6. **Alert Runbooks**: Detailed alert response procedures\n7. **SLO Definitions**: Service level objectives and error budgets\n8. **Integration Guide**: Service instrumentation instructions\n\nFocus on creating a monitoring system that provides actionable insights, reduces MTTR, and enables proactive issue detection.","contentHash":"797020d120c882ebd14d93313a47018bff33e21985dc93a147658eb59caf5c0d","copies":0,"createdAt":"2025-08-12T16:09:38.630Z","description":"Set up comprehensive monitoring and observability","github":{"repoUrl":"https://github.com/Commands-com/commands","lastSyncDirection":"from-github","metadata":{"importedFrom":"github_repository","repoPrivate":false,"repoDefaultBranch":"main","connectedAt":"2025-08-12T16:09:38.630Z"},"importedAt":"2025-08-12T16:09:38.630Z","lastSyncAt":"2025-08-17T17:57:47.983Z","fileMapping":{"license":null,"readme":null,"assets":[],"mainFile":"tools/monitor-setup.md"},"selectedCommand":"monitor-setup","fileShas":{"mainFile":"0782d96d0bec1a9430ec1fd3243bbcc22f6e8695","yamlPath":"313647b1fb381389da33b7913e95baf617c4b392"},"branch":"main","connectionType":"commands_yaml","connected":true,"lastSyncCommit":"01591bc061d236bde47bf23b0f47e8afcf1a5144","importSource":"repository_import","installationId":"69232615","syncStatus":"synced"},"githubRepoUrl":"https://github.com/Commands-com/commands","id":"44573a12-314b-46b2-949c-d3cc4424446d","inputParameters":[{"defaultValue":"prometheus-grafana","name":"monitoring_stack","options":["prometheus-grafana","elk-stack","datadog","new-relic","aws-cloudwatch","gcp-stackdriver","azure-monitor"],"description":"Monitoring stack to implement","label":"Monitoring Stack","type":"select","required":false},{"name":"alerting_channels","description":"Where to send alerts (comma-separated)","label":"Alerting Channels","type":"text","required":false,"defaultValue":"slack,email"}],"instructions":"Set up comprehensive monitoring and observability","likes":0,"mcp_search_content":"","organizationUsername":"commands-com","price":"free","search_content":"monitor setup set up comprehensive monitoring and observability /monitor-setup deployment claude-code@2025.06","title":"Monitor Setup","type":"command","updatedAt":"2025-08-17T17:57:47.983Z","userId":"W0V8NAw5AhWRwcuwSoFLOi1Yem83","visibility":"public","name":"monitor-setup","userInteraction":{"userHasStarred":false}}