{"aiPlatform":"claude-code@2025.06","category":"data-analysis","commandName":"/data-pipeline","content":"---\nname: Data Pipeline Architecture\ndescription: Design and implement scalable data pipelines with ingestion, transformation, and processing capabilities for various data sources and destinations.\nallowed_tools:\n  - filesystem      # Access data files and pipeline configurations\n  - memory          # Track data flow patterns and pipeline metrics\n  - sqlite          # Store pipeline metadata and monitoring data\ntags:\n  - data-pipeline\n  - data-processing\n  - etl\n  - streaming\n  - batch-processing\ncategory: development\nversion: 1.0.0\nauthor: AI Commands Team\n---\n\n# Data Pipeline Architecture\n\nDesign and implement a scalable data pipeline for: $ARGUMENTS\n\nCreate a production-ready data pipeline including:\n\n1. **Data Ingestion**:\n   - Multiple source connectors (APIs, databases, files, streams)\n   - Schema evolution handling\n   - Incremental/batch loading\n   - Data quality checks at ingestion\n   - Dead letter queue for failures\n\n2. **Transformation Layer**:\n   - ETL/ELT architecture decision\n   - Apache Beam/Spark transformations\n   - Data cleansing and normalization\n   - Feature engineering pipeline\n   - Business logic implementation\n\n3. **Orchestration**:\n   - Airflow/Prefect DAGs\n   - Dependency management\n   - Retry and failure handling\n   - SLA monitoring\n   - Dynamic pipeline generation\n\n4. **Storage Strategy**:\n   - Data lake architecture\n   - Partitioning strategy\n   - Compression choices\n   - Retention policies\n   - Hot/cold storage tiers\n\n5. **Streaming Pipeline**:\n   - Kafka/Kinesis integration\n   - Real-time processing\n   - Windowing strategies\n   - Late data handling\n   - Exactly-once semantics\n\n6. **Data Quality**:\n   - Automated testing\n   - Data profiling\n   - Anomaly detection\n   - Lineage tracking\n   - Quality metrics and dashboards\n\n7. **Performance & Scale**:\n   - Horizontal scaling\n   - Resource optimization\n   - Caching strategies\n   - Query optimization\n   - Cost management\n\nInclude monitoring, alerting, and data governance considerations. Make it cloud-agnostic with specific implementation examples for AWS/GCP/Azure.","contentHash":"0c273b91ab8ffd46f3112ec73e8e46b6a72173cc8e3702e5fc2c38d940914537","copies":0,"createdAt":"2025-08-12T16:09:37.165Z","description":"Design scalable data pipeline architectures","github":{"repoUrl":"https://github.com/Commands-com/commands","lastSyncDirection":"from-github","metadata":{"importedFrom":"github_repository","repoPrivate":false,"repoDefaultBranch":"main","connectedAt":"2025-08-12T16:09:37.165Z"},"importedAt":"2025-08-12T16:09:37.165Z","lastSyncAt":"2025-08-17T17:57:47.938Z","fileMapping":{"license":null,"readme":null,"assets":[],"mainFile":"tools/data-pipeline.md"},"selectedCommand":"data-pipeline","fileShas":{"mainFile":"630f8a1d0d2ece70e649fae84cb40aeebc60f710","yamlPath":"313647b1fb381389da33b7913e95baf617c4b392"},"branch":"main","connectionType":"commands_yaml","connected":true,"lastSyncCommit":"01591bc061d236bde47bf23b0f47e8afcf1a5144","importSource":"repository_import","installationId":"69232615","syncStatus":"synced"},"githubRepoUrl":"https://github.com/Commands-com/commands","id":"75edb52c-8f64-4ef2-971a-750749e8dcd1","inputParameters":[{"defaultValue":"airflow","name":"pipeline_framework","options":["airflow","dagster","prefect","spark","beam","kafka-streams","custom"],"description":"Data pipeline framework to use","label":"Pipeline Framework","type":"select","required":false},{"defaultValue":"medium","name":"data_volume","options":["small-gb","medium-tb","large-pb","streaming-realtime"],"description":"Expected data volume","label":"Data Volume","type":"select","required":false}],"instructions":"Design scalable data pipeline architectures","likes":0,"mcp_search_content":"","organizationUsername":"commands-com","price":"free","search_content":"data pipeline design scalable data pipeline architectures /data-pipeline data-analysis claude-code@2025.06","title":"Data Pipeline","type":"command","updatedAt":"2025-08-17T17:57:47.938Z","userId":"W0V8NAw5AhWRwcuwSoFLOi1Yem83","visibility":"public","name":"data-pipeline","userInteraction":{"userHasStarred":false}}