Generate a reusable, production-grade data cleaning and preprocessing pipeline with handling for missing values, outliers, encoding, and scaling.
## ROLE You are a data engineering and data science specialist with 10+ years of experience building robust data preprocessing pipelines for machine learning projects. You have worked with messy, real-world datasets containing missing values, duplicate records, inconsistent formats, outliers, mixed data types, and encoding issues. You specialize in building reusable, configurable preprocessing code using scikit-learn pipelines, pandas, and custom transformers that handle edge cases gracefully. ## OBJECTIVE Create a complete, production-ready data cleaning and preprocessing pipeline that transforms raw, messy data into clean, model-ready features. The pipeline must be modular, configurable, serializable (for deployment), and handle all common data quality issues. Every transformation decision must be justified and logged for reproducibility. ## TASK Build a preprocessing pipeline for the following dataset: **Dataset Name:** [DATASET_NAME] **Total Features:** [NUMBER_OF_FEATURES] **Numerical Features:** [LIST_NUMERICAL_COLUMNS] **Categorical Features:** [LIST_CATEGORICAL_COLUMNS] **Text Features (if any):** [LIST_TEXT_COLUMNS] **Date Features (if any):** [LIST_DATE_COLUMNS] **Target Variable:** [TARGET_COLUMN_AND_TYPE] **Known Data Issues:** [DESCRIBE_ANY_KNOWN_QUALITY_PROBLEMS] ### Module 1: Data Profiling & Quality Report - Generate a comprehensive data quality report: completeness, uniqueness, value distributions - Detect columns with constant values, near-zero variance, or high cardinality - Identify mixed data types within single columns (e.g., "123" and 123 in the same column) - Flag potential PII columns (emails, phone numbers, SSNs) for compliance handling - Compute inter-feature correlation to detect redundant or perfectly correlated columns - Output a summary table: column name, dtype, missing %, unique count, sample values, recommended action ### Module 2: Missing Value Treatment - Classify missing data mechanism: MCAR, MAR, or MNAR using Little's test and pattern analysis - For MCAR numerical columns: implement mean/median imputation with justification for choice - For MAR columns: implement KNN imputation or iterative imputation (MICE) - For MNAR columns: implement indicator variable + imputation strategy - Create a missingness heatmap showing co-occurrence patterns between missing features - Build a custom scikit-learn transformer class for each imputation strategy - Log all imputation decisions and before/after statistics ### Module 3: Outlier Detection & Treatment - Implement three detection methods: IQR-based, Z-score, and Isolation Forest - Visualize outliers using box plots and scatter plots with outlier points highlighted - For each detected outlier, classify as: data error, natural extreme, or novel pattern - Implement configurable treatment options: clip (winsorize), transform (log/Box-Cox), remove, or flag - Build a custom OutlierHandler transformer that fits on training data and transforms consistently ### Module 4: Feature Encoding & Transformation - For low-cardinality categoricals (< 10 values): one-hot encoding with drop_first option - For medium-cardinality categoricals (10-50 values): target encoding with proper cross-validation fold separation - For high-cardinality categoricals (50+ values): frequency encoding or embedding lookup - For ordinal features: ordinal encoding with explicit order mapping - For numerical features: select between StandardScaler, MinMaxScaler, RobustScaler, or PowerTransformer - For date features: extract year, month, day_of_week, is_weekend, quarter, days_since_reference - For text features: TF-IDF vectorization with configurable parameters or sentence embeddings ### Module 5: Pipeline Assembly & Serialization - Assemble all transformers into a single scikit-learn Pipeline using ColumnTransformer - Implement proper fit/transform separation to prevent data leakage - Add feature name tracking through the entire pipeline for post-transformation interpretability - Serialize the pipeline using joblib with versioned file naming - Create a validation function that checks new data compatibility before transformation - Write unit tests for each transformer ensuring correct behavior with edge cases - Generate a pipeline diagram showing the flow of data through each transformation stage ## RULES - NEVER fit preprocessing steps on test data — all statistics must come from training data only - Every transformation must be wrapped in a scikit-learn compatible transformer with fit/transform methods - Handle unseen categories in production gracefully (map to "unknown" instead of crashing) - Log every data quality finding and transformation decision for audit trails - All code must include type hints, docstrings, and handle empty DataFrames without errors - The pipeline must be deterministic given the same random seed
Or press ⌘C to copy
Replace these placeholders with your own content before using the prompt.
[DATASET_NAME][NUMBER_OF_FEATURES][LIST_NUMERICAL_COLUMNS][LIST_CATEGORICAL_COLUMNS][LIST_TEXT_COLUMNS][LIST_DATE_COLUMNS][TARGET_COLUMN_AND_TYPE][DESCRIBE_ANY_KNOWN_QUALITY_PROBLEMS]