Для реализации собственного бизнеса его необходимо официально зарегистрировать. Банки…
- User
Для реализации собственного бизнеса его необходимо официально зарегистрировать. Банки заинтересованы в том, чтобы люди обращались к ним за этой услугой. Информация о том, кто из клиентов готов начать свое дело, помогает банкам оптимизировать маркетинговые кампании, улучшить персонализированные предложения и экономить ресурсы. Ваша задача — найти людей, наиболее похожих на тех, кто уже открыл бизнес после коммуникации с банком.
Данные:
В этой задаче вам предстоит разработать модель, которая будет определять вероятность регистрации клиентами ИП или ООО в ближайшие дни после персональной коммуникации с банком. train.csv csv включает как данные о клиентах, зарегистрировавших бизнес в период с мая по август 2023 года в течение 14 дней после коммуникации, так и данные о клиентах, которые не зарегистрировали бизнес в этот промежуток времени. test.csv содержит информацию о клиентах, для которых необходимо предоставить предсказание вашей модели. В вашем распоряжении будут разнообразные признаки клиентов, такие как: mb — информация о различных сессиях в мобильном приложении, application — заявки на продукты банка, bki — характеристики обращения в бюро кредитных историй, partner — информация о подписках на партнерские сервисы, graph — данные о контактных книжках клиентов, feature — другие анонимизированные признаки, target — значение целевой переменной, указывающей наличие (1) или отсутствие (0) регистрации ИП или ООО. В качестве решения загрузите csv-файл, содержащий индексы объектов из тестовой выборки и предсказания вероятностей регистрации, полученные с помощью вашего решения.
Метрика: recall@k - полнота, посчитанная среди k объектов с наибольшими вероятностями положительного класса.
r e c a l l @ k
число верно предсказанных моделью объектов положительного класса среди top-k общее число объектов положительного класса
На практике производить коммуникацию возможно только с некоторой выборкой наиболее подходящих кандидатов. Поэтому решение оценивается с помощью полноты на топ-15% тестовых данных, т.е. сначала результаты сортируются по убыванию вероятности, предсказанной моделью, а затем оценивается доля успешных регистраций, попавших в этот топ. Подробнее: https://habr.com/ru/companies/otus/articles/732842/
Количество баллов рассчитывается после завершения раунда по формуле: P o i n t s
400 ∗ p e r c e n t i l e o f s c o r e ( r e c a l l @ k ′ s , r e c a l l @ k ) , где p e r c e n t i l e o f s c o r e – функция, вычисляющая percentile rank, r e c a l l @ k ′ s – значения метрик recall @ k всех участников, r e c a l l @ k – значение метрики recall @ k данного участника. Файлы train.tsv test.tsv
train.tsv и test.tsv содержат в себе graph_1 graph_2 graph_3 feature_1 bki_1 graph_4 feature_2 graph_5 feature_3 graph_6 graph_7 graph_8 graph_9 graph_10 feature_4 graph_11 graph_12 graph_13 bki_2 graph_14 feature_5 mb_1 graph_15 mb_2 graph_16 bki_3 feature_6 feature_7 mb_3 feature_8 feature_9 graph_17 bki_4 feature_10 feature_11 feature_12 feature_13 feature_14 mb_4 feature_15 mb_5 graph_18 bki_5 feature_16 partner_1 feature_17 feature_18 partner_2 bki_6 partner_3 feature_19 feature_20 mb_6 feature_21 bki_7 feature_22 mb_7 feature_23 mb_8 bki_8 mb_9 mb_10 bki_9 feature_24 partner_4 feature_25 feature_26 feature_27 partner_5 application_1 feature_28 mb_11 mb_12 feature_29 bki_10 bki_11 feature_30 feature_31 bki_12 feature_32 mb_13 graph_19 graph_20 graph_21 partner_6 mb_14 feature_33 bki_13 feature_34 feature_35 feature_36 graph_22 application_2 bki_14 bki_15 feature_37 bki_16 bki_17 mb_15 feature_38
- Assistant
Understanding the Problem
You are tasked with building a predictive model to identify clients who are most likely to register as an individual entrepreneur (ИП) or a limited liability company (ООО) within 14 days after personal communication with the bank. This is valuable for the bank as it helps them target marketing efforts more effectively.
Data Description
- Training Data (
train.tsv): Contains historical data of clients, including whether they registered a business (target variable). - Test Data (
test.tsv): Contains similar features but without the target variable. Your model will generate predictions for these clients.
Features:
- mb_: Information about various sessions in the mobile application.
- application_: Applications for bank products.
- bki_: Features related to credit history bureau inquiries.
- partner_: Information about subscriptions to partner services.
- graph_: Data about clients' contact books.
- feature_: Other anonymized features.
- target: Indicates if the client registered a business (1) or not (0) within the specified period (available only in training data).
Objective
Develop a model that predicts the probability of each client in the test set registering a business. Your predictions will be evaluated using the Recall@Top15% metric.
Evaluation Metric: Recall@Top15%
-
Recall@K measures the proportion of actual positive cases that are correctly identified in the top K predictions.
-
For this task, K is set to 15% of the test dataset (i.e., top 15% clients with the highest predicted probabilities).
-
Formula:
[ \text{Recall@Top15%} = \frac{\text{Number of true positives in top 15%}}{\text{Total number of positives in the test set}} ]
Approach to Solving the Problem
-
Data Exploration and Preprocessing
-
Load the Data:
- Read both
train.tsvandtest.tsvinto pandas DataFrames.
- Read both
-
Explore the Data:
- Check for missing values and data types.
- Analyze the distribution of features and the target variable.
- Identify categorical and numerical features.
-
Handle Missing Values:
- Decide whether to impute, fill, or drop missing values.
- Common strategies include mean/median imputation for numerical features and mode or a separate category for categorical features.
-
Feature Engineering:
- Create new features if possible (e.g., interactions between features).
- Convert categorical variables using encoding techniques (e.g., One-Hot Encoding, Label Encoding).
- Normalize or scale numerical features if necessary.
-
Dimensionality Reduction (Optional):
- If there are many features, consider techniques like PCA to reduce dimensionality.
-
-
Model Selection
-
Choose Appropriate Algorithms:
- Since the metric focuses on the top predictions, algorithms that can predict probabilities are preferred.
- Consider models like Logistic Regression, Random Forest, Gradient Boosting (e.g., XGBoost, LightGBM, CatBoost).
-
Handling Imbalanced Data:
- If the target classes are imbalanced, consider techniques such as:
- Resampling Methods: Oversampling the minority class or undersampling the majority class.
- Use of Class Weights: Adjust the algorithm to pay more attention to the minority class.
- Specialized Algorithms: Use algorithms that are robust to class imbalance.
- If the target classes are imbalanced, consider techniques such as:
-
-
Training the Model
-
Split the Training Data:
- Use techniques like cross-validation to evaluate model performance.
- Since the evaluation metric is Recall@Top15%, ensure your validation strategy reflects this.
-
Model Training:
- Train the model using the preprocessed features.
- Optimize for metrics that correlate with Recall@Top15%, such as AUC-ROC, but focus on the recall at high thresholds.
-
-
Model Evaluation
-
Custom Evaluation Metric:
- Implement a function to calculate Recall@Top15% on your validation set.
- Use this function to guide model selection and hyperparameter tuning.
-
Hyperparameter Tuning:
- Use techniques like Grid Search or Random Search with cross-validation to find the best parameters.
-
-
Making Predictions
-
Predict Probabilities:
- For the test set, obtain predicted probabilities for the positive class.
-
Prepare Submission File:
- Create a CSV file with two columns:
client_id(or the appropriate index) andprediction(the predicted probability). - Ensure that the indices match those in the test data.
- Create a CSV file with two columns:
-
-
Submission
-
Check Submission Format:
- Verify that your submission file meets the required format.
-
Final Validation:
- Before submitting, double-check your predictions and ensure there are no errors in the file.
-
Code Example
Here's a simplified example using Python and common libraries:
import pandas as pd import numpy as np from sklearn.model_selection import train_test_split, StratifiedKFold from sklearn.metrics import recall_score from sklearn.preprocessing import LabelEncoder from lightgbm import LGBMClassifier # Step 1: Load Data train = pd.read_csv('train.tsv', sep='\t') test = pd.read_csv('test.tsv', sep='\t') # Step 2: Preprocessing # Separate features and target X = train.drop('target', axis=1) y = train['target'] # Combine train and test for consistent preprocessing combined = pd.concat([X, test], axis=0) # Handle missing values, encode categorical variables, etc. # For illustration, let's encode categorical variables using Label Encoding categorical_cols = combined.select_dtypes(include=['object', 'category']).columns for col in categorical_cols: le = LabelEncoder() combined[col] = le.fit_transform(combined[col].astype(str)) # Split back into train and test X_processed = combined.iloc[:len(train)] X_test_processed = combined.iloc[len(train):] # Step 3: Model Training # Split data for validation X_train, X_val, y_train, y_val = train_test_split(X_processed, y, test_size=0.2, stratify=y, random_state=42) # Initialize model model = LGBMClassifier(class_weight='balanced', random_state=42) # Train model model.fit(X_train, y_train) # Step 4: Evaluation # Predict probabilities on validation set val_preds = model.predict_proba(X_val)[:, 1] # Calculate Recall@Top15% def recall_at_k(y_true, y_scores, k): # Number of positives in the validation set num_positives = sum(y_true) # Number of samples to select k_samples = int(len(y_scores) * k) # Get indices of the top k_scores top_k_indices = np.argsort(y_scores)[::-1][:k_samples] # Number of true positives in top k num_true_positives_in_k = sum(y_true.iloc[top_k_indices]) return num_true_positives_in_k / num_positives recall_top_15 = recall_at_k(y_val.reset_index(drop=True), val_preds, 0.15) print(f'Recall@Top15%: {recall_top_15:.4f}') # Step 5: Predict on Test Data test_preds = model.predict_proba(X_test_processed)[:, 1] # Prepare submission submission = pd.DataFrame({ 'client_id': test.index, # Or the appropriate identifier 'prediction': test_preds }) submission.to_csv('submission.csv', index=False)Tips for Improving Recall@Top15%
-
Threshold Adjustment:
- Since we're interested in the top 15%, focus on accurately ranking the positives higher.
-
Use of Ranking Algorithms:
- Consider using algorithms designed for ranking tasks (e.g., XGBoost's rank objective).
-
Feature Selection:
- Identify features that have a higher impact on the target variable.
- Remove noisy or irrelevant features that might confuse the model.
-
Ensemble Methods:
- Combine predictions from multiple models to improve performance.
-
Domain Knowledge:
- If possible, incorporate business insights or domain knowledge into feature engineering.
Important Considerations
-
Data Privacy:
- Ensure that you're compliant with any data privacy regulations when handling client data.
-
Validation Strategy:
- Make sure that your validation strategy mimics the test scenario as closely as possible.
-
Documentation:
- Keep track of the preprocessing steps and model parameters for reproducibility.
Additional Resources
-
Understanding Recall and Precision:
-
Imbalanced Classification:
-
Model Evaluation Metrics:
Conclusion
By following the steps outlined above and focusing on optimizing the Recall@Top15% metric, you'll increase the likelihood of correctly identifying clients who are most likely to register a business after communication with the bank. Remember to iterate on your model, use cross-validation, and potentially explore different algorithms and feature transformations to improve your results.
- Training Data (