Azure Blob Storage integration

Overview

The Azure Blob Storage integration provides secure and scalable blob storage capabilities for log ingestion and data processing. This integration leverages Azure Event Grid and Storage Queues to enable real-time processing of blob storage events with robust authentication and access control mechanisms.

Prerequisites

• Azure subscription with appropriate permissions
• Microsoft Entra ID (Azure AD) access
• Resource group creation permissions
• Storage account management access
• Event Grid system topic creation permissions
• Bloo platform access credentials

Configuration Steps

1. Register Resource Providers

1. Sign in to the Azure Portal
2. Navigate to Subscriptions
3. Select your target subscription
4. Click Resource providers
5. Search for and locate Microsoft.EventGrid and Microsoft.Storage
6. Ensure both providers show Registered status
7. If not registered, click Register for each provider

2. Create Resource Group and Storage Account

Step 1: Create Resource Group

1. In Azure Console, navigate to Subscriptions
2. Select your target subscription
3. Click Resource groups
4. Click + Create
5. Provide values for:
   • Name: Enter a descriptive name (save this for later)
   • Region: Select appropriate region
6. Click Review and create
7. Click Create

Step 2: Create Storage Account

1. Click on your newly created resource group
2. Click Create
3. Search for "storage account" and click Create
4. On the Create a storage account page:
   • Storage account name: Enter unique name
   • Region: Select same region as resource group
5. Click Review
6. Click Create

3. Add App Registration and Client Secret

1. In the top search bar, search for "Microsoft Entra ID"
2. Click Microsoft Entra ID
3. Click + Add → App registration
4. Configure the app registration:
   • Name: Enter descriptive name
   • Click Register
   • Securely copy and store the Application (client) ID
5. Click on your newly registered app
6. On the right side, click Add a certificate or secret
7. Click + New client secret
8. Configure the secret:
   • Description: Enter descriptive name
   • Click Add
   • Securely copy and store the Client Secret value

4. Create Queue and Add Permission

1. Navigate to your newly created storage account
2. In the left navigation bar, select Queues
3. Click + Queue to create new queue
4. Configure the queue:
   • Name: Enter descriptive name (save this for later)
   • Click OK
5. Click on your newly created queue
6. In the left navigation bar, click Access Control (IAM)
7. Click + Add → Add Role Assignment
8. Search for "Storage Queue Data Message Processor" and select it
9. Click Members tab
10. Click + Select Members
11. Search for your registered app name and click Select

5. Create System Topic and Event Subscription

Step 1: Create System Topic

1. Search for "Event Grid System Topics"
2. Click + Create
3. On the Create Event Grid System Topic page:
   • Topic Types: Select Storage Accounts (Blob & GPv2)
   • Subscription: Select your subscription
   • Resource Group: Select your resource group
   • Resource: Select your storage account
   • Name: Enter descriptive topic name
4. Click Review + create
5. Click Create

Step 2: Create Event Subscription

1. Navigate back to your storage account
2. In the left navigation bar, click Events
3. Click + Event Subscription
4. On the Create Event Subscription page:
   • Name: Enter descriptive name
   • Event Types: Select Blob Created
   • Endpoint Type: Select Storage Queue
   • Endpoint: Select your queue
5. Click Create

6. Create Container and Add Permission

1. Navigate to your storage account
2. In the left navigation bar, select Containers
3. Click + Container to create new container
4. Configure the container:
   • Name: Enter descriptive name (save this for later)
   • Click Create
5. Click on your newly created container
6. In the left navigation bar, click Access Control (IAM)
7. Click + Add → Add Role Assignment
8. Search for "Storage Blob Data Reader" and select it
9. Click Members tab
10. Click + Select Members
11. Search for your registered app name and click Select
12. Click Review + Assign

7. Get Azure Configuration Values

1. Tenant ID: Found on Azure Console's Microsoft Entra ID home page
2. Client ID: Application (client) ID from Step 3
3. Storage Account Name: Name from Step 2
4. Storage Queue Name: Name from Step 4
5. Client Secret: Client secret value from Step 3

Bloo Platform Configuration

Connector Setup

1. Log in to your Bloo platform
2. Navigate to Integrations > Connectors
3. Click Add New Connector
4. Configure the following fields:

Field Name	Description
Connector Name	Enter a descriptive name for your connector
Tenant ID	Enter Azure Tenant ID
Client ID	Enter the Azure Client ID
Client Secret	Enter the Azure Client Secret
Storage Account Name	Enter the Azure Storage Account Name
Queue Name	Enter the Azure Queue Name
Prefix (Optional)	Enter the Prefix path from which you want to pull data from the container
Delete Unmatched Prefix	Choose True or False to delete Unmatched Prefix messages from the Azure queue

1. Click Save after entering all required details
2. Click Test Connection to test the configuration
3. Verify Connection successful message appears with timestamp

Validation

1. Navigate to Collection Status screen
2. Verify your Azure Blob Storage connector appears with Active status
3. Confirm data ingestion is working by checking the data flow

Features

• Event Grid Integration: Real-time event processing using Azure Event Grid
• Storage Queue Processing: Reliable message processing with Azure Storage Queues
• Blob Data Ingestion: Direct ingestion from Azure Blob Storage containers
• App Registration Authentication: Secure authentication using Azure AD app registrations
• Role-Based Access Control: Granular permissions using Azure RBAC
• Container Management: Flexible container and blob management
• Prefix Filtering: Optional prefix-based data filtering
• Real-Time Processing: Immediate processing of blob creation events

Troubleshooting

Common Issues

1. Connection Failed

   • Verify Tenant ID, Client ID, and Client Secret are correct
   • Check app registration permissions
   • Ensure resource providers are registered

2. No Data Ingestion

   • Verify Event Grid system topic is created
   • Check event subscription configuration
   • Ensure blob creation events are being generated

3. Permission Errors

   • Verify app registration has proper role assignments
   • Check Storage Queue Data Message Processor role
   • Ensure Storage Blob Data Reader role is assigned

4. Queue Processing Issues

   • Verify queue exists and is accessible
   • Check queue permissions
   • Ensure event subscription is properly configured

Support

For additional support and troubleshooting:

• Check Azure Blob Storage documentation
• Review Azure Event Grid documentation
• Check Bloo platform logs
• Contact Bloo support team

Security Considerations

• Use managed identities when possible
• Implement proper access controls
• Enable encryption in transit and at rest
• Regularly rotate client secrets
• Monitor access patterns and usage
• Use least privilege principle for role assignments

Best Practices

1. Resource Organization: Use separate resource groups for different environments
2. Naming Conventions: Follow consistent naming patterns
3. Access Control: Implement proper RBAC policies
4. Monitoring: Set up proper monitoring and alerting
5. Cost Optimization: Monitor usage and optimize costs regularly
6. Security: Follow Azure security best practices
7. Backup: Implement proper backup and recovery procedures
8. Documentation: Maintain up-to-date configuration documentation