Parallel Processing Template
Process multiple tasks simultaneously instead of sequentially. Dramatically reduce execution time for workflows that handle multiple independent items.
Time to build: 20 minutes Difficulty: Intermediate Perfect for: Batch processing, multi-source data fetching, concurrent API calls, image processing
What It Does
Input (multiple items) → Process all simultaneously → Combine results → Output
Instead of processing items one-by-one, process them all at once for 5-10x speedup.
Step-by-Step
1. Create Workflow
- New workflow → Name it "Parallel Lead Enrichment"
- Webhook trigger with array of leads
2. Add Loop with Parallel Execution
- Drag Loop Block
- Connect Webhook to Loop
Items: <webhook.body.leads>
Variable name: lead
Execution: Parallel (not sequential)
Max concurrent: 10
3. Enrich Lead with AI (Inside Loop)
- Drag Agent Block inside Loop
Model: GPT-4o-mini (fast)
System Prompt:
"Analyze this lead and provide enrichment: company size estimate, likely budget, urgency level, qualification score (1-10)."
User Prompt:
"Lead: <lead.name>
Company: <lead.company>
Title: <lead.title>
Email: <lead.email>
Message: <lead.message>"
4. Fetch Company Data (Inside Loop)
- Drag HTTP Request Block in parallel with Agent
Method: GET
URL: https://api.clearbit.com/v2/companies/find?domain=<lead.domain>
Headers:
Authorization: Bearer <env.CLEARBIT_API_KEY>
5. Check LinkedIn (Inside Loop)
- Drag another HTTP Request Block in parallel
Method: GET
URL: https://api.linkedin.com/v2/people?email=<lead.email>
Headers:
Authorization: Bearer <env.LINKEDIN_API_KEY>
6. Combine Data (Inside Loop)
- Drag Agent Block to merge results
Model: GPT-4o-mini
System Prompt:
"Combine this data into a complete lead profile with a qualification recommendation."
User Prompt:
"AI Analysis: <agent_analysis.content>
Company Data: <http_clearbit.body>
LinkedIn Data: <http_linkedin.body>"
7. Store Result (Inside Loop)
- Drag Database Query Block
INSERT INTO enriched_leads (
lead_id,
name,
company,
title,
email,
company_size,
budget_estimate,
urgency_level,
qualification_score,
linkedin_url,
enriched_at
) VALUES (
'<lead.id>',
'<lead.name>',
'<lead.company>',
'<lead.title>',
'<lead.email>',
'<http_clearbit.body.metrics.employees>',
'<agent_analysis.budget_estimate>',
'<agent_analysis.urgency_level>',
'<agent_analysis.qualification_score>',
'<http_linkedin.body.profile_url>',
NOW()
)
8. Wait for All to Complete
- Loop automatically waits for all parallel executions
- Collects all results when complete
9. Create Summary (After Loop)
- Drag Agent Block after Loop
Model: GPT-4o-mini
System Prompt:
"Summarize these enriched leads. Highlight top prospects."
User Prompt:
"Processed <loop.count> leads
Results: <loop.results>"
10. Send to Sales Team
- Drag Slack Tool
Channel: #sales-leads
Message:
"🎯 Lead Enrichment Complete
<agent_summary.content>
Top prospects: <loop.top_leads>
View dashboard: https://crm.yourapp.com/leads"
Performance Comparison
Sequential Processing (Old Way)
Lead 1: 3 seconds
Lead 2: 3 seconds
Lead 3: 3 seconds
Lead 4: 3 seconds
Lead 5: 3 seconds
Total: 15 seconds
Parallel Processing (Klyntos Way)
Lead 1, 2, 3, 4, 5: All process simultaneously
Total: 3 seconds
Speedup: 5x faster
Batch of 100 Leads
Sequential: 5 minutes
Parallel (10 concurrent): 30 seconds
Speedup: 10x faster
Real-World Examples
Image Processing
- Process 100 images simultaneously
- Apply AI enhancement to each
- 10x faster than sequential
Multi-Source Data Gathering
- Fetch from 5 different APIs simultaneously
- Combine results
- Single response in API call time (not 5x)
Email Campaign Personalization
- Personalize 1,000 emails concurrently
- Each gets unique AI-generated content
- Complete in minutes, not hours
Document Analysis
- Analyze 50 contracts simultaneously
- Extract key terms from each
- Generate comparison report
Advanced Patterns
Parallel + Map + Reduce
- Map: Process each item independently (parallel)
- Reduce: Combine all results into final output
Nested Parallel Processing
- Outer loop: Process companies (parallel)
- Inner loop: Process employees per company (parallel)
- 2-dimensional parallelism
Adaptive Concurrency
- Start with max_concurrent: 5
- Monitor failure rate
- Increase to 10 if success rate > 95%
- Decrease to 2 if failures occur
Priority-Based Processing
- Sort items by priority
- Process high-priority items first
- Process remaining in parallel
Concurrency Limits
Recommended Settings
API Rate Limits:
Max concurrent: 5-10
- Most APIs handle this well
- Reduces 429 rate limit errors
AI Processing:
Max concurrent: 20-50
- AI providers handle high concurrency
- Limited by your account limits
Database Operations:
Max concurrent: 10-20
- Depends on connection pool
- Monitor database load
Mixed Workload:
Max concurrent: 10
- Balance between speed and stability
- Adjust based on monitoring
Error Handling in Parallel
Fail-Safe Strategy
- One failure doesn't stop others
- Failed items collected separately
- Retry failed items after main batch
Circuit Breaker
- If failure rate > 20%, pause processing
- Investigate issue
- Resume after fix
Partial Success
- Return successful results immediately
- Process failures in background
- Notify about failures separately
Enhancements
Add Progress Tracking
- Store processing state in database
- Show progress bar to users
- Resume interrupted batches
Add Result Caching
- Cache results for duplicate inputs
- Skip processing if already done
- Reduces cost and time
Add Dynamic Batching
- Split large batches into smaller chunks
- Process chunks in waves
- Better error isolation
Add Priority Queue
- High priority items processed first
- Low priority during off-peak hours
- Better resource utilization
Cost
Cost is same as sequential, but faster:
Example: 100 leads enriched
- Sequential: 5 minutes, ~$2
- Parallel: 30 seconds, ~$2
- Same cost, 10x faster execution
Value of speed:
- Faster user responses
- Real-time processing possible
- Better user experience
- Higher throughput
Next Step
Combine with Error Handling to make parallel workflows resilient to failures!