Automated Syndication System for 25+ Real Estate Portals

Full ETL pipeline extracting real estate listings from PIM Akeneo, transforming them into each partner's required format (XML, CSV, JSON), and delivering to SeLoger, LeBonCoin, BienIci, LogicImmo and more - 4 years of continuous operation.

January 2019 - 2023

~4 years

Technical Lead then Project Manager - Export Distribution

PHPSymfonyAkeneo PIM v2REST APIXMLCSVJSONFTP/SFTPGitLab CIDockerKubernetesMySQL

Automated Syndication System for 25+ Real Estate Portals

January 2019 - 2023

~4 years

Technical Lead then Project Manager - Export Distribution

PHPSymfonyAkeneo PIM v2REST APIXMLCSVJSONFTP/SFTPGitLab CIDockerKubernetesMySQL

Partner Portals

27+

Migrated, integrated and maintained

Export Formats

XML, CSV, JSON

Project Duration

~4 years

Continuous evolution

Daily Volume

~2,000/day

Listings processed across all portals

Availability

99.5%+

Over 4 years of continuous operation

GitLab Branches

12+

Features and hotfixes documented

Presentation

Project definition and scope

The "Export Ligneurs" system is the automated real estate listing distribution engine of the Groupe Pichet. It extracts program and lot data from the PIM Akeneo, transforms it into the specific format required by each partner (XML, CSV, or JSON), and automatically exports it to real estate distribution platforms.

The system serves as the critical link between the company's product data and its commercial visibility: every property listing published on major French real estate portals (SeLoger, LeBonCoin, BienIci, LogicImmo...) passes through this pipeline. Any interruption or data inconsistency directly translates into lost leads and missed sales opportunities.

As the sole technical owner of this system, I was responsible for all architecture decisions, development, deployment, monitoring, and incident response - with full accountability for a pipeline feeding an estimated 400K euros/month in lead acquisition.

Nature

Automated ETL pipeline (Extract-Transform-Load) for multi-channel real estate ad distribution

Domain

Real Estate / PropTech - B2B (internal teams, partner portals) and B2C (indirect, end buyers)

Functional Scope

Automated data extraction from PIM Akeneo v2 REST API
Per-partner format transformation (XML, CSV, JSON)
FTP/SFTP automated delivery to 25+ partner platforms
Image management with multi-format adaptation (4/3, 16/9, panoramic, square, portal-specific ratios)
Property typology mapping (apartment, house, duplex, triplex, studio, T1-T5+)
Execution monitoring with email alerts and SOFT Monitor integration
Individual partner activation/deactivation capability
SKU matching algorithm for real vs. manually-created PIM programs

System Architecture

Export Ligneurs - System Architecture Overview

Technology Choices & Rationale

PHP / Symfony

Consistent with Groupe Pichet backend ecosystem. Symfony Console provided a solid framework for scheduled batch command execution.

Akeneo PIM v2

Strategic company choice for product catalog management. Its REST API provided structured access to all program and lot data with versioned endpoints.

Docker / Kubernetes

Each export job isolated in its own container, preventing resource conflicts between partner modules. K8s on AWS EKS handled scheduling and auto-recovery of failed jobs.

GitLab CI

Automated the build-test-deploy cycle for each partner module independently, allowing targeted deployments without impacting other active feeds.

Objectives, Context, Stakes & Risks

Strategic vision and constraints

Objectives

1Migrate all export feeds from legacy PIM v1.4 to the new PIM v2 Akeneo
2Execute migration partner by partner with business validation at each step
3Verify data consistency between source PIM and feeds sent to portals
4Handle each portal's specificities (image formats, typologies, required fields)
5Automate feed supervision (error alerts, execution reports)

Context

The project was initiated during the knowledge transfer from Andoni L. (Kariba) in January 2019. The existing system ran on the legacy PIM v1.4 and needed to be fully migrated to PIM v2 Akeneo while maintaining continuous service to all partner portals.

The migration had to be performed portal by portal - each with its own format specifications, required fields, image constraints, and property typology mappings - making it impossible to execute as a single "big bang" migration. Each partner required individual validation by the business teams before going live.

The system was embedded in a larger data ecosystem: upstream data came from Qualiac, G2P, and Oracle ERPs feeding the PIM, while downstream the feeds connected to 75+ lead suppliers generating an estimated 1 lead every 2 seconds across all portals.

Stakes

The partner portals (SeLoger, LeBonCoin, BienIci...) are the primary prospect acquisition channels for the Groupe Pichet, feeding an estimated 400K euros/month lead pipeline. Any interruption or error in the feeds directly translates into lost leads and reduced commercial pipeline. With 25+ partners to migrate individually, the project required sustained attention over multiple years while maintaining zero downtime on active feeds.

Risks

Data Inconsistency

Risk of publishing incorrect prices, wrong images, or missing properties on partner portals - directly impacting buyer trust and commercial results.

Service Interruption

Any feed failure means properties disappear from partner portals, causing immediate lead loss for the commercial teams.

Format Divergence

Each portal has unique requirements (image ratios, typology codes, required fields) - a generic approach was impossible.

API Instability

PIM Akeneo API connection issues could block all exports simultaneously, requiring solid error handling and retry logic.

Key Architecture Decisions

Modular per-partner architecture

Decision: One isolated module per partner portal instead of a generic configurable engine

Rationale: Each portal had unique constraints (image ratios, field mapping, typology codes). A generic engine would have become a monolithic configuration nightmare. The modular approach provided fault isolation: a bug in the BienIci module could not impact SeLoger exports. It also allowed independent deployment and testing of each partner feed.

Progressive migration over big-bang

Decision: Portal-by-portal migration with business validation at each step, spread over 4 years

Rationale: With 25+ partners directly feeding commercial lead acquisition, a simultaneous migration carried unacceptable risk. Each portal had unique specs to verify. The incremental approach limited blast radius to one partner at a time and allowed immediate rollback if issues arose.

ETL batch processing over real-time streaming

Decision: Scheduled batch exports via CRON jobs rather than event-driven real-time publishing

Rationale: Partner portals consumed data via FTP/SFTP file drops, not webhooks or APIs. They processed files on their own schedule (typically once or twice daily). Real-time processing would have added complexity without benefit given the downstream consumption pattern.

Multi-format image pre-generation

Decision: Pre-generate all image format variants (4/3, 16/9, square, panoramic) centrally rather than on-demand per partner

Rationale: Partner portals had strict image constraints and no resizing capability on their end. Pre-generation at the source verified compliance upfront and cached results, avoiding redundant processing of the same image across multiple portals.

ETL Data Pipeline

Extract-Transform-Load pipeline for partner feed generation

The Steps - What I Did

Chronological progression of the project

Phase 1

Knowledge Transfer & Initial Migration

January 2019

Before writing any code, I conducted a full audit of the existing system: mapped all 25+ partner specifications, assessed the technical debt, and defined a migration priority matrix based on commercial impact. High-traffic portals (SeLoger, LogicImmo) were migrated first to secure the majority of lead volume.

Managed the complete knowledge transfer from Andoni L. (Kariba) - became the sole technical owner within 2 weeks
Obtained Master access on the Export Ligneurs GitLab repository
Migrated the first batch of portals: SeLoger Neuf, LogicImmo, TULN, Paru Vendu, Mister Bell, Explorimmoneuf
Defined validation criteria with Leslie A. and Gaetan B. - established the acceptance checklist reused for all subsequent migrations

Phase 2

Feature Development & New Integrations

June - September 2019

Prioritized BienIci integration based on its rapid market share growth in the French real estate sector. Chose to build partner-specific image adaptation (16/9 to 4/3) rather than a generic resizer, because each portal's constraints were too different to abstract into a single solution.

Developed feature/add-hermes-lots - adding Hermes lots to exports with CI pipeline corrections
Integrated BienIci portal (feature/add-bienici-in-master) with custom image adaptation
Adapted ImmoNeuf feed for 16/9 to 4/3 image format conversion
Debugged SeLoger and Knock feeds (hotfix/debug-seloger, hotfix/debug-knock)

Phase 3

Stabilization & Critical Fixes

January 2020

Pricing errors required same-day resolution given direct revenue impact. I implemented defensive coding patterns: circuit breakers on PIM API calls, idempotent retry logic with exponential backoff, and structured execution logging to reduce mean time to diagnosis from hours to minutes.

Fixed pricing errors in exports (hotfix/debug-prices) - resolved within hours to prevent commercial impact
Resolved PIM API connection issues (hotfix/debug-pim-api-connexion)
Implemented circuit breaker pattern and exponential backoff retry logic for API calls
Added structured logging for export execution tracking, reducing incident diagnosis time

Phase 4

New Partners & Continuous Evolution

June 2020 - 2023

Transitioned the project from migration mode to platform mode. Established a standardized onboarding process for new partners (specification analysis, module scaffolding, validation checklist), reducing integration time from weeks to days for the Investimeo and BienIci builds.

Created Investimeo integration from scratch (June-October 2020) using the standardized onboarding process
Created BienIci feed (October 2020) - new portal integration
Updated Criteo retargeting feed (October 2020)
Deactivated Marketshot partner (October 2020) - clean module removal without affecting other feeds
Evolved NEEDOCS flux for PI lignage (JIRA LIGNEURS-74, February 2022)
Resolved BienIci automated export anomaly (KESD-58153, July 2022)
Fixed Green Valley missing lots in feeds (KESD-72984, 2023)

Cumulative Partner Migration Timeline

Partner Migration Process

Step-by-step process for each partner migration

Actors & Interactions

Collaborative ecosystem

As the sole technical owner, I coordinated directly with business stakeholders, external vendors, and partner portals. Each migration involved defining acceptance criteria, piloting validation cycles, and making go/no-go decisions for production deployment. This required translating technical constraints into business terms and vice versa.

Andoni L.

Predecessor (Kariba)

Managed the complete knowledge transfer. Achieved autonomous operation of the full export system within 2 weeks, becoming the sole technical reference for the entire scope.

Gaetan B.

Business referent

Defined and enforced acceptance criteria together for each migration: data accuracy, image compliance, typology mapping. Established a reusable validation checklist.

Leslie A.

Business referent

Led the functional acceptance process, coordinating between technical fixes and business priorities to maintain migration velocity.

Franck C.

Manager (N+1)

Reported migration progress, risk assessment, and resource needs. Provided technical recommendations for vendor coordination decisions.

Sebastien B.

Kariba team

Coordinated production deployment scheduling. Established a deployment protocol: preprod validation, business sign-off, prod deployment, 24h monitoring window.

Results

Impact for me and for the company

For Me

Full technical ownership of a business-critical system directly impacting revenue generation - primary lead acquisition channel for the company
Autonomous architecture decisions on a high-value scope with full accountability for system reliability and data accuracy
Proven ability to pilot a 4-year technical project with multiple stakeholders: business teams, external vendors, and 25+ partner portals
End-to-end system lifecycle management: architecture design, development, deployment, monitoring, incident response, and continuous evolution
Cross-functional leadership: defined validation processes, coordinated business and technical teams, established operational procedures for partner onboarding

For the Company

25+ partner portals migrated from PIM v1.4 to PIM v2 Akeneo with zero service interruption - protecting an estimated 400K euros/month lead acquisition pipeline
2 new partner integrations built from scratch (BienIci, Investimeo), expanding the distribution network by ~8%
~2,000 listings processed daily across 27+ portals, feeding the primary commercial lead generation channel
99.5%+ system availability over 4 years of continuous operation, with average incident resolution under 4 hours
Standardized property typology management across all partner feeds, reducing data inconsistency reports from business teams

Partner Type Distribution

Export Format Distribution

Migration Status Breakdown

Property Types Handled

Project Aftermath

What happened after delivery

Immediate aftermath: After the initial migration wave (2019), the system entered a continuous maintenance and evolution phase. New partners were added as business needs arose, existing feeds were updated to match evolving portal specifications, and anomalies were resolved as they were detected.

Medium term: The system proved its resilience over 4 years of continuous operation, handling format changes from partners (BienIci specification updates, SeLoger field additions) and internal data model evolutions (new property types, price structure changes).

Long-term perspective: The export system became a foundational piece of infrastructure at Groupe Pichet, directly feeding the commercial pipeline. The architectural choices - individual partner modules, solid error handling, automated monitoring - allowed the system to scale from the initial batch of portals to 27+ without requiring fundamental redesign. The modular architecture I chose made it possible for any developer to add a new partner integration by following the established patterns, without needing deep knowledge of the entire system.

Technical Effort Distribution

Critical Reflection

Honest retrospective analysis

What Worked Well

The portal-by-portal migration strategy was the right call - it minimized risk and allowed business validation at each step, with immediate rollback capability
Solid error handling and email alerting caught issues before they impacted business results - average detection time under 30 minutes
The modular architecture (one module per partner) made it easy to add, modify, or deactivate individual feeds without side effects on other partners
Establishing a standardized onboarding process for new partners reduced integration time from weeks to days

What Could Have Been Better

A centralized dashboard for feed status monitoring would have reduced the time spent checking individual email alerts
Automated integration tests for each partner format would have caught format regressions earlier
Better documentation of each partner's specific requirements would have accelerated onboarding of new team members

If I Had to Redo It Today

I would implement a partner specification registry from day one: a structured database of each portal's format requirements, field mappings, and image constraints. This would have cut onboarding time for new partners by at least 50%.
I would add automated integration tests generating sample exports and validating them against each partner's schema before deployment. The manual verification cycles added 1-2 days per migration.
I would build a centralized monitoring dashboard with real-time feed status instead of relying on email alerts. The context-switching cost of checking individual emails was significant over 4 years.
I would push for an event-driven notification layer on top of the batch ETL, to alert the business team proactively when a feed completes or fails, rather than waiting for periodic manual checks.

Key Lessons Learned

In multi-partner systems, there is no "one size fits all" - each integration has unique constraints that must be respected
Long-running projects require a maintenance mindset from day one, not just a "build and ship" approach
Business validation at each migration step is non-negotiable when the system directly impacts revenue generation
As a sole technical owner, the most valuable investment is in system observability - the ability to diagnose issues quickly matters more than preventing every possible failure