---
title: "Flux - Supplier Data Synchronization"
description: "An internal ETL platform automating the daily synchronization of product data (stock, prices, catalogs) from 30+ European suppliers via FTP, HTTP and REST APIs into the European Sourcing marketplace."
locale: "en"
canonical: "https://portfolio.josedacosta.info/en/achievements/flux-synchronisation-fournisseurs"
source: "https://portfolio.josedacosta.info/en/achievements/flux-synchronisation-fournisseurs.md"
html_source: "https://portfolio.josedacosta.info/en/achievements/flux-synchronisation-fournisseurs"
author: "José DA COSTA"
date: "2016"
type: "achievement"
slug: "flux-synchronisation-fournisseurs"
tags: ["PHP 5.3+", "Symfony 2.4", "MySQL", "Doctrine DBAL", "Twig", "Bootstrap", "jQuery", "FTP", "REST API", "CSV", "XML", "JSON", "Elasticsearch", "Cron"]
generated_at: "2026-04-23T15:48:46.353Z"
---

# Flux - Supplier Data Synchronization

An internal ETL platform automating the daily synchronization of product data (stock, prices, catalogs) from 30+ European suppliers via FTP, HTTP and REST APIs into the European Sourcing marketplace.

**Date:** 2013 - 2016  
**Duration:** ~3 years  
**Role:** Senior Software Engineer - ETL & Data Integration  
**Technologies:** PHP 5.3+, Symfony 2.4, MySQL, Doctrine DBAL, Twig, Bootstrap, jQuery, FTP, REST API, CSV, XML, JSON, Elasticsearch, Cron

### Key Metrics

- Integrated Suppliers: **-** - 8+ European countries
- Connector Classes: **-** - Strategy pattern (Lib/*.php)
- Data Formats: **-** - CSV, XML, JSON, XLS, TSV, TXT, GZ
- Database Size: **-** - Full SQL dump (March 2019)

## Presentation

_The nerve center of European supplier data integration_

- Flux (v1) and FluxV2 are internal web applications developed for European Sourcing, a company specialized in sourcing promotional and advertising products across Europe. These applications form the backbone of product data synchronization between suppliers and the European Sourcing platform.
- The platform operates in the B2B e-commerce / promotional products marketplace domain. European Sourcing acts as a catalog aggregator for promotional product suppliers (goodies, textiles, accessories, office supplies, etc.) targeting European resellers. The company collects, normalizes and redistributes product data (stock levels, pricing, docs, technical specifications) from dozens of suppliers to its online platform, serving approximately 160 reseller websites.

**Arch Caption:** Overall architecture of the Flux system within the European Sourcing ecosystem

## Objectives, Context, Stakes & Risks

_Understanding the strategic vision behind the data pipeline_

### Context

- Part of a 20+ application ecosystem under *.europeansourcing.com (extranet, API, search engine, export, statistics, translation, etc.)
- Development under the medialeads GitHub organization (~8 developers) based in Bordeaux, France
- Shared MySQL master/slave database (~15 GB) with all other platform projects
- Multiple environments: developer (local), staging (es-recette.com), production (OVH)

## The Steps - What I Did

_A concrete, phase-by-phase journey through the build_

**Etl Caption:** Full ETL pipeline showing extraction, transformation and loading steps

**Data Model Caption:** Database tables used by Flux for synchronization configuration and reporting

**Supplier Caption:** Complete map of 30+ integrated suppliers, classified by protocol (FTP, HTTP, API REST) with country of origin and data format

**Infra Caption:** Deployment environments from local development to OVH production

## The Actors - Interactions

_A small, focused team within a larger ecosystem_

### Development Team

The Flux project was primarily developed by 2 developers within the medialeads organization (~8 people total). Thomas C. was the lead developer, implementing the majority of supplier connectors (28 commits, 65% of the codebase). I contributed 15 commits (35%) including the initial commit, SVN migration, and key connectors for BIC France, PF Concept and Midocean.

### External Partners

The project interacted directly with 30+ European supplier APIs and data feeds - each with their own protocols, formats, authentication methods and technical contacts. This required constant adaptation and communication with external technical teams.

## Results

_Measurable impact for the business and personal growth_

### Business Impact

- Daily automated synchronization guarantees up-to-date product data for resellers on europeansourcing.com → improved conversion, reduced order errors
- 6,120 product variants updated in a single 45-second execution (Anda supplier, August 2019)
- ~100 daily executions across all suppliers, ensuring continuous data freshness
- Massive catalog coverage: database of ~15 GB serving 160+ reseller websites
- Pan-European geographic reach: suppliers from 8+ countries integrated seamlessly

### Personal Growth

- Deep expertise in heterogeneous data integration (ETL) across multiple protocols and formats
- Mastery of the Strategy design pattern applied to real-world supplier connector architecture
- Understanding of B2B e-commerce catalog management at European scale
- Experience with MySQL master/slave architecture and large-scale data operations
- Practical knowledge of Symfony 2 CLI commands, services and event system

## Project Aftermath

_Beyond delivery - lifecycle and évolution_

## Critical Reflection

_Honest retrospective on strengths, weaknesses and lessons learned_