Open Food Facts — ODbL alterations notice

Source

Open Food Facts publishes a full CSV export (en.openfoodfacts.org.products.csv.gz) and a daily incremental delta feed (newline-delimited JSON, listed at static.openfoodfacts.org/data/delta/index.txt). Nyamly's backend downloads both directly from Open Food Facts; no other party's copy is used.

What Nyamly does to the data

This describes the actual ingest code (backend/api/build_off_db.py for the weekly full rebuild, backend/api/apply_off_delta.py for daily deltas), field by field, so a third party could reproduce the same derivative from a fresh Open Food Facts export.

1. Field selection

Only a fixed subset of Open Food Facts' CSV columns is read: code, product_name, brands, quantity, countries_tags, categories_tags, the core macro columns (energy-kcal_100g, proteins_100g, fat_100g, carbohydrates_100g, fiber_100g, sugars_100g, sodium_100g, saturated-fat_100g), fruits-vegetables-legumes_100g, additives_tags, ingredients_text, nova_group, nutriscore_grade, allergens_tags, ingredients_analysis_tags, popularity signals (unique_scans_n, popularity_tags, completeness), and the front-of-pack photo URLs (image_small_url, falling back to image_url). Ingredient-photo and nutrition-photo images, and every other CSV column, are dropped and never stored.

2. Row filtering

A row is kept only if its code field, after stripping non-digit characters, matches an 8–14 digit barcode. Rows without a plausible barcode are discarded (not corrected or guessed). Every other field is read defensively: a missing or truncated column on an otherwise-valid row yields a null for that field rather than dropping the row.

3. Normalization

• Numbers: macro fields are parsed as floats; unparseable or empty cells become null (never zero). Sodium is converted from grams to milligrams.
• Categories (categories_tags): only tags carrying Open Food Facts' en: taxonomy prefix are kept, and only if the remainder is a canonical lowercase-ASCII slug (letters, digits, single hyphens). Free text OFF tagged en: as a fallback despite it being untranslated, non-English text (e.g. literal French words) is filtered out, regardless of its prefix.
• Tag lists (additives_tags, allergens_tags, ingredients_analysis_tags): the en: prefix is stripped from each entry; additive codes are upper-cased. No canonical-slug filter is applied to these (unlike categories).
• Nutri-Score: Open Food Facts' "unknown" and "not-applicable" sentinel strings are converted to a database null — never stored as if they were a grade, and never displayed as if the product had been assessed and passed.
• Fruit/vegetable/nut percentage: stored under its own provenance tag fvl_export rather than being conflated with the UK Nutrient Profiling Model's own fruit/veg/nut measure, because Open Food Facts' current export folds legumes into this figure and the two are not the same quantity.
• Empty strings are consistently normalized to null across every field, so "not provided" is never stored in a form indistinguishable from "provided and empty".

4. Deduplication and merging

Ingest itself keeps one row per barcode (INSERT OR IGNORE on the primary key code — the first row seen for a barcode in the export wins, later duplicates in the same file are dropped). Cross-source merging and near-duplicate grouping happen later, at query time, in the API layer rather than in the stored derivative:

• Rows sharing a barcode across different sources (Open Food Facts, USDA Branded, …) have missing nutrient fields filled in from the other source, never overwritten, and the filled field's source and licence are recorded per field.
• Rows that are not exact barcode duplicates but represent close variants (same normalized name and brand — different pack size, region, or minor formulation) are grouped behind one representative row for display, with a variant count; per-barcode fields such as additives, allergens, NOVA group and Nutri-Score are taken only from the representative's own row, never unioned across variants.

See docs/DECISIONS.md, entries dated 2026-09-02 ("cross-source field-merge") and 2026-09-06 ("variant grouping"), for the full detail of this stage.

5. Ranking (search only, not stored)

Search results are ordered by text relevance first, then by an Open Food Facts popularity signal (unique_scans_n), bucketed into order-of-magnitude tiers. This reorders how results are presented; it does not alter or drop any stored field.

6. Daily deltas

The incremental delta feed carries a different (nested, newer) Open Food Facts document shape than the CSV export. apply_off_delta.py re-derives the same normalized fields from that shape using the same helper functions as the full rebuild (normalize_categories, normalize_tag_list, numeric parsing), so a product updated via delta ends up in the same normalized form as one loaded by the weekly full rebuild. Product images are deliberately left untouched by deltas: the delta document has no flat image URL, only a nested structure that would need to be resolved against Open Food Facts' own image-serving convention, and this script does not attempt that resolution.

Where the transformation code lives, and what is still open

Licence

Open Food Facts data is used under the Open Database Licence (ODbL) 1.0. Contributor-supplied product images are separately licensed under CC BY-SA 3.0 and are not covered by this notice. Open Food Facts itself: world.openfoodfacts.org.

See Data sources & licences for every dataset Nyamly uses, and Privacy for how personal data is handled.