footballmatrix

Football Matrix (footballmatrix.net) is a football results, squads and transfers website. It collects data from many providers, merges it into one clean "Matrix" dataset, and publishes only that merged data.

Code: /home/rob/code/footballmatrix · Last checked against commit 9f946c3 on 2026-10-14

Related: footballmatrix-redesign is an unreleased new look for the same site. Football Manager data used here is explained in football-manager.

What it does

Public pages: home, competitions, teams, players and fixtures (matches), plus transfers on those pages. Every item has a Matrix ID, used in the web address, such as /team/123. A password-protected /admin area lets the owner compare what each provider says about the same thing. It is read-only, and the older editing screens were deliberately deleted.

Which competitions are public is decided by one database column, matrix_competitions.coverage:

Value	Meaning
`squads_transfers`	Squads and transfers only.
`fixtures_basic`	Plus fixtures and scores.
`fixtures_detailed`	Plus lineups and match detail.
empty (NULL)	Not tracked, not published.

Data is collected from the 2020 season onwards.

How data flows

The order matters and is the most important rule in the project.

Scrape "tf". tf means Transfermarkt for men's competitions and SoccerDonna (same company) for women's. This brings in competitions, fixtures, teams, squads, lineups and players.
Build Matrix from tf (matrix:build). Only tf is allowed to create Matrix competitions, teams, fixtures and people.
Scrape the other providers (SofaScore, FotMob, FBref, WhoScored, Football Manager data, and others).
Match those to the tf-created rows, in the order: competitions, teams, fixtures, people.
A match only fills in a link column on the provider's own row. A row that cannot be matched stays unlinked. It never creates a Matrix row. Builders that did this were removed in October 2026 because they made duplicates.
Fill in Matrix fields by combining the linked sources.
The public site reads only the matrix_* tables, never provider tables and never files in storage/.

Provider data is kept as the provider gave it, mistakes included. If one provider has two entries for the same player, the extra gets duplicate_of_id pointing to one main row, and only the main row carries the Matrix link. Two Matrix people that are really one are merged, keeping the older ID; the old address redirects. Never merge on name alone: a birth date plus club or match evidence is the minimum.

Provider code lives in app/Sources/<Provider> (BeSoccer, FBRef, FMRef, FootballManager, FootyStats, FotMob, Opta, PlaymakerStats, SoccerBase, SoccerDonna, SoccerWay, SofaScore, TransferMarket, UnderStat, WhoScored). Downloaded pages are cached on disk in the folder set by SCRAPES_DIR (/home/rob/scrapes/ here).

Running it

Production is served by nginx with PHP 8.4 from the repo's public folder (site footballmatrix.net, behind Cloudflare). Local steps:

cd /home/rob/code/footballmatrix
composer install
npm install
npm run build
php artisan serve --port={{PORT}}

Tests

cd /home/rob/code/footballmatrix
vendor/bin/phpunit

Tests use an in-memory SQLite database, which is more forgiving than the real MySQL. Always also check pages against the real database (see Deploying).

Configuration

DB_*: the MySQL database.
SCRAPES_DIR: where downloaded provider pages are stored.
ADMIN_EMAILS: comma-separated list of people allowed into /admin.
QUEUE_CONNECTION, CACHE_STORE, SESSION_*: usual settings.

Decisions about the data (current squads, current seasons, which team is in which league, merges) live in database tables, not config files. A small number of reviewed-decision files remain in config/matrix_*.php and are still honoured, but new decisions go in tables.

Scheduled jobs

The server crontab runs the Laravel scheduler every minute (php8.4 artisan schedule:run on this repo). The schedule is in routes/console.php:

Time (server)	Command	What it does
05:55	`matrix:team-coverage --persist`	Refreshes which teams are eligible for collection.
06:00	`footballmatrix:sync-transfers`	Scrapes transfer feeds.
06:00	`footballmatrix:sync-daily-fixtures`	Scrapes recent and upcoming fixtures.
06:00	`footballmatrix:sync-competitions`	Scrapes squads and match details for covered competitions.
06:00	`matrix:daily-build`	Waits for the three scrapes above, then updates the Matrix tables: new and changed fixtures, lineups, people, current seasons, team leagues, squads and links.

The build report is written to storage/app/private/daily-scrapes/matrix-build/.

Commands

Run from /home/rob/code/footballmatrix. Commands that change data preview by default and need --apply (or --persist for coverage) to write.

Refresh one league now

php artisan footballmatrix:sync-competitions --competition={{COMP_ID}}
php artisan footballmatrix:sync-competitions --dry-run

Also accepts --nation=. --dry-run lists the steps without scraping. The other two sync commands also take --dry-run.

Build Matrix from tf

php artisan matrix:build --competition={{COMP_ID}} --season={{SEASON}}
php artisan matrix:build --competition={{COMP_ID}} --season={{SEASON}} --apply
php artisan matrix:daily-build --dry-run

Other options: --nation=, --since=2021 (earliest season), --missing (only create missing rows), and --steps= to run a subset such as people,fixtures.

Change what is published

php artisan matrix:coverage --set={{COMP_ID}}:{{COVERAGE}}
php artisan matrix:coverage --set={{COMP_ID}}:{{COVERAGE}} --persist --reason="{{REASON}}"

The first line previews. The second applies it and keeps a journal entry.

Link or unlink one provider row

php artisan matrix:link {{TABLE}} {{SOURCE_ID}} {{MATRIX_ID}} --reason="{{REASON}}"
php artisan matrix:link {{TABLE}} {{SOURCE_ID}} {{MATRIX_ID}} --reason="{{REASON}}" --apply
php artisan matrix:unlink {{TABLE}} {{SOURCE_ID}} --expect={{MATRIX_ID}} --reason="{{REASON}}" --apply
php artisan matrix:link-history {{TABLE}} {{SOURCE_ID}}

Unlinking records a "not the same" decision so matchers cannot relink the pair. Add --allow-relink if the link was only premature.

Scrape or process one provider feed

php artisan scrape {{SOURCE}} {{CLASS}} --filter=...
php artisan process {{SOURCE}} {{CLASS}} --from=...
php artisan scrape:failed {{SOURCE}}

Every download must go through a scraper class in app/Sources/*/Scrapers and through a proxy. Never fetch a provider any other way. A new feed gets a new scraper class; never point an old one at a new address, because its cache would be overwritten.

Other tools

php artisan matrix:identity plan|apply: preview or apply a bulk change of which Matrix row a provider row belongs to. Apply needs --actor.
tools/matrix/: about a hundred one-off PHP and Python scripts for audits, repairs and merges (for example merge-reviewed-people.php, refresh-current-editions.php, refresh-contracted-teams.php). They preview unless given --apply. tools/scrapes/ is for archive housekeeping.
run_scrapers.sh and get_seasons.sh: older helper scripts that start many scrapers in a loop. The daily jobs above replace most of this.

Deploying

There is no build-and-release step for PHP. nginx serves the repo folder directly, and PHP checks file times on every request, so changed PHP is live the moment it is saved. Data changes are also live at once. That means you must test before saving files, not after.

cd /home/rob/code/footballmatrix
git pull origin {{BRANCH}}
composer install --no-dev --optimize-autoloader
npm run build
php artisan migrate --force

Checks after any change:

Open a match page, a team page, a player page and the home page, and look at the log for production.ERROR.
Merges and exclusions refresh the page cache themselves. Other data changes may need the coverage revision bumped: Cache::forever('footballmatrix:coverage:revision', Str::uuid()) in tinker.

The nginx configuration is kept in deploy/nginx and installed with:

sudo bash /home/rob/code/footballmatrix/deploy/nginx/install.sh

It also sets up scrapes.footballmatrix.net, a password-protected view of the downloaded files. Cloudflare must be set to "Full (strict)" SSL, otherwise visitors get a redirect loop.

How it works

Current state is stored, not guessed. "Current squad" is on the person (contracted_team_id, loan_team_id). "Current season" is set explicitly on competitions, stages and teams. A team's league is computed automatically and should never be set by hand. Split seasons (Apertura/Clausura) are one competition with stages.
Safe writes. Changes go through services that lock, record the before-image in matrix_coverage_runs and refuse out-of-date plans. A guard blocks deletes on Matrix and provider tables.
Coverage work is done in bulk, ranked by how many rows are affected, never as one-off code for one player or club. See the coverage skill in .agents/skills/coverage.
Temporary downloads for investigations go under scrapes/tmp/<purpose>/ with a dated README (see docs/temporary-scrapes.md).
The repo's docs/ has detailed notes: identity, transfers, competition repair, audits. Read AGENTS.md first.

Gotchas

Never change database structure while a long transaction is open. The change waits, every page query queues behind it, and the site hangs (this happened on 4 October 2026). Check SHOW PROCESSLIST first.
Never reset, truncate or rebuild tables on the live database. Additive migrations are fine; commit them straight away, since they take effect before the repo does.
SQLite tests accept queries MySQL rejects. Test real pages.
Do not make the public site read files from storage/; it is not in git.
Do not add "SofaScore-first" or any non-tf path that creates Matrix rows.
A linked provider ID does not prove a player's biography is complete or right.
AGENTS.md.old is history only; AGENTS.md wins.

Changelog

2026-10-14 Initial version.