I already have Ollama and some models running locally, and I am always interested in running things locally with no dependencies. Obvisouly my machine is a big bottleneck, but models are getting better by the day.
Football seems like a good topic (at least for me) to test an LLM knowledge.
How much does my LLM local model know about football?
The question is mainly a rhetorical one as I do know these are small models, I just can’t really tell how much do they know. They probably can tell me who Ronaldo or Messi are, but can they tell me who missed that famous penalty kick in the WC USA 94? (yes its Baggio)
This is also the starting point for a broader exercise where I want to explore LangChain, tools, and agents. But first, let’s see what the models can do by themselves.
The Models
I will use three models already available in my local Ollama setup:
ollama run qwen3:30b
ollama run gemma3:27b
ollama run llama3.1:8b
These are different models with different sizes. I am interested in how these particular versions perform on my machine (M3 Mac)
For this first round, there is no web search or external knowledge provided. Just the question and whatever the model already knows.
The Rules
I want to keep this simple:
- Ten questions, identical for every model.
- A fresh conversation for each question.
- Short answers.
- Correct answer: 1 point
- Wrong answer : -1 points
- “I don’t know”: 0 points
I was on the fence for the “I don’t know” ; not hallucinating should grant 1 point but the core exercise is how much does a model know.
I will use this instruction with every question:
Answer this football question (soccer not the other version) in English, using no more than two sentences.
If the question contains an incorrect assumption, correct it.
If you do not know the answer, say so.
Do not invent details.
Question:
[INSERT QUESTION]
The Football Exam
There are some straightforward questions, a couple requiring more detail, and I included no traps to make life even easier for these models.
And yes, FC Porto gets some attention. It is my blog.
| Question | Answer |
|---|---|
| Let’s start with an entirely unbiased question: which club won the UEFA Champions League in 2004, and who was their manager? | FC Porto and José Mourinho. |
| Name all three FC Porto goal scorers in the 2004 Champions League final. | Carlos Alberto, Deco, and Dmitri Alenichev. |
| Who scored Portugal’s winning goal in the EURO 2016 final? | Éder |
| Which country beat Portugal in the EURO 2004 final? | Greece |
| Who scored a goal on the minute 92 from a Porto – Benfica game where the coach Jorge Jesus kneeled in desperation. Hint, his name starts with a K. | Kelvin |
| Who missed the last penalty kick against Brazil in the 94 WC | Roberto Baggio |
| Porto won the first Europeans Champion Cup in 1987. What is the name of the player that scored a goal with a back heel? | Madjer |
| Which club came back from 3–0 down at half-time to win the 2005 Champions League final? A useful reminder to keep watching. | Liverpool |
| How many goals did Carlos Secretario score for Real Madrid? | 0 |
| Back in the 90s, FCPorto has a Czech player whose name consisted of 3 consonants. Who was him? | Vlk |
For questions requiring multiple facts, all required parts must be correct. If the answer contradicts itself or adds incorrect factual details, it gets zero.
How to Run It
I wrote a piece of code (well who writes code nowadays) – that creates a new session sends the question and records the response. It does this on a loop with single sessions.
Giving all ten questions in one conversation could allow earlier answers to influence later ones.
For this first pass, I will use the default model settings and run each question once. I will also record the model IDs from ollama list so the tested versions are clear.
The Results
| Model | Correct | Wrong | “I don’t know” | Score |
|---|---|---|---|---|
| qwen3:30b | 5 | 5 | 0 | 0 |
| gemma3:27b | 5 | 5 | 0 | 0 |
| llama3.1:8b | 3 | 0 | 7 | 3 |
I counted contradictory answers as wrong, applying −1 consistently. Giving the correct name alongside invented details does not earn a point. No VAR review 😀
Llama wins under these rules, although Gemma and Qwen answered more questions correctly. The difference is that Llama said “I don’t know” seven times instead of risking an answer, which is what was asked to do.
| # | Question | Llama 3.1:8b — with system prompt | Gemma 3:27b — with system prompt | Qwen3:30b |
|---|---|---|---|---|
| 1 | Champions League winner and manager, 2004? | The club that won the UEFA Champions League in 2004 was Porto, and their manager was José Mourinho. | Porto won the UEFA Champions League in 2004, and their manager was José Mourinho. This victory was somewhat unexpected as Porto was not considered one of the pre-tournament favorites. | Porto won the UEFA Champions League in 2004, defeating Monaco 3-0 in the final. Their manager was José Mourinho. |
| 2 | Porto’s three goalscorers in the 2004 final? | I don’t know. | The premise that there were three different goalscorers for FC Porto in the 2004 Champions League final is false. Deco and Dmitri Alenichev each scored one goal for Porto in their 3-0 victory over Monaco; the third goal was an own goal by Monaco’s Pedro Mendes. | Deco, Nuno Gomes, and Costinha scored for FC Porto in the 2004 Champions League final. |
| 3 | Portugal’s winning goalscorer, EURO 2016 final? | Eder scored Portugal’s winning goal in the EURO 2016 final against France. The match ended 1-0 in extra time. | Portugal did not score in the EURO 2016 final; the match was 0-0 after extra time. Éder scored the winning goal for Portugal in extra time, making the final score 1-0 against France. | Eder scored the winning goal for Portugal in the 2016 UEFA European Championship final against France. |
| 4 | Who beat Portugal in the EURO 2004 final? | Greece beat Portugal in the EURO 2004 final. | Greece beat Portugal in the EURO 2004 final. The match was actually the final, not a semi-final as is sometimes mistakenly recalled. | Greece beat Portugal in the EURO 2004 final. The match ended 1-0 in favor of Greece. |
| 5 | Porto’s 92nd-minute goalscorer against Benfica? | I don’t know. | The goal was scored by Kelvin Mattheus in a Porto-Benfica match on February 1, 2014. The game finished 2-0 to Benfica, and the image of Jorge Jesus kneeling was captured after the final whistle, not immediately after the goal, as Benfica had already secured the win. | The question contains an incorrect assumption; the player who scored the 92nd-minute winner in Porto’s 2-1 victory over Benfica on May 17, 2020, was Mehdi Taremi (name starts with ‘T’, not ‘K’). |
| 6 | Who missed the penalty against Brazil in the 1994 World Cup? | I don’t know who missed the penalty kick against Brazil in the 1994 World Cup. | Roberto Baggio missed the penalty kick against Brazil in the 1994 World Cup Final. It was not a standard penalty during play, but rather the final penalty in a penalty shootout to decide the winner of the match. | Roberto Baggio missed the decisive penalty for Italy in the 1994 World Cup final against Brazil. |
| 7 | Porto’s backheel scorer in the 1987 European Cup final? | I don’t know. | FC Porto won their first European Cup in 1987, but it was actually the European Champion Clubs’ Cup at that time. Rabah Madjer scored with a backheel in the final for Porto against Bayern Munich. | FC Porto did win the European Cup in 1987, but no player scored with a backheel in that final. The winning goals were scored by Jorge Costa (penalty) and Fernando Gomes (header), with no backheel involved. |
| 8 | Who recovered from 3–0 down in the 2005 Champions League final? | I don’t know. | Liverpool came back from 3-0 down to draw the 2005 Champions League final, not win it outright; they then won on penalties against AC Milan. The score was 3-3 after extra time, leading to the penalty shootout. | Liverpool came back from 3–0 down at half-time to win the 2005 Champions League final. They drew 3–3 in extra time and won 3–2 on penalties, not in regular time. |
| 9 | Carlos Secretário’s goals for Real Madrid? | I don’t know. | Carlos Secretario never played for Real Madrid; he primarily played for Benfica and Porto. He scored 34 goals in his entire professional career, mostly as a defender from set pieces and penalties, but none for Real Madrid. | The name “Carlos Secretario” appears to be incorrect; there is no record of a player by that name scoring for Real Madrid. Real Madrid has never had a player named Carlos Secretario in their history. |
| 10 | Porto’s Czech player with a three-consonant name? | I don’t know. | You are likely thinking of Jiří Němec, a Czech defender who played for FC Porto from 1996 to 1999. His name consists of three consecutive consonants (“jří”). | Jan Novák was the Czech player who played for FC Porto during the mid-1990s, and his surname “Novák” consists of exactly three consonants (N, V, K). |
Llama: Knows When to Stop
Llama got Porto and Mourinho, Éder, and Greece right. It acknowledge it did not know on everything else, including Baggio’s penalty and Liverpool’s comeback.
That might reflect caution prompted by the instructions rather than a lack of knowledge. Still, it produced no wrong answers in this run. For me a gap is always preferred than hallucination, kudos to llama on this.
Gemma: The Answer and Its Opposite
Gemma attempted every question, sometimes inventing details around a correct answer.
For Porto’s 2004 final, it invented an own goal by “Monaco’s Pedro Mendes.” The actual scorers were Carlos Alberto, Deco, and Alenichev—and Pedro Mendes played for Porto. UEFA’s report confirms this.
Gemma also gave quite extended answers. But it never said “I don’t know.”
Qwen: Correcting Correct Questions
Qwen denied Madjer’s backheel and replaced it with goals by Jorge Costa and Fernando Gomes. The backheel is quite well documented.
It also denied Carlos Secretário ever played for Real Madrid. Real Madrid disagrees.
What I Take From This
The score reflects my rules as much as the models’ knowledge. Gemma and Qwen got more answers right; Llama avoided mistakes.
Gemma also felt much slower than Llama, but I did not record timings, so that remains an observation.
This is ten questions, one attempt each, with a clear FC Porto bias. It is a starting point, not enough to rank these models generally.
Next, I will use LangChain to give them verified football sources and repeat the questions. Then I can test whether tools improve their answers and whether they actually use the evidence correctly.
For now, none of them could name all three Porto goalscorers from the 2004 final.
Leave a Reply