I asked three local models to build the same portfolio. All eventually passed ten functional checks. My judgement of the websites remained very different: Qwen was aesthetically the best, Ornith very good, and I rejected LFM. After the study I uninstalled LFM.
The gap between those two assessments is why the work deserves a close look. A filter can work on an unconvincing page. A successful composition can hide four project cards that will not open. The time needed to reach a result also depends on how much help we are willing to give.
What follows is the account of a study on my Mac on 23 and 24 October 2026: one task, one documented path for each of the three configurations tested. Files, failed steps and corrections were preserved. Images show what changed; logs verify what a screenshot cannot demonstrate.
A fictional portfolio, concrete requirements
Nora Valli is a fictional designer. The portfolio projects are fictional too: Trame and Luce lenta for editorial, Soglia and Orto urbano for branding, Atlante and Marea for digital. Names, categories and years were fixed in the brief. The models could write the descriptions.
The brief required three working files, HTML, CSS and JavaScript, with no frameworks, CDNs or remote fonts. Search and filters had to combine, the counter follow results and the empty state appear when needed. Each project had to open consistent details, with closing, Escape and focus management. A persistent theme, introduction, demonstration form and layouts without overflow at 390 and 1440 pixels were also required.
For the initial version, each model could request one image from the same external generator. It chose the prompt and where to place the returned file. That service produced the pixels: crediting them to the local model would confuse two different jobs. The three image files stayed unchanged during recovery, as their matching hashes confirm.
An AI assistant coordinated the study: it set up tools and checks, wrote feedback and, in LFM’s later stages, supplied diagnoses and algorithmic guidance. The local models wrote and edited files through controlled tools; they had neither browser nor shell and could not see competitors’ work. No manual corrections were applied to the portfolios.
File access used Pi, a coding agent that connects model responses to reading and writing tools. The models ran with MLX, a framework for computation on Apple Silicon, through the two backends in the technical sheet. This software organises and runs the study; page verification happened separately in the browser.
The Mac and the test environment
| Component | Configuration of the machine used |
|---|---|
| Computer | MacBook Pro 16-inch, 2026; identifier Mac17,8 |
| Processor | Apple M5 Pro |
| CPU | 18 cores: 6 super cores and 12 performance cores |
| GPU | 20 cores |
| RAM | 48 GB unified memory; the system reports 51,539,607,552 bytes, equivalent to 48 GiB |
| Memory bandwidth | 307 GB/s nominal, according to Apple; not measured in this study |
| Neural Engine | 16 cores according to Apple; its use is not measured by the benchmark |
| Operating system | macOS 27.0.1, build 26A434 |
| Power | Connected to mains power at the initial reading |
| Agent | Pi 1.2.0 |
| Runtime | Node 26.11.1; Python 3.15.0 for the study scripts |
| Verification browser | Chrome 155.0.8059.85 |
| Qwen backend | mlx-vlm 0.6.15, MLX 0.32.1 |
| Ornith and LFM backend | mlx-lm 0.32.2, MLX 0.32.4 |
Chip, memory, power, operating system and GPU are documented in the experiment logs. The exact Mac model and CPU core breakdown were added in a later local inventory, also on 24 October: no serial numbers or UUIDs are included. The identifier matches the 2026 16-inch MacBook Pro; core names and nominal specifications come from Apple’s specifications.
Models ran one at a time. The Mac was also doing other work: system load was not isolated. Physical SSD capacity, power consumption and thermal performance were not measured. Nominal memory bandwidth must not be read as observed generation speed.
| Name used in this account | Actual checkpoint | Revision |
|---|---|---|
| Qwen | orcarouter/Qwen3.8-27B-Uncensored-MLX, 4-bit variant | 34c32b5b1b2c24181e121a5cba3df8924fa66bce |
| Ornith | ornith-ai/Ornith-1.5-35B-A3B-MLX-4bit | 19504d912fa8fc7622bf6b1de3db5d5d890b1f02 |
| LFM | LiquidAI/LFM2.5-24B-A2B-MLX-4bit | 667408d494bde191c2cb9dd4f4549827816b9275 |
These exact model versions, called checkpoints, identify the configurations tested. They are not interchangeable representatives of whole families. Their identities were checked against repository cards at the specified revisions. Qwen’s community name is retained in full: this study did not assess capabilities suggested by the word “uncensored”.
Three visual directions, before the scores
Qwen builds a light page with a large studio image, serif headings, earth-coloured italics and plenty of space. Ornith places heading and image side by side, uses red and green accents and organises projects into cards with large initials. LFM presents a broad blue gradient band, with an unresolved relationship between heading, text and primary control. Its generated file exists but is not yet integrated into the baseline page; it appears during recovery.
My preference is Qwen, then Ornith, then LFM. This is an explicitly personal judgement made after the study. To explore its possible reasons, compare proportions, hierarchy and image integration; the number of checks passed answers another question.
All captures below are later editorial reproductions from preserved artefacts, without modifying them or running new inference. The baseline is the first version, also called iteration 0. Desktop viewports are 1440×900; a full-page image can be taller than 900 pixels. Each image links to its enlargable original file.
Behind a good first impression
Qwen started with nine complete checks and one partial pass. The problem concerned project details: four of six dialogs did not work. Clicking Soglia left the page in view without opening the requested content.
After two additional iterations, the same action opens details with title, year, role and description. The solution adds a JavaScript fallback: data are recovered even when the original HTML attributes containing JSON are truncated. The required behaviour passes; malformed markup remains a debt to fix before real use.
Ornith started with eight complete checks. The Branding filter changed appearance but left all six projects visible. The control communicated a selection that the collection did not honour.
In its only recovery round, Ornith corrects filter variable references. Soglia and Orto urbano remain, as required. The typo “2 progettoi trovatoi” is still there: the check verifies count and state, without certifying every interface word.
Full checks combine filtering and search. The expected sequence goes from six results to two, then one and zero, returning to six after reset. If a search for “Soglia” still shows that project under Editoriale, filter and query are operating separately: one screenshot of a successful filter cannot verify their combination.
An iteration can correct, repeat or make things worse
An additional iteration here means a new Pi run, with a new session and prompt, starting from the preceding files and followed by all external checks. A tool call or a single model response is not a new iteration.
| Configuration | Starting point | Progress of complete checks | Additional iterations |
|---|---|---|---|
| Qwen | 9, plus one partial | 9 → 9 → 10 | 2 |
| Ornith | 8 | 8 → 10 | 1 |
| LFM | 4 | 4 → 4 → 4 → 4 → 4 → 4 → 4 → 3 → 5 → 6 → 8 → 9 → 9 → 10 | 13 |
Every arrow in LFM’s row corresponds to a preserved run, including attempts without improvement. The drop from four to three is round 07: a rewrite loses the form handler. Mobile overflow is a separate, pre-existing problem and does not explain that lost point.
In the first recovery, Qwen uses all 24 available tool calls. The sequence includes reading script.js, replacements and repeated check_files, then more replacements and checks. The second round again reaches 24 tool calls, this time passing all ten external checks.
This is operational repetition observable in the logs. Qwen and Ornith already had enable_thinking=false in native requests. These steps cannot reconstruct an internal mental process or prove a general flaw in the Qwen family. check_files checked files and syntax: the model could repeat it without seeing the still-closed browser dialog.
Ornith makes eight tool calls and three replacements in its only recovery. The runner records about 48 seconds. The defect is resolved quickly on this trajectory, but the number and difficulty of initial problems differed from its competitors’.
How much help LFM received
LFM’s treatment changed along the way. After the first two attempts without progress, the prompt becomes more focused. From round 04 the per-response limit rises from 8192 to 16384 tokens, retaining the overall cap of 24,000. From round 06 the message format, or template, also changes: an experimental prefix closing the think block is abandoned for the native format, with thinking requested.
After the round 07 regression, the coordinator limits tools to small replacements, uses short English instructions and supplies implementation diagnoses. In round 10 it also gives the Italian form-confirmation wording. In round 13 it explains how to attach links while creating cards, use the current project, activate the class opening the dialog and keep focus on the close button.
That guidance is available in the original round 13 prompt, alongside the resulting diff. The model writes the patches but receives a substantial part of the solution. The final 10/10 must be read as the result of heavily assisted recovery.
Checks that need a sequence
An open dialog is only part of the check. Each of the six projects must open with consistent content, close with its button and Escape, and return focus to the starting link. Tab and Shift+Tab were tested on a representative dialog. Check identities and outcomes are in the audit JSON: a picture of the dialog cannot alone prove focus containment.
The empty state must be understandable and reversible. In Ornith, an unmatched search shows zero, a message and reset. Language defects remain visible here too.
A dark page does not prove theme persistence. Verification requires switching, reloading and comparing the saved state. The evaluator initially read the colour too early, at the start of Ornith’s transition: a false negative, corrected and documented without crediting the model with another attempt.
The form had to reject missing data and invalid emails, then confirm a demonstration submission locally. Synthetic data were used. The statement “nothing sent” is necessary for readers, but absence of external transmissions also rests on inspecting the form code, not merely on the page’s written promise.
The phone reveals other problems
The following four images use a 390×844 viewport. The browser reserves 15 pixels for the scrollbar, leaving 375 pixels of content width. In the final versions of all three portfolios, scrollWidth and clientWidth both equal 375.
Absence of overflow is a precise, limited check. It does not measure the comfort of every gesture, readability of all text or compositional balance. Assessing those qualities requires continued use and observation of the page.
All reach ten, three different websites
These are the final pages, again from desktop viewports of 1440×900. Original portfolio images remained identical during recovery. Code changed some behaviours; the visual structure still makes three distinct results recognisable.
In my judgement after the study, Qwen is the aesthetic result I prefer, Ornith is very good and LFM does not convince me. Its 10/10 does not change that choice. The number counts functional checks defined before recovery and clarified during audit: it is not an overall web design grade.
Defects outside the suite’s scope also remain. LFM’s final image alternative text is generic. Qwen’s HTML attributes need revision. Form and counter wording need editorial attention. A website ready for other people to use would need further checks.
Recorded time and work missing from the count
| Configuration | Initial version | Recovery | Total | Recorded recovery output |
|---|---|---|---|---|
| Qwen | 1656.4 s | 2470.1 s | 4126.6 s | at least 7581 tokens |
| Ornith | 505.3 s | 48.2 s | 553.4 s | 873 tokens |
| LFM | 161.3 s | 2070.6 s | 2232.0 s | 139,532 tokens |
Durations are measured by the programme that starts and ends each run, called the runner. They include startup, server waits, agent work and final cleanup. They are not GPU-only time and do not include all coordination and evaluation work. Request times include context processing, generation and local communication, and overlap runner time: adding them would double-count.
Qwen has timeouts and an interrupted request without a final token count. Recorded tokens are therefore a lower bound, in both baseline and recovery. During recovery, new input and cache reads were respectively 849,146 and zero for Qwen, 18,149 and 76,388 for Ornith, 863,073 and 1,108,820 for LFM. These are repeated per-request counts, not corpora of unique words. Tokenisers and backends differ.
The API measure of separate reasoning tokens is not available, marked NA. Normalised zero in Pi’s records does not prove an absence of reasoning. For LFM, 84 emitted thinking blocks were preserved and their text re-tokenised into 87,101 tokens: a derived measure of part of the output, not to be added to the total or described as access to internal thought. No reliable measure exists of seconds devoted to reasoning.
The coordinator’s tokens and image generator cost were not measured either. I derive no monetary, energy or total system cost from these numbers.
After the study I uninstalled LFM, removing about 13.59 GB of weights and launch files. The experiment archive remains intact.
What makes this study verifiable
At the end, the same frozen suite was rerun on all 19 preserved versions: three baselines and sixteen recoveries. The 190 individual verdicts match the canonical scores. This reassessment checks artefact consistency after evaluator corrections; it does not add inference replications.
The recovery protocol was written after seeing the baselines. Later adaptations are recorded in amendments. Order was not randomised; initial bugs differed; assistance became very different, particularly for LFM. Cache, backends and system activity were neither identical nor isolated. The operational coordinator was an AI, not an independent human judge. The later aesthetic judgement is mine.
With one trajectory per checkpoint and a single portfolio, the data describe a case. They cannot support a universal ranking, a statistical estimate of success probability or a causal explanation of thinking’s effect. Tests were known and feedback concerned them: reaching ten out of ten measures correction of that task.
I take a concrete criterion into my work: verify promised actions, observe the page even after the score and count the assistance needed to get there. Whether to use a result remains a matter for the judgement of its users.
Data, sources and provenance
- Per-iteration data, CSV, JSON results and JSON summary.
- Recovery protocol, to read alongside amendments, roles and aesthetic judgement.
- Evidence index and actions, URLs and viewports of editorial captures. Historical checks are in the materials bundle; they are not new inference runs. Some peripheral marks belong to the browser or extensions.
- Qwen repository at the tested revision, Ornith with the same pinned identification, LFM quantisation card.
The public bundle contains selected data, method documents, checks and images. The SHA256 manifest allows verification of downloaded files. Local paths and URLs were removed from document copies; original raster images are preserved without retouching. The later aesthetic judgement remains separate from functional scores.
















