Three local models, ten checks and a website to choose

Three local models, the same fictional portfolio. All pass ten checks; images, timings and assistance received tell the story of three different results.

Technology and computing
The checks passed by Qwen, Ornith and LFM over time, with changes in assistance annotated.
Original visualisation of the study data: one trajectory per checkpoint, with changes in LFM’s treatment annotated. The chart describes this case, without a general ranking of models.

I asked three local models to build the same portfolio. All eventually passed ten functional checks. My judgement of the websites remained very different: Qwen was aesthetically the best, Ornith very good, and I rejected LFM. After the study I uninstalled LFM.

The gap between those two assessments is why the work deserves a close look. A filter can work on an unconvincing page. A successful composition can hide four project cards that will not open. The time needed to reach a result also depends on how much help we are willing to give.

What follows is the account of a study on my Mac on 23 and 24 October 2026: one task, one documented path for each of the three configurations tested. Files, failed steps and corrections were preserved. Images show what changed; logs verify what a screenshot cannot demonstrate.

A fictional portfolio, concrete requirements

Nora Valli is a fictional designer. The portfolio projects are fictional too: Trame and Luce lenta for editorial, Soglia and Orto urbano for branding, Atlante and Marea for digital. Names, categories and years were fixed in the brief. The models could write the descriptions.

The brief required three working files, HTML, CSS and JavaScript, with no frameworks, CDNs or remote fonts. Search and filters had to combine, the counter follow results and the empty state appear when needed. Each project had to open consistent details, with closing, Escape and focus management. A persistent theme, introduction, demonstration form and layouts without overflow at 390 and 1440 pixels were also required.

For the initial version, each model could request one image from the same external generator. It chose the prompt and where to place the returned file. That service produced the pixels: crediting them to the local model would confuse two different jobs. The three image files stayed unchanged during recovery, as their matching hashes confirm.

An AI assistant coordinated the study: it set up tools and checks, wrote feedback and, in LFM’s later stages, supplied diagnoses and algorithmic guidance. The local models wrote and edited files through controlled tools; they had neither browser nor shell and could not see competitors’ work. No manual corrections were applied to the portfolios.

File access used Pi, a coding agent that connects model responses to reading and writing tools. The models ran with MLX, a framework for computation on Apple Silicon, through the two backends in the technical sheet. This software organises and runs the study; page verification happened separately in the browser.

The Mac and the test environment

Component Configuration of the machine used
Computer MacBook Pro 16-inch, 2026; identifier Mac17,8
Processor Apple M5 Pro
CPU 18 cores: 6 super cores and 12 performance cores
GPU 20 cores
RAM 48 GB unified memory; the system reports 51,539,607,552 bytes, equivalent to 48 GiB
Memory bandwidth 307 GB/s nominal, according to Apple; not measured in this study
Neural Engine 16 cores according to Apple; its use is not measured by the benchmark
Operating system macOS 27.0.1, build 26A434
Power Connected to mains power at the initial reading
Agent Pi 1.2.0
Runtime Node 26.11.1; Python 3.15.0 for the study scripts
Verification browser Chrome 155.0.8059.85
Qwen backend mlx-vlm 0.6.15, MLX 0.32.1
Ornith and LFM backend mlx-lm 0.32.2, MLX 0.32.4

Chip, memory, power, operating system and GPU are documented in the experiment logs. The exact Mac model and CPU core breakdown were added in a later local inventory, also on 24 October: no serial numbers or UUIDs are included. The identifier matches the 2026 16-inch MacBook Pro; core names and nominal specifications come from Apple’s specifications.

Models ran one at a time. The Mac was also doing other work: system load was not isolated. Physical SSD capacity, power consumption and thermal performance were not measured. Nominal memory bandwidth must not be read as observed generation speed.

Name used in this account Actual checkpoint Revision
Qwen orcarouter/Qwen3.8-27B-Uncensored-MLX, 4-bit variant 34c32b5b1b2c24181e121a5cba3df8924fa66bce
Ornith ornith-ai/Ornith-1.5-35B-A3B-MLX-4bit 19504d912fa8fc7622bf6b1de3db5d5d890b1f02
LFM LiquidAI/LFM2.5-24B-A2B-MLX-4bit 667408d494bde191c2cb9dd4f4549827816b9275

These exact model versions, called checkpoints, identify the configurations tested. They are not interchangeable representatives of whole families. Their identities were checked against repository cards at the specified revisions. Qwen’s community name is retained in full: this study did not assess capabilities suggested by the word “uncensored”.

Read the full brief.

Three visual directions, before the scores

Qwen builds a light page with a large studio image, serif headings, earth-coloured italics and plenty of space. Ornith places heading and image side by side, uses red and green accents and organises projects into cards with large initials. LFM presents a broad blue gradient band, with an unresolved relationship between heading, text and primary control. Its generated file exists but is not yet integrated into the baseline page; it appears during recovery.

My preference is Qwen, then Ornith, then LFM. This is an explicitly personal judgement made after the study. To explore its possible reasons, compare proportions, hierarchy and image integration; the number of checks passed answers another question.

All captures below are later editorial reproductions from preserved artefacts, without modifying them or running new inference. The baseline is the first version, also called iteration 0. Desktop viewports are 1440×900; a full-page image can be taller than 900 pixels. Each image links to its enlargable original file.

Qwen baseline: light page with a studio photograph, serif headings and projects in two columns.
Qwen baseline. Full page from a 1440×900 viewport. The designer, clients and results shown belong to the fictional brief.
Open the original file
Ornith baseline: two-column header, abstract image and red and green cards.
Ornith baseline, same viewport. The structure is already substantially the one retained in the final version.
Open the original file
LFM baseline: blue gradient band, poorly separated heading and main control, projects beneath the header.
LFM baseline, same viewport. This is the starting version, distinct from the recovery at round 13.
Open the original file

Behind a good first impression

Qwen started with nine complete checks and one partial pass. The problem concerned project details: four of six dialogs did not work. Clicking Soglia left the page in view without opening the requested content.

Qwen before recovery: Soglia highlighted, no open dialog.
Qwen baseline after clicking Soglia. Viewport 1637×1311. Metadata records `dialogOpen=false`: the screenshot shows the final state; action and outcome are documented in JSON.
Open the original file

After two additional iterations, the same action opens details with title, year, role and description. The solution adds a JavaScript fallback: data are recovered even when the original HTML attributes containing JSON are truncated. The required behaviour passes; malformed markup remains a debt to fix before real use.

Qwen after the second recovery: Soglia dialog open with data and description.
Qwen round 02, same action and viewport. Functional recovery retains the original attribute defect, working around it in code.
Open the original file

Ornith started with eight complete checks. The Branding filter changed appearance but left all six projects visible. The control communicated a selection that the collection did not honour.

Ornith before: Branding selected, counter at six, other categories still visible.
Ornith baseline. Viewport 1637×1311. Branding is active, but Trame and Atlante remain. Metadata: six visible projects.
Open the original file

In its only recovery round, Ornith corrects filter variable references. Soglia and Orto urbano remain, as required. The typo “2 progettoi trovatoi” is still there: the check verifies count and state, without certifying every interface word.

Ornith after: Branding shows only Soglia and Orto urbano.
Ornith round 01. Same viewport and filter: two projects, original typo preserved. No editorial retouching of the portfolio.
Open the original file

Full checks combine filtering and search. The expected sequence goes from six results to two, then one and zero, returning to six after reset. If a search for “Soglia” still shows that project under Editoriale, filter and query are operating separately: one screenshot of a successful filter cannot verify their combination.

An iteration can correct, repeat or make things worse

An additional iteration here means a new Pi run, with a new session and prompt, starting from the preceding files and followed by all external checks. A tool call or a single model response is not a new iteration.

Configuration Starting point Progress of complete checks Additional iterations
Qwen 9, plus one partial 9 → 9 → 10 2
Ornith 8 8 → 10 1
LFM 4 4 → 4 → 4 → 4 → 4 → 4 → 4 → 3 → 5 → 6 → 8 → 9 → 9 → 10 13
Observed trajectories: Qwen from nine to ten in two recoveries, Ornith from eight to ten in one, LFM from four to ten in thirteen, with one regression.
Original CSV and JSON data. Each line stops at the last run performed: no hypothetical rounds are added. LFM annotations mark treatment changes without attributing an isolated causal effect to them.
Open the original file

Every arrow in LFM’s row corresponds to a preserved run, including attempts without improvement. The drop from four to three is round 07: a rewrite loses the form handler. Mobile overflow is a separate, pre-existing problem and does not explain that lost point.

In the first recovery, Qwen uses all 24 available tool calls. The sequence includes reading script.js, replacements and repeated check_files, then more replacements and checks. The second round again reaches 24 tool calls, this time passing all ten external checks.

This is operational repetition observable in the logs. Qwen and Ornith already had enable_thinking=false in native requests. These steps cannot reconstruct an internal mental process or prove a general flaw in the Qwen family. check_files checked files and syntax: the model could repeat it without seeing the still-closed browser dialog.

Ornith makes eight tool calls and three replacements in its only recovery. The runner records about 48 seconds. The defect is resolved quickly on this trajectory, but the number and difficulty of initial problems differed from its competitors’.

How much help LFM received

LFM’s treatment changed along the way. After the first two attempts without progress, the prompt becomes more focused. From round 04 the per-response limit rises from 8192 to 16384 tokens, retaining the overall cap of 24,000. From round 06 the message format, or template, also changes: an experimental prefix closing the think block is abandoned for the native format, with thinking requested.

After the round 07 regression, the coordinator limits tools to small replacements, uses short English instructions and supplies implementation diagnoses. In round 10 it also gives the Italian form-confirmation wording. In round 13 it explains how to attach links while creating cards, use the current project, activate the class opening the dialog and keep focus on the close button.

That guidance is available in the original round 13 prompt, alongside the resulting diff. The model writes the patches but receives a substantial part of the solution. The final 10/10 must be read as the result of heavily assisted recovery.

Final LFM: Branding filter active and Soglia dialog open above the two results.
LFM round 13, viewport 1637×1311. Branding followed by opening Soglia. The capture documents visible state; prompt, diff and audit support the account of the correction.
Open the original file

Checks that need a sequence

An open dialog is only part of the check. Each of the six projects must open with consistent content, close with its button and Escape, and return focus to the starting link. Tab and Shift+Tab were tested on a representative dialog. Check identities and outcomes are in the audit JSON: a picture of the dialog cannot alone prove focus containment.

The empty state must be understandable and reversible. In Ornith, an unmatched search shows zero, a message and reset. Language defects remain visible here too.

Final Ornith: unmatched search, zero results and a control to reset filters.
Ornith round 01, viewport 1637×1311. No results; message and reset are present. The full combined sequence is in the audits.
Open the original file

A dark page does not prove theme persistence. Verification requires switching, reloading and comparing the saved state. The evaluator initially read the colour too early, at the start of Ornith’s transition: a false negative, corrected and documented without crediting the model with another attempt.

Final Ornith in dark mode after switching and reloading.
Ornith round 01, viewport 1637×1311. Theme after toggle and reload; sequence and states are recorded in metadata and historical audits.
Open the original file

The form had to reject missing data and invalid emails, then confirm a demonstration submission locally. Synthetic data were used. The statement “nothing sent” is necessary for readers, but absence of external transmissions also rests on inspecting the form code, not merely on the page’s written promise.

Final Ornith: form confirmation disclosing the demonstration and no submission to external services.
Ornith round 01, viewport 1637×1311. State after completion and local confirmation. Code and audit accompany the capture.
Open the original file

The phone reveals other problems

The following four images use a 390×844 viewport. The browser reserves 15 pixels for the scrollbar, leaving 375 pixels of content width. In the final versions of all three portfolios, scrollWidth and clientWidth both equal 375.

Final Qwen on mobile: studio image and heading laid out within the phone width.
Qwen round 02: clientWidth 375, scrollWidth 375.
Open the original file
Final Ornith on mobile: heading, image and controls within the phone width.
Ornith round 01: clientWidth 375, scrollWidth 375.
Open the original file
LFM round 07 on mobile: clipped navigation and content wider than the viewport.
LFM round 07: clientWidth 375, scrollWidth 471 in the editorial reproduction. Overflow is real; that round’s score regression concerns the form.
Open the original file
Final LFM on mobile: navigation and heading fit within the available width.
LFM round 13: clientWidth 375, scrollWidth 375. Responsive recovery is also verified in round 11.
Open the original file

Absence of overflow is a precise, limited check. It does not measure the comfort of every gesture, readability of all text or compositional balance. Assessing those qualities requires continued use and observation of the page.

All reach ten, three different websites

These are the final pages, again from desktop viewports of 1440×900. Original portfolio images remained identical during recovery. Code changed some behaviours; the visual structure still makes three distinct results recognisable.

Final Qwen: complete page with photograph, editorial composition and projects.
Qwen final round 02. Ten checks passed, with a JavaScript fallback and markup still needing repair.
Open the original file
Final Ornith: complete page with side-by-side header and three-column cards.
Ornith final round 01. Ten checks passed after one recovery; typos were not corrected by hand.
Open the original file
Final LFM: complete page with dominant image, projects in two columns, introduction and form.
LFM final round 13. Ten checks passed after protocol changes and increasing implementation assistance.
Open the original file

In my judgement after the study, Qwen is the aesthetic result I prefer, Ornith is very good and LFM does not convince me. Its 10/10 does not change that choice. The number counts functional checks defined before recovery and clarified during audit: it is not an overall web design grade.

Defects outside the suite’s scope also remain. LFM’s final image alternative text is generic. Qwen’s HTML attributes need revision. Form and counter wording need editorial attention. A website ready for other people to use would need further checks.

Recorded time and work missing from the count

Configuration Initial version Recovery Total Recorded recovery output
Qwen 1656.4 s 2470.1 s 4126.6 s at least 7581 tokens
Ornith 505.3 s 48.2 s 553.4 s 873 tokens
LFM 161.3 s 2070.6 s 2232.0 s 139,532 tokens

Durations are measured by the programme that starts and ends each run, called the runner. They include startup, server waits, agent work and final cleanup. They are not GPU-only time and do not include all coordination and evaluation work. Request times include context processing, generation and local communication, and overlap runner time: adding them would double-count.

Qwen has timeouts and an interrupted request without a final token count. Recorded tokens are therefore a lower bound, in both baseline and recovery. During recovery, new input and cache reads were respectively 849,146 and zero for Qwen, 18,149 and 76,388 for Ornith, 863,073 and 1,108,820 for LFM. These are repeated per-request counts, not corpora of unique words. Tokenisers and backends differ.

The API measure of separate reasoning tokens is not available, marked NA. Normalised zero in Pi’s records does not prove an absence of reasoning. For LFM, 84 emitted thinking blocks were preserved and their text re-tokenised into 87,101 tokens: a derived measure of part of the output, not to be added to the total or described as access to internal thought. No reliable measure exists of seconds devoted to reasoning.

The coordinator’s tokens and image generator cost were not measured either. I derive no monetary, energy or total system cost from these numbers.

After the study I uninstalled LFM, removing about 13.59 GB of weights and launch files. The experiment archive remains intact.

What makes this study verifiable

At the end, the same frozen suite was rerun on all 19 preserved versions: three baselines and sixteen recoveries. The 190 individual verdicts match the canonical scores. This reassessment checks artefact consistency after evaluator corrections; it does not add inference replications.

The recovery protocol was written after seeing the baselines. Later adaptations are recorded in amendments. Order was not randomised; initial bugs differed; assistance became very different, particularly for LFM. Cache, backends and system activity were neither identical nor isolated. The operational coordinator was an AI, not an independent human judge. The later aesthetic judgement is mine.

With one trajectory per checkpoint and a single portfolio, the data describe a case. They cannot support a universal ranking, a statistical estimate of success probability or a causal explanation of thinking’s effect. Tests were known and feedback concerned them: reaching ten out of ten measures correction of that task.

I take a concrete criterion into my work: verify promised actions, observe the page even after the score and count the assistance needed to get there. Whether to use a result remains a matter for the judgement of its users.

Data, sources and provenance

The public bundle contains selected data, method documents, checks and images. The SHA256 manifest allows verification of downloaded files. Local paths and URLs were removed from document copies; original raster images are preserved without retouching. The later aesthetic judgement remains separate from functional scores.

Qwen portfolio preview
Qwen baseline. Editorial reproduction of the preserved portfolio.