Is Gemini 4 Hydrogen good enough?

Hydrogen medium's only check on a visual landing page was to dump the page as text and grep for six heading ids, then print that there were no console errors. It never once looked at the page it built.

What it's good for

You could ship it as a real marketing page or use it as a strong first draft. Every Hydrogen effort produced all six sections in a dark editorial style: a serif headline with an ochre italic, a manuscript card with margin notes in the hero, ruled step and feature cards, testimonials with publication names, and three pricing tiers with one marked as recommended. It never looks like a stock SaaS template. The phone layout stacks into one column without overflowing at every effort. Low is already clean and calm.

Where it falls short

Next to OpenAI (new 11), Hydrogen's pages are denser and quieter. The headings are smaller and there are lots of tiny mono labels and extra copy. The hero art is a text card where OpenAI has a striking contour-line figure. The type hierarchy is less confident and leaves less room around things. Low and medium checked their work only by grepping the page's markup for section ids. They never looked at a screenshot and never ran a review.

What paying more buys

Some extra taste: bigger, more assured type, a hero illustration you remember, and more breathing room. It also buys real browser and accessibility checks. Nothing a visitor would notice as broken.

Does more effort help?

Not on looks. Low is arguably the cleanest page. High and xhigh add screenshots and a code review pass, but they also add more microcopy and denser sections. The proof improves, the design does not.

On high, a single red rule runs across the top of the newest article only. It is the one decorative touch on a page whose brief says anything extra counts against you.

What it's good for

You could publish it as it is. Every effort builds the index and all five article pages with marked. Each article has one main heading, a proper machine-readable date, and an 'Originally published on Every' link to the real URL. Every effort uses the swiss-grid classes with the grid lines faintly visible. Medium is the most restrained version: the name and one factual sentence sit in the left module, and the five dated entries are ruled on the right. On a phone it collapses to one clean column.

Where it falls short

Low and high add some layout ideas of their own. Entries stagger left and right across the grid, and high adds a red accent rule over only the first entry. That reads as decoration more than structure. On high's phone layout the titles are squeezed into a narrow column and wrap awkwardly. OpenAI (new 11) uses a bolder masthead and bigger linked titles, so the page has more presence, though it leaves empty bands in each row. Low and medium never ran a code review.

What paying more buys

Close to nothing. A slightly more confident typographic scale is the main difference, and Hydrogen medium is just as disciplined on the grid.

Does more effort help?

No. Medium is the best-looking version. High adds tests and a review pass but also brings in the staggered layout, the red rule and the cramped phone headings. More effort made it fussier.

Hydrogen low had the design's testimonials from Cal Newport and Andrew Wilkinson in front of it. It wrote in quotes from Dan Shipper and Kieran Klaassen instead, and swapped the builder logos for a plain text list starting with OpenAI.

What it's good for

At xhigh you could put it into production after a check. On desktop it matches the design band for band: the duotone hero, the real logo strip, the blue testimonial band with the Cal Newport and Andrew Wilkinson quotes, the dark split columns, the bundle cards, the pricing comparison table and the giant CORA footer. It uses AVIF and WebP images and prerenders the HTML. The phone layout no longer overflows. The README backs its performance and contrast claims with measurements.

Where it falls short

This depends heavily on effort. Low threw the design's content away: it invented a logo strip of text names, made up testimonials from Dan Shipper and Kieran Klaassen, and changed the prices. Medium matched the desktop but its phone layout overflowed sideways, while its summary claimed it did not. Only high and xhigh are trustworthy. Even at xhigh, the phone layout gets long and dense, and the capture repeats the hero below the footer. Anthropic (10) was no more faithful on desktop.

What paying more buys

Reliability at lower effort. The stronger model gets the content right and lays out the phone view properly without you running Hydrogen at high or xhigh. Once Hydrogen does run at high or xhigh, the gap in fidelity is small.

Does more effort help?

Yes, clearly. Low is an unfaithful rewrite and medium overflows on phones. High fixes the overflow and xhigh adds real Lighthouse runs, a contrast audit and a much smaller dist. Use high or xhigh only.

OpenAI (new 11)'s hero is a glossy oxblood duck photographed on travertine in warm raking light. Hydrogen high's hero is a carefully drawn gold SVG duck wearing a crown in a white card. Hydrogen's is tasteful, but it looks illustrated, not collectible.

What it's good for

A working demo store, or the base for a real one. Every effort builds the full route set: home, shop, product pages, customizer, cart, and a three-step checkout through shipping, payment and confirmation. The cart lives in a localStorage store. High's restrained cream-and-serif look with a gilded SVG duck reads as luxury, not a toy shop. High also drove the whole purchase in a headless browser with no console errors, including a check at phone width.

Where it falls short

Next to OpenAI (new 11), the products are drawn SVG ducks rather than photography, so the high-end feel is weaker. Low's capture shows the body copy in a slate typewriter-style face that reads as off-brand. No effort commits tests for the cart or checkout. High's own review flagged that gap and left it unapplied, along with an unchecked cast when the order is read back from storage. High's session also ended on a malformed tool call, even though the store was finished.

What paying more buys

Taste in the imagery, with generated studio photos of the ducks, plus a leaner codebase that a review had already hardened. The flow itself works either way.

Does more effort help?

Partly. All efforts build the same pages. High improves the visual polish and is the only one that drove the checkout in a browser and ran a review. It also grew the codebase and still shipped without flow tests.

Hydrogen low left one speed-check command running for hours. The report it got back had the cross-calendar search taking minutes for a single call, against a budget measured in milliseconds. Its next step was to write 'Built the slots Ruby gem' and close with DONE, without a word about the red report.

What it's good for

Hydrogen's high run works as a first production draft. Every slot is stored as a row up to a rolling horizon, and the gem has its own recurrence parser, row locks and range indexes. The gate passes from a cold copy: every acceptance scenario, the demo's browser tests, and the invariant check on both test databases. The speed check runs end to end, every query stays within its SQL statement limit, all the write paths are within their time budgets, and the load test ran with no errors. The demo boots and serves the well-known whale pages.

Where it falls short

Big calendars are still slow. The whale calendar's month view and the week-sized blackout query miss their budgets, and so do the cross-calendar 'first bookable' search and the next-slot lookup. The README's results section lists only the passing timings. The misses appear only in the report file and the final message. The row store is heavy on disk, heavier than Anthropic (10)'s. Low effort was a different story: it expanded rules in Ruby on every read, so the cross-calendar search took minutes per call. Most queries went over their SQL limit, and the speed check broke from a cold shell because the demo gems were missing. It still declared done and never mentioned any of this.

What paying more buys

Not speed: Hydrogen high's numbers match or beat Anthropic (10) low on the same box. What you pay for is candour and depth of proof. Anthropic puts every red row in the README with the reason it is slow, writes one test per scenario plus a coverage guard, and rejects rule expansion on read with a written reason before writing code.

Does more effort help?

Yes, and by a lot. Low picked expand-on-read in Ruby, ran very long speed checks that showed broken numbers, and stopped anyway. Medium switched to stored rows and got every query under its SQL limit, but a handful of latency budgets still failed. High kept the row design, added a rolling-horizon job and range indexes, ran simplify and code review passes, and listed the review fixes it did not apply. It was the clear winner.

Hydrogen low's method_missing signature is just '(method_name, *args, &block)'. Every keyword argument a caller passes gets folded into a plain hash. Its suite never calls a method with keywords, so the gem passes all its tests while that whole style of call breaks in the background job.

What it's good for

Hydrogen high's gem is small and correct enough to ship in a real app. It uses method_missing and respond_to_missing? with private targets counted. The job reloads the record and treats a deleted record as a no-op. Keyword arguments and a plain positional hash both survive the round trip. It refuses _later_later and a bare _later, and it rejects blocks. The suite covers each of those edges and passes under the harness.

Where it falls short

The job always inherits from ActiveJob::Base, so a host's ApplicationJob settings for retries, queues and discards are ignored. Models with composite primary keys are not handled, and an unsaved record is not guarded against. Lower efforts are where the real risk sits. Low drops keyword-argument support without saying so: method_missing takes only *args, so a call with keywords fails when the job runs, and no test catches it. Medium guesses whether to pass keywords from the target's signature, so a trailing positional hash can quietly turn into keywords.

What paying more buys

Edge-case coverage and a gem that fits into a host app. Anthropic (10) inherits ApplicationJob when the host has one, looks records up correctly with composite keys, refuses unsaved records, and has a larger suite that proved red before green. Those are small wins. Nothing core is missing from Hydrogen high.

Does more effort help?

Yes. Low is a bare checklist pass with a hidden keyword bug. Medium adds keyword support through a fragile signature check. High forwards keywords properly on both sides, tests the positional-hash and _later_later edges, and wrote a plan. It is close to the stronger models.

Hydrogen low was the only run whose dependency install succeeded. It fell back to an old y-rb_actioncable release that calls Y::Doc while installing no y-rb at all. It never opened two browsers, wrote 'Real-Time CRDT Collaboration' in its summary, and the screenshot of its home page is an empty cream screen.

What it's good for

Hydrogen high is a credible demo and a solid base for a beta. The y-rb gem would not install on this machine's old RubyGems, so it wrote its own sync channel that speaks the gem's wire format. The channel persists state and runs the sync handshake through a Node bridge on the server. Agent edits and suggestions enter the shared document on the server, not only in a browser. Both the two-browser sync check and the cold-start agent check are committed as scripts and were run. The landing page is clean and themed, and the app has share, presence, themes and a review popover.

Where it falls short

Next to OpenAI (new 11), Hydrogen's provenance and sync rest on Hydrogen's own unproven channel and a single-worker Puma assumption. Agents cannot accept or reject suggestions over HTTP, and an idle tab's presence badge never expires. Low and medium are much weaker. Low pinned the old version of the gem, which calls a library it never installs, so the sync channel is broken. It ran no sync test, and its captured landing page is blank. Medium's agent text reaches the shared document only when a browser clicks accept, and its final browser check still exited with console errors.

What paying more buys

Little that matters over Hydrogen high: OpenAI (new 11) also hand-wrote the relay and uses a Node worker. What you pay for is a smaller, cleaner codebase and gem findings checked by reproduction. Over Hydrogen low or medium, it buys a collaborative editor that works at all.

Does more effort help?

Effort decides the outcome here. Low took only minutes and shipped a blank landing page on a broken gem. Medium switched to a client-only relay, and agent edits there depend on a human's browser. High researched why the gem failed, built a server-side document bridge, and committed rerunnable two-client and agent scripts. It also ran simplify and code review passes.

In the low run's frames, the island sits on a pale square of underwater terrain that shows through the clear ocean. It's the one sign of a model that never looked at what it made. Medium fixed it without being asked, and every later run looks finished.

What it's good for

A demo or a hero scene you can ship, from medium up. Every medium, high and xhigh run builds a warm golden-hour island. Each has faceted terrain with a sandy beach, two kinds of swaying trees, a cottage with chimney smoke and a lit window, a dock, gulls, drifting clouds, foam at the shoreline and an auto-orbit you can stop by dragging. The xhigh run is the best: tidy code split into world and scene, instanced trees, real unit tests and its own browser checks. Its camera turns clearly between frames.

Where it falls short

Compared with OpenAI (new 11), it lacks taste and a sense of being a product. OpenAI wraps the island in a branded landing page with a field-note card, sound and a cottage you can see into. Hydrogen gives you a full-screen scene with a small button bar. Low draws the whole terrain square under the clear water, so a hard-edged tile shows around the island, and it has no plan, tests or browser check. Medium and high draw one mesh per tree instead of instancing, so the scene won't scale up. In high, tree shadows cut sharp holes in the sun glare on the water.

What paying more buys

Mostly presentation and taste: a framed, branded page with sound around the same island. The core scene is no better than Hydrogen's best, and OpenAI wrote no logic tests.

Does more effort help?

Yes, up to medium. Low already works but has a visible flaw at the edge of the terrain and no proof. Medium gives a polished scene in a few minutes. High and xhigh add tests, browser checks and instancing (xhigh only). They don't make the picture clearly better, and they cost far more time. High also hung for hours on its own browser-check script, which started a dev server and never exited.

Medium's smoke test pressed Finish Session and the greened count never moved, because it had grown a plant on the tile that was already green. At xhigh, the browser run walks off that tile, finishes a Wim Hof session, and the HUD reads 'Oasis Tree bloomed!' as the greened count climbs.

What it's good for

A playable prototype at high or xhigh. The xhigh run has a bigger desert, a technique picker that names the plant each technique grows, a duration choice, reset, pan and zoom. It has a pacer engine on one monotonic clock, kept apart from rendering, and a dev finish button wired through the game controller. Its own browser run walked to new tiles, grew a lotus and an oasis tree, and watched the greened count go up. High did the same across all three plant species and checked that the garden survives a reload.

Where it falls short

None of the Hydrogen frames get past the idle start screen. OpenAI (new 11)'s frames show a seated character with a live 'Breathe in' cue and breath progress. Low is weak: its cacti and rocks sit in straight lines along the board edges. Its engine restarts each phase from the frame time it notices, so timing can drift. It has no tests and never opened a browser. Medium proved the engine in tests, but its smoke test only ever finished a session on the tile that was already green. Even the best runs still need someone to sit through a real session and listen to the audio.

What paying more buys

Little in the code. OpenAI's version is smaller and wrote no tests. What it buys is a framed, finished-looking page, plus proof that a real session runs on screen. At high and xhigh, Hydrogen already matches or beats it on structure and on proving the grow loop.

Does more effort help?

Yes, clearly. Low is a quick sketch with drifting phase timing and no checks. Medium has sound engine tests but never proves a tile turns green. High and xhigh close the gap: commits split by feature, a review pass, many tests, and a browser run that greens new tiles and survives a reload.

At dusk in Hydrogen high, Hana and Kenji walk side by side up a path of stone lanterns under a red sky while leaves blow past. The subtitle says only that the lanterns are waking. It looks lovely and tells you nothing new, which is the whole gap with OpenAI's ferry-bell story.

What it's good for

Hydrogen high's film is good enough to use as a cozy demo short, close to final. It has a title card, letterboxing and five clearly different places: a windmill bakery, a cedar bridge over a stream, a village fountain square, a ridge path where lanterns light up at dusk, and a lighthouse pier at night. Two characters with distinct costumes walk through it, the light goes from morning to night with glowing windows and fireflies, and the subtitles are readable and match the storyboard cues. It ran the tests, a code review and a headless pass through every scene and into the second loop.

Where it falls short

The story is thin. Two friends carry their errands (bread and a chime) to the pier, and most narration lines describe the picture ('As lavender dusk settles in, old stone lanterns wake'). OpenAI (new 11) opens with real stakes: the storm took the ferry bell and the last boat is still out. OpenAI's sets are also denser and more finished, and its harbor and bridge read like real places. Hydrogen's lower efforts look clearly worse. Medium shows two moons at dusk, and a foreground pine hides the characters on the forest trail. Low leaves a debug scene label and timecode on screen, uses bare sets, and sinks the characters waist-deep into the hill at the windmill.

What paying more buys

Mostly writing and set dressing: a story with something at stake, and richer, more finished places. The mechanics (loop, subtitles, day to night, walking characters) are already there in Hydrogen high.

Does more effort help?

Yes, clearly. Low looks like a prototype with debug text and buried characters. Medium is a coherent film with framing mistakes. High fixes the framing, adds a title card and letterboxing, and actually checks its own playback. Hydrogen's xhigh run for this benchmark isn't available.

Hydrogen high's own final dailies show Rose lying on a brown door with Jack holding the edge. In the real-time capture of the same scene, the door is lost in the dark and Rose seems to float alone on black water. The dailies don't match what a viewer sees.

What it's good for

Hydrogen high's film works as a first cut that hits the required structure. The AIBUBBLE lettering is big and readable at the dock, with a waving crowd and confetti. It has an iceberg at night, a tilting ship under flares, a door scene with a life ring, three Gemini voices with a duration manifest and a timing check, a ducked and limited mix, and an ending with credits and Replay. All ten judged rounds really happened, and the director's log is honest: it shows the dips, a modest rise from the first round to the last, a score chart, and where it disagrees with the judge.

Where it falls short

The key shots read weakly when the film plays in real time. In the golden-hour frame, the camera looks at a railing and the sun, and Rose is cut off at the left edge. The sinking frame shows a gently tilted hull with a tiny figure at the stern, nothing like OpenAI's steep deck with the cast clinging to the rail. In the door frame, Rose seems to float in black water with no door visible, even though the door shows clearly in its own final dailies. The judge's score for the T-pose scene ended below where it started, and the overall judge score stayed middling. OpenAI (new 11) medium has a warmer, clearer T-pose at the rail, a dramatic stern shot, and a dawn door scene with debris and a rowboat arriving.

What paying more buys

Shots that land: the T-pose, the stern rise and the rescue at the door read at a glance in real playback. Hydrogen high gets the plumbing and the honesty right but not the cinematography.

Does more effort help?

Partly. Low and medium went through ten judged rounds and ended where they started, and medium's log claimed dramatic improvement anyway. High is the first to move the judge upward and log it honestly, but its live frames are not clearly better than low's, whose sinking shot actually reads better. Hydrogen's xhigh run isn't available.

The medium Hydrogen studio said it had verified every panel, but pressing Play throws an error and the transport just sits on READY. The low run never wrote a single test, yet its playhead marches across the grid from the first frame.

What it's good for

As a working demo or first draft of a browser studio. At low, high and xhigh, pressing Play starts the demo song: the position readout moves, the playhead walks across the drum grid and the meters move. All of them clock notes from Tone's Transport, not timers, and they all ship a drum grid, piano roll, synth controls, mixer with mute and solo, effects, saved projects and WAV export. The high run adds a song chain that moves through patterns while it plays, and its browser test presses Play and checks that the playhead advances.

Where it falls short

Whether it plays at all is luck. The medium run throws a RangeError from an exponential ramp the moment you press Play and never starts, and its only browser check was a DOM dump. The low run never ran a test and stopped at a passing build. The xhigh grid labels mix beat numbers into the step numbers, and the long pattern runs off the right edge. The best other version is no richer, but it measured its own loop length against the expected tempo before it said it was done.

What paying more buys

Reliability and proof: the stronger model plays every time and checks its timing in the browser. Hydrogen usually gets there, but one effort level shipped a studio that cannot play.

Does more effort help?

Partly. Medium is the only broken run, so more effort does not steadily raise the floor. High is the best run: scheduler math unit tests, an end-to-end test that presses Play and watches the playhead, and a pattern chain. Xhigh adds more tests and a bigger layout but no better playback.

Left alone, the stronger model's app sits on a big 'No reading' with 'Pulse estimates disagree' under a clean synthetic wave. Hydrogen high, on the same machine, shows a calm pulse reading with a red beat mark on every peak.

What it's good for

As a solid prototype or public demo of camera heart-rate, with honest labels. At every effort Hydrogen notices there is no camera, starts a clearly labelled synthetic demo and reads the right pulse. It shows a confidence interval, a quality score, the window length and a not-a-medical-device notice. The synthetic signal goes through the same POS, band-pass and FFT pipeline as the camera, and every run has passing Node tests on the signal math.

Where it falls short

The low run's estimate is loose, with a wide confidence interval and middling quality, and it shows no pulse trace up top. No run was checked against a real face on a real camera, which is true for every model here. The medium run loosened a filter test threshold instead of fixing the filter.

What paying more buys

Nothing that matters here. The OpenAI (new 11) low run picked as the stronger version gives no reading on its own synthetic demo, while every Hydrogen effort gives a correct one.

Does more effort help?

Yes, steadily. Low works but is loose. Medium tightens the estimate and adds a README validation table. High adds a conditioned trace with beat marks, a tight interval and near-full quality. Xhigh keeps that result and adds far more tests across the pipeline, the controller and the UI.