The project notes summarize Image Generators are Generalist Vision Learners around one central question: whether strong image generators already contain rich visual structure priors that can be released through lightweight instruction tuning.
The paper assumes that high-quality generators have learned boundaries, spatial relations, scale, material, geometry, and semantics. Instruction tuning may expose those internal priors as vision-understanding ability.
Semantic segmentation, instance segmentation, depth, surface normals, and referring segmentation are unified as generated, decodable result images: input image plus prompt, generated map, then postprocessed structure.
Vision Banana can specify targets and colors through natural language, JSON, RGB tuples, or hex colors. In medicine, this must become controlled terminology, fixed palettes, and reproducible protocols.
In plain terms, conventional segmentation models behave like pixel-level classifiers: each pixel is judged as target, background, or a class label. Vision Banana-style methods ask an image generator to draw a decodable answer map: target regions, instances, depth, or normals are rendered with specified colors and decoded afterward.
This page summarizes several GPT-Images-2 probes. The natural portrait case produced a detailed RGB mask, but medical images showed registration drift and anatomical mismatch that are unacceptable for serious medical segmentation.
The key Vision Banana idea is to reframe vision tasks as image generation rather than traditional logit prediction: input an image and prompt, then output a visual, decodable RGB structure map.
Use boundaries, geometry, spatial relations, and semantic structure learned during image generation pretraining, then expose dense prediction through instruction tuning.
Semantic segmentation, instance segmentation, depth, and surface normals can all be represented as RGB result images and decoded into structured outputs.
Text specifies the target and color schema, such as background = black and target = yellow, followed by color projection and connected-component analysis.
Medical tasks must additionally handle absent targets, 3D continuity, uncertainty, color drift, and hallucinated anatomy. Visual appeal is not enough.
The images below come from the project directory. Failed examples are intentionally kept because they decide whether this direction is ready for serious medical research.
This case acts as a positive control for the natural-person task: both the source image and
GPT output are 1254 x 1254. The mask contour is usable, the estimated offset is
dx=-15px, dy=+50px, and the aligned result produces a clean
transparent PNG.
The GPT-Images-2 mask responds well to the person boundary and fine hair details. The main
issue is not contour quality, but global displacement. The estimated offset is
dx=-24px, dy=+52px.
This image is harder than the first two natural-person cases: the hand sign and foreground
sign are real occluders in front of the child, so removing all props would contradict the
visible image. The selected result uses the second-best mask that preserves foreground
occlusion, with a re-estimated offset of dx=-5px, dy=+25px.
It is useful for natural-image probing, but already requires judgment and postprocessing.
The result resembles a medical structure, but it is not a strict segmentation in the source-image coordinate system. It behaves more like a visually plausible spine shape, which is unacceptable for medical segmentation.
Compared with the X-ray case, vertebral block structures in the MRI are easier for the model to capture. However, after overlaying the result, vertebral edges, lower anatomy, and source-image boundaries still differ too much to serve as a medical mask.
The normal-style output enhances structures around vertebrae, discs, and the spinal canal, but it is not a physical surface normal. The palette-constrained version combines a GPT mask with a GPT normal map through postprocessing; it is not a single pure GPT output.
Good-looking natural-image contours do not imply medical usability. Medical imaging requires verifiable coordinates, boundaries, structure existence, and uncertainty.
GPT-Images-2 is useful for rapidly probing the Vision Banana paradigm, but zero-shot medical-image outputs should not be used as main paper results. Deeper research should convert the idea into a reproducible training framework and quantitative evaluation protocol.
The current results are suitable as a preliminary feasibility study and failure-mode note, but they are not enough to support a full medical-imaging paper. The following evidence is still missing.
The current work is mainly case-based. A paper-level claim needs tens to hundreds of medical images across modalities, acquisition quality, and anatomical variation.
Medical segmentation must be compared against expert annotations. Without ground truth, registration drift, boundary error, and missing structures cannot be measured rigorously.
The study needs Dice, IoU, Hausdorff distance, boundary error, centroid displacement, color decoding error, and failure rate, rather than visual inspection alone.
Prompts, output size, RGB palette, threshold decoding, connected-component filtering, and whether post-hoc translation is allowed must be fixed before evaluation.
The method should be compared with conventional medical segmentation models, general-purpose segmentation models, and generative structure-map baselines.
Observed failures include global offset, local deformation, occlusion handling, generative completion, and anatomical hallucination; their frequency and causes need dataset-level analysis.
If this becomes a paper project, the current results should not be packaged as a success demo. The value is in designing rigorous methods around the observed failure modes.
Vision Banana project page: vision-banana.github.io · Paper: Image Generators are Generalist Vision Learners