Graphics protocol: Support for positioning images relative to other images

Fixes #6617

docs/changelog.rst

@@ -52,6 +52,8 @@ Detailed list of changes
 
 - A new :doc:`escape code <pointer-shapes>` that can be used by programs running in the terminal to change the shape of the mouse pointer (:iss:`6711`)
 
+- Graphics protocol: Support for positioning :ref:`images relative to other images <relative_image_placement>` (:iss:`6400`)
+
 - A new option :opt:`single_window_padding_width` to use a different padding when only a single window is visible (:iss:`6734`)
 
 - A new mouse action ``mouse_selection word_and_line_from_point`` to select the current word under the mouse cursor and extend to end of line (:pull:`6663`)

docs/graphics-protocol.rst

@@ -640,6 +640,58 @@ placements from the protocol perspective. They cannot be manipulated using
 graphics commands, instead they should be moved, deleted, or modified by
 manipulating the underlying Unicode placeholder as normal text.
 
+.. _relative_image_placement:
+
+Relative placements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 0.31.0
+   Support for positioning images relative to other images
+
+You can specify that a placement is positioned relative to another placement.
+This is particularly useful in combination with
+:ref:`graphics_unicode_placeholders` above. It can be used to specify a single
+transparent pixel image using a Unicode placeholder, which moves around
+naturally with the text, the real image(s) can base their position relative to
+the placeholder.
+
+To specify that a placement should be relative to another, use the
+``P=<image_id>,Q=<placement_id>`` keys, when creating the relative placement.
+For example::
+
+    <ESC>_Ga=p,i=<image_id>,p=<placement_id>,P=<parent_img_id>,Q=<parent_placement_id><ESC>\
+
+This will create a *relative placement* that refers to the *parent placement* specified
+by the ``P`` and ``Q`` keys. When the parent placement moves, the relative
+placement moves along with it. The relative placement can be offset from the
+parent's location by a specified number of cells, using the ``H`` and ``V``
+keys for horizontal and vertical displacement. Positive values move right and
+down. Negative values move left and up.
+
+The lifetime of a relative placement is tied to the lifetime of its parent. If
+its parent is deleted, it is deleted as well. If the image that the relative
+placement is a placement of, has no more placements, the image is deleted as
+well. Thus, a parent and its relative placements form a *group* that is managed
+together.
+
+A relative placement can refer to another relative placement as its parent.
+Thus the relative placements can form a chain. It is implementation dependent
+how long a chain of such placements is allowed, but implementation must allow
+a chain of length at least 8. If the implementation max depth is exceeded, the
+terminal must respond with the ``ETOODEEP`` error code.
+
+Virtual placements created for Unicode placeholder based images cannot also be
+relative placements. However, a relative placement can refer to a virtual
+placement as its parent. If a client attempts to make a virtual placement
+relative the terminal must respond with the ``EINVAL`` error code.
+
+Terminals are required to reject the creation of a relative placement
+that would create a cycle, such as when A is relative to B and B is relative to
+C and C is relative to A. In such cases, the terminal must respond with the
+``ECYCLE`` error code.
+
+If a client attempts to create a reference to a placement that does not exist
+the terminal must respond with the ``ENOPARENT`` error code.
 
 Deleting images
 ---------------------
@@ -962,6 +1014,10 @@ Key      Value                 Default    Description
 ``U``    Positive integer      ``0``      Set to ``1`` to create a virtual placement for a Unicode placeholder.
                                           ``1`` is to not move the cursor at all when placing the image.
 ``z``    32-bit integer        ``0``      The *z-index* vertical stacking order of the image
+``P``    Positive integer      ``0``      The id of a parent image for relative placement
+``Q``    Positive integer      ``0``      The id of a placement in the parent image for relative placement
+``H``    32-bit integer        ``0``      The offset in cells in the horizontal direction for relative placement
+``V``    32-bit integer        ``0``      The offset in cells in the vertical direction for relative placement
 
 **Keys for animation frame loading**
 -----------------------------------------------------------

gen/apc_parsers.py

@@ -282,6 +282,10 @@ def graphics_parser() -> None:
         'z': ('z_index', 'int'),
         'C': ('cursor_movement', 'uint'),
         'U': ('unicode_placement', 'uint'),
+        'P': ('parent_id', 'uint'),
+        'Q': ('parent_placement_id', 'uint'),
+        'H': ('offset_from_parent_x', 'int'),
+        'V': ('offset_from_parent_y', 'int'),
     }
     text = generate('parse_graphics_code', 'screen_handle_graphics_command', 'graphics_command', keymap, 'GraphicsCommand')
     write_header(text, 'kitty/parse-graphics-command.h')

kitty/graphics.c

@@ -65,6 +65,7 @@ next_id(id_type *counter) {
     if (UNLIKELY(ans == 0)) ans = ++(*counter);
     return ans;
 }
+static const unsigned PARENT_DEPTH_LIMIT = 8;
 
 GraphicsManager*
 grman_alloc(void) {
@@ -140,14 +141,14 @@ dealloc(GraphicsManager* self) {
 }
 
 static Image*
-img_by_internal_id(GraphicsManager *self, id_type id) {
+img_by_internal_id(const GraphicsManager *self, id_type id) {
     Image *ans;
     HASH_FIND(hh, self->images, &id, sizeof(id), ans);
     return ans;
 }
 
 static Image*
-img_by_client_id(GraphicsManager *self, uint32_t id) {
+img_by_client_id(const GraphicsManager *self, uint32_t id) {
     for (Image *img = self->images; img != NULL; img = img->hh.next) {
         if (img->client_id == id) return img;
     }
@@ -155,7 +156,7 @@ img_by_client_id(GraphicsManager *self, uint32_t id) {
 }
 
 static Image*
-img_by_client_number(GraphicsManager *self, uint32_t number) {
+img_by_client_number(const GraphicsManager *self, uint32_t number) {
     // get the newest image with the specified number
     Image *ans = NULL;
     for (Image *img = self->images; img != NULL; img = img->hh.next) {
@@ -164,6 +165,24 @@ img_by_client_number(GraphicsManager *self, uint32_t number) {
     return ans;
 }
 
+static ImageRef*
+ref_by_internal_id(const Image *img, id_type id) {
+    ImageRef *ans;
+    HASH_FIND(hh, img->refs, &id, sizeof(id), ans);
+    return ans;
+}
+
+
+static ImageRef*
+ref_by_client_id(const Image *img, uint32_t id) {
+    for (ImageRef *ref = img->refs; ref != NULL; ref = ref->hh.next) {
+        if (ref->client_id == id) {
+            return ref;
+        }
+    }
+    return NULL;
+}
+
 static void
 remove_image(GraphicsManager *self, Image *img) {
     free_image(self, img);
@@ -709,6 +728,9 @@ create_ref(Image *img, ImageRef *clone_from) {
     return ans;
 }
 
+static inline bool
+is_cell_image(const ImageRef *self) { return self->virtual_ref_id != 0; }
+
 // Create a real image ref for a virtual image ref (placement) positioned in the
 // given cells. This is used for images positioned using Unicode placeholders.
 //
@@ -761,7 +783,7 @@ Image *grman_put_cell_image(GraphicsManager *self, uint32_t screen_row,
     // Create the ref structure on stack first. We will not create a real
     // reference if the image is completely out of bounds.
     ImageRef ref = {0};
-    ref.is_cell_image = true;
+    ref.virtual_ref_id = virt_img_ref->internal_id;
 
     uint32_t img_rows = virt_img_ref->num_rows;
     uint32_t img_columns = virt_img_ref->num_cols;
@@ -877,26 +899,93 @@ Image *grman_put_cell_image(GraphicsManager *self, uint32_t screen_row,
     return img;
 }
 
+static void remove_ref(Image *img, ImageRef *ref);
+
+static bool
+has_good_ancestry(GraphicsManager *self, ImageRef *ref) {
+    ImageRef *r = ref;
+    unsigned depth = 0;
+    while (r->parent.img) {
+        if (r == ref && depth) {
+            set_command_failed_response("ECYCLE", "This parent reference creates a cycle");
+            return false;
+        }
+        if (depth++ >= PARENT_DEPTH_LIMIT) {
+            set_command_failed_response("ETOODEEP", "Too many levels of parent references");
+            return false;
+        }
+        Image *parent = img_by_internal_id(self, r->parent.img);
+        if (!parent) {
+            set_command_failed_response("ENOENT", "One of the ancestors of this ref with image id: %u not found", r->parent.img);
+            return false;
+        }
+        ImageRef *parent_ref = ref_by_internal_id(parent, r->parent.ref);
+        if (!parent_ref) {
+            set_command_failed_response("ENOENT", "One of the ancestors of this ref with image id: %u and ref id: %u not found", r->parent.img, r->parent.ref);
+            return false;
+        }
+        r = parent_ref;
+    }
+    return true;
+}
+
 static uint32_t
 handle_put_command(GraphicsManager *self, const GraphicsCommand *g, Cursor *c, bool *is_dirty, Image *img, CellPixelSize cell) {
+    if (g->unicode_placement && g->parent_id) {
+        set_command_failed_response("EINVAL", "Put command creating a virtual placement cannot refer to a parent"); return g->id;
+    }
     if (img == NULL) {
         if (g->id) img = img_by_client_id(self, g->id);
         else if (g->image_number) img = img_by_client_number(self, g->image_number);
         if (img == NULL) { set_command_failed_response("ENOENT", "Put command refers to non-existent image with id: %u and number: %u", g->id, g->image_number); return g->id; }
     }
     if (!img->root_frame_data_loaded) { set_command_failed_response("ENOENT", "Put command refers to image with id: %u that could not load its data", g->id); return img->client_id; }
-    *is_dirty = true;
+    id_type parent_id = 0, parent_placement_id = 0;
-    self->layers_dirty = true;
+    if (g->parent_id) {
+        Image *parent = img_by_client_id(self, g->parent_id);
+        if (!parent) {
+            set_command_failed_response("ENOPARENT", "Put command refers to a parent image with id: %u that does not exist", g->parent_id);
+            return g->id;
+        }
+        if (!parent->refs) {
+            set_command_failed_response("ENOPARENT", "Put command refers to a parent image with id: %u that has no placements", g->parent_id);
+            return g->id;
+        }
+        ImageRef *parent_ref = parent->refs;
+        if (g->parent_placement_id) {
+            parent_ref = ref_by_client_id(parent, g->parent_placement_id);
+            if (!parent_ref) {
+                set_command_failed_response("ENOPARENT", "Put command refers to a parent image placement with id: %u and placement id: %u that does not exist", g->parent_id, g->parent_placement_id);
+                return g->id;
+            }
+        }
+        parent_id = parent->internal_id;
+        parent_placement_id = parent_ref->internal_id;
+    }
     ImageRef *ref = NULL;
     if (g->placement_id && img->client_id) {
         for (ImageRef *r = img->refs; r != NULL; r = r->hh.next) {
             if (r->client_id == g->placement_id) {
                 ref = r;
+                if (parent_id && parent_id == img->internal_id && parent_placement_id && parent_placement_id == r->internal_id) {
+                    set_command_failed_response("EINVAL", "Put command refers to itself as its own parent");
+                    return g->id;
+                }
+                if (parent_id && parent_placement_id) {
+                    id_type rp = ref->parent.img, rpp = ref->parent.ref;
+                    ref->parent.img = parent_id; ref->parent.ref = parent_placement_id;
+                    bool ok = has_good_ancestry(self, ref);
+                    ref->parent.img = rp; ref->parent.ref = rpp;
+                    if (!ok) return g->id;
+                }
                 break;
             }
         }
     }
     if (ref == NULL) ref = create_ref(img, NULL);
+
+    *is_dirty = true;
+    self->layers_dirty = true;
     img->atime = monotonic();
     ref->src_x = g->x_offset; ref->src_y = g->y_offset; ref->src_width = g->width ? g->width : img->width; ref->src_height = g->height ? g->height : img->height;
     ref->src_width = MIN(ref->src_width, img->width - ((float)img->width > ref->src_x ? ref->src_x : (float)img->width));
@@ -909,13 +998,25 @@ handle_put_command(GraphicsManager *self, const GraphicsCommand *g, Cursor *c, b
     if (img->client_id) ref->client_id = g->placement_id;
     update_src_rect(ref, img);
     update_dest_rect(ref, g->num_cells, g->num_lines, cell);
+    ref->parent.img = parent_id;
+    ref->parent.ref = parent_placement_id;
+    ref->parent.offset.x = g->offset_from_parent_x;
+    ref->parent.offset.y = g->offset_from_parent_y;
+    ref->is_virtual_ref = false;
     if (g->unicode_placement) {
         ref->is_virtual_ref = true;
         ref->start_row = ref->start_column = 0;
     }
-    // Move the cursor, the screen will take care of ensuring it is in bounds
+    if (ref->parent.img) {
-    if (g->cursor_movement != 1 && !g->unicode_placement) {
+        if (!has_good_ancestry(self, ref)) {
-        c->x += ref->effective_num_cols; c->y += ref->effective_num_rows - 1;
+            remove_ref(img, ref);
+            return g->id;
+        }
+    } else {
+        // Move the cursor, the screen will take care of ensuring it is in bounds
+        if (g->cursor_movement != 1 && !g->unicode_placement) {
+            c->x += ref->effective_num_cols; c->y += ref->effective_num_rows - 1;
+        }
     }
     return img->client_id;
 }
@@ -940,6 +1041,48 @@ gpu_data_for_image(ImageRenderData *ans, float left, float top, float right, flo
     ans->group_count = 1;
 }
 
+static bool
+resolve_cell_ref(const Image *img, id_type virt_ref_id, int32_t *start_row, int32_t *start_column) {
+    *start_row = 0; *start_column = 0;
+    bool found = false;
+    for (ImageRef *ref = img->refs; ref != NULL; ref = ref->hh.next) {
+        if (ref->virtual_ref_id == virt_ref_id) {
+            if (!found || ref->start_row < *start_row) *start_row = ref->start_row;
+            if (!found || ref->start_column < *start_column) *start_column = ref->start_column;
+            found = true;
+
+        }
+    }
+    return found;
+}
+
+static bool
+resolve_parent_offset(const GraphicsManager *self, const ImageRef *ref, int32_t *start_row, int32_t *start_column, bool *is_virtual_ref) {
+    *start_row = 0; *start_column = 0;
+    int32_t x = 0, y = 0;
+    unsigned depth = 0;
+    ImageRef cell_ref = {0};
+    while (ref->parent.img) {
+        if (depth++ >= PARENT_DEPTH_LIMIT) return false;  // either a cycle or too many ancestors
+        Image *img = img_by_internal_id(self, ref->parent.img);
+        if (!img) return false;
+        ImageRef *parent = ref_by_internal_id(img, ref->parent.ref);
+        if (!parent) return false;
+        if (parent->is_virtual_ref) {
+            *is_virtual_ref = true;
+            if (!resolve_cell_ref(img, parent->internal_id, &cell_ref.start_row, &cell_ref.start_column)) return false;
+            parent = &cell_ref;
+        }
+        x += ref->parent.offset.x;
+        y += ref->parent.offset.y;
+        ref = parent;
+    }
+    *start_row = ref->start_row + y;
+    *start_column = ref->start_column + x;
+    return true;
+}
+
+
 bool
 grman_update_layers(GraphicsManager *self, unsigned int scrolled_by, float screen_left, float screen_top, float dx, float dy, unsigned int num_cols, unsigned int num_rows, CellPixelSize cell) {
     if (self->last_scrolled_by != scrolled_by) self->layers_dirty = true;
@@ -950,7 +1093,7 @@ grman_update_layers(GraphicsManager *self, unsigned int scrolled_by, float scree
     self->num_of_below_refs = 0;
     self->num_of_negative_refs = 0;
     self->num_of_positive_refs = 0;
-    Image *img; ImageRef *ref;
+    Image *img, *tmpimg; ImageRef *ref, *tmpref;
     ImageRect r;
     float screen_width = dx * num_cols, screen_height = dy * num_rows;
     float screen_bottom = screen_top - screen_height;
@@ -960,19 +1103,32 @@ grman_update_layers(GraphicsManager *self, unsigned int scrolled_by, float scree
 
     // Iterate over all visible refs and create render data
     self->render_data.count = 0;
-    for (img = self->images; img != NULL; img = img->hh.next) {
+
+    HASH_ITER(hh, self->images, img, tmpimg) {
         bool was_drawn = img->is_drawn;
+        bool ref_removed = false;
         img->is_drawn = false;
 
-        for (ref = img->refs; ref != NULL; ref = ref->hh.next) {
+        HASH_ITER(hh, img->refs, ref, tmpref) {
             if (ref->is_virtual_ref) continue;
-            r.top = y0 - ref->start_row * dy - dy * (float)ref->cell_y_offset / (float)cell.height;
+            int32_t start_row = ref->start_row, start_column = ref->start_column;
-            if (ref->num_rows > 0) r.bottom = y0 - (ref->start_row + (int32_t)ref->num_rows) * dy;
+            if (ref->parent.img) {
+                bool is_virtual_ref;
+                if (!resolve_parent_offset(self, ref, &start_row, &start_column, &is_virtual_ref)) {
+                    if (!is_virtual_ref) {
+                        remove_ref(img, ref);
+                        ref_removed = true;
+                    }
+                    continue;
+                }
+            }
+            r.top = y0 - start_row * dy - dy * (float)ref->cell_y_offset / (float)cell.height;
+            if (ref->num_rows > 0) r.bottom = y0 - (start_row + (int32_t)ref->num_rows) * dy;
             else r.bottom = r.top - screen_height * (float)ref->src_height / screen_height_px;
             if (r.top <= screen_bottom || r.bottom >= screen_top) continue;  // not visible
 
-            r.left = screen_left + ref->start_column * dx + dx * (float)ref->cell_x_offset / (float) cell.width;
+            r.left = screen_left + start_column * dx + dx * (float)ref->cell_x_offset / (float) cell.width;
-            if (ref->num_cols > 0) r.right = screen_left + (ref->start_column + (int32_t)ref->num_cols) * dx;
+            if (ref->num_cols > 0) r.right = screen_left + (start_column + (int32_t)ref->num_cols) * dx;
             else r.right = r.left + screen_width * (float)ref->src_width / screen_width_px;
 
             if (ref->z_index < ((int32_t)INT32_MIN/2))
@@ -986,10 +1142,14 @@ grman_update_layers(GraphicsManager *self, unsigned int scrolled_by, float scree
             zero_at_ptr(rd);
             rd->dest_rect = r; rd->src_rect = ref->src_rect;
             self->render_data.count++;
-            rd->z_index = ref->z_index; rd->image_id = img->internal_id;
+            rd->z_index = ref->z_index; rd->image_id = img->internal_id; rd->ref_id = ref->internal_id;
             rd->texture_id = img->texture_id;
             img->is_drawn = true;
         }
+        if (ref_removed && !img->refs) {
+            remove_image(self, img);
+            continue;
+        }
         if (img->is_drawn && !was_drawn && img->animation_state != ANIMATION_STOPPED && img->extra_framecnt && img->animation_duration) {
             self->has_images_needing_animation = true;
             global_state.check_for_active_animated_images = true;
@@ -1691,7 +1851,7 @@ grman_scroll_images(GraphicsManager *self, const ScrollData *data, CellPixelSize
 
 static bool
 cell_image_row_filter_func(const ImageRef *ref, Image UNUSED *img, const void *data, CellPixelSize cell UNUSED) {
-    if (ref->is_virtual_ref || !ref->is_cell_image)
+    if (ref->is_virtual_ref || !is_cell_image(ref))
         return false;
     int32_t top = *(int32_t *)data;
     int32_t bottom = *((int32_t *)data + 1);
@@ -1700,7 +1860,7 @@ cell_image_row_filter_func(const ImageRef *ref, Image UNUSED *img, const void *d
 
 static bool
 cell_image_filter_func(const ImageRef *ref, Image UNUSED *img, const void *data UNUSED, CellPixelSize cell UNUSED) {
-    return !ref->is_virtual_ref && ref->is_cell_image;
+    return !ref->is_virtual_ref && is_cell_image(ref);
 }
 
 // Remove cell images within the given region.
@@ -1726,7 +1886,7 @@ clear_filter_func(const ImageRef *ref, Image UNUSED *img, const void UNUSED *dat
 
 static bool
 clear_filter_func_noncell(const ImageRef *ref, Image UNUSED *img, const void UNUSED *data, CellPixelSize cell UNUSED) {
-    if (ref->is_virtual_ref || ref->is_cell_image) return false;
+    if (ref->is_virtual_ref || is_cell_image(ref)) return false;
     return ref->start_row + (int32_t)ref->effective_num_rows > 0;
 }
 
@@ -1758,21 +1918,21 @@ number_filter_func(const ImageRef *ref, Image *img, const void *data, CellPixelS
 
 static bool
 x_filter_func(const ImageRef *ref, Image UNUSED *img, const void *data, CellPixelSize cell UNUSED) {
-    if (ref->is_virtual_ref || ref->is_cell_image) return false;
+    if (ref->is_virtual_ref || is_cell_image(ref)) return false;
     const GraphicsCommand *g = data;
     return ref->start_column <= (int32_t)g->x_offset - 1 && ((int32_t)g->x_offset - 1) < ((int32_t)(ref->start_column + ref->effective_num_cols));
 }
 
 static bool
 y_filter_func(const ImageRef *ref, Image UNUSED *img, const void *data, CellPixelSize cell UNUSED) {
-    if (ref->is_virtual_ref || ref->is_cell_image) return false;
+    if (ref->is_virtual_ref || is_cell_image(ref)) return false;
     const GraphicsCommand *g = data;
     return ref->start_row <= (int32_t)g->y_offset - 1 && ((int32_t)g->y_offset - 1) < ((int32_t)(ref->start_row + ref->effective_num_rows));
 }
 
 static bool
 z_filter_func(const ImageRef *ref, Image UNUSED *img, const void *data, CellPixelSize cell UNUSED) {
-    if (ref->is_virtual_ref || ref->is_cell_image) return false;
+    if (ref->is_virtual_ref || is_cell_image(ref)) return false;
     const GraphicsCommand *g = data;
     return ref->z_index == g->z_index;
 }
@@ -1780,13 +1940,13 @@ z_filter_func(const ImageRef *ref, Image UNUSED *img, const void *data, CellPixe
 
 static bool
 point_filter_func(const ImageRef *ref, Image *img, const void *data, CellPixelSize cell) {
-    if (ref->is_virtual_ref || ref->is_cell_image) return false;
+    if (ref->is_virtual_ref || is_cell_image(ref)) return false;
     return x_filter_func(ref, img, data, cell) && y_filter_func(ref, img, data, cell);
 }
 
 static bool
 point3d_filter_func(const ImageRef *ref, Image *img, const void *data, CellPixelSize cell) {
-    if (ref->is_virtual_ref || ref->is_cell_image) return false;
+    if (ref->is_virtual_ref || is_cell_image(ref)) return false;
     return z_filter_func(ref, img, data, cell) && point_filter_func(ref, img, data, cell);
 }
 
@@ -1842,7 +2002,7 @@ grman_resize(GraphicsManager *self, index_type old_lines UNUSED, index_type line
         const unsigned int vertical_shrink_size = num_content_lines_before - num_content_lines_after;
         for (img = self->images; img != NULL; img = img->hh.next) {
             for (ref = img->refs; ref != NULL; ref = ref->hh.next) {
-                if (ref->is_virtual_ref || ref->is_cell_image) continue;
+                if (ref->is_virtual_ref || is_cell_image(ref)) continue;
                 ref->start_row -= vertical_shrink_size;
             }
         }
@@ -1855,7 +2015,7 @@ grman_rescale(GraphicsManager *self, CellPixelSize cell) {
     self->layers_dirty = true;
     for (img = self->images; img != NULL; img = img->hh.next) {
         for (ref = img->refs; ref != NULL; ref = ref->hh.next) {
-            if (ref->is_virtual_ref || ref->is_cell_image) continue;
+            if (ref->is_virtual_ref || is_cell_image(ref)) continue;
             ref->cell_x_offset = MIN(ref->cell_x_offset, cell.width - 1);
             ref->cell_y_offset = MIN(ref->cell_y_offset, cell.height - 1);
             update_dest_rect(ref, ref->num_cols, ref->num_rows, cell);
@@ -2050,7 +2210,7 @@ W(update_layers) {
         ImageRenderData *r = self->render_data.item + i;
 #define R(which) Py_BuildValue("{sf sf sf sf}", "left", r->which.left, "top", r->which.top, "right", r->which.right, "bottom", r->which.bottom)
         PyTuple_SET_ITEM(ans, i,
-            Py_BuildValue("{sN sN sI si sK}", "src_rect", R(src_rect), "dest_rect", R(dest_rect), "group_count", r->group_count, "z_index", r->z_index, "image_id", r->image_id)
+            Py_BuildValue("{sN sN sI si sK sK}", "src_rect", R(src_rect), "dest_rect", R(dest_rect), "group_count", r->group_count, "z_index", r->z_index, "image_id", r->image_id, "ref_id", r->ref_id)
         );
 #undef R
     }

kitty/graphics.h

@@ -11,7 +11,7 @@
 
 typedef struct {
     unsigned char action, transmission_type, compressed, delete_action;
-    uint32_t format, more, id, image_number, data_sz, data_offset, placement_id, quiet;
+    uint32_t format, more, id, image_number, data_sz, data_offset, placement_id, quiet, parent_id, parent_placement_id;
     uint32_t width, height, x_offset, y_offset;
     union { uint32_t cursor_movement, compose_mode; };
     union { uint32_t cell_x_offset, blend_mode; };
@@ -23,6 +23,7 @@ typedef struct {
     union { int32_t z_index, gap; };
     size_t payload_sz;
     bool unicode_placement;
+    int32_t offset_from_parent_x, offset_from_parent_y;
 } GraphicsCommand;
 
 typedef struct {
@@ -36,13 +37,19 @@ typedef struct {
     int32_t start_row, start_column;
     uint32_t client_id;
     ImageRect src_rect;
-    // Indicates whether this reference represents a cell image that should be
+    // Indicates whether this reference represents a cell ref that should be
     // removed when the corresponding cells are modified.
-    bool is_cell_image;
+    // The internal id of the virtual ref this cell image was created from. Is a cell ref if this is non-zero.
+    id_type virtual_ref_id;
     // Virtual refs are not displayed but they can be used as prototypes for
     // refs placed using unicode placeholders.
     bool is_virtual_ref;
 
+    struct {
+        id_type img, ref;
+        struct { int32_t x, y; } offset;
+    } parent;
+
     id_type internal_id;
     hash_handle_type hh;
 } ImageRef;
@@ -86,7 +93,7 @@ typedef struct {
     ImageRect src_rect, dest_rect;
     uint32_t texture_id, group_count;
     int z_index;
-    id_type image_id;
+    id_type image_id, ref_id;
 } ImageRenderData;
 
 typedef struct {

kitty/parse-graphics-command.h

@@ -40,7 +40,11 @@ static inline void parse_graphics_code(Screen *screen,
     cell_y_offset = 'Y',
     z_index = 'z',
     cursor_movement = 'C',
-    unicode_placement = 'U'
+    unicode_placement = 'U',
+    parent_id = 'P',
+    parent_placement_id = 'Q',
+    offset_from_parent_x = 'H',
+    offset_from_parent_y = 'V'
   };
 
   enum KEYS key = 'a';
@@ -128,6 +132,18 @@ static inline void parse_graphics_code(Screen *screen,
       case unicode_placement:
         value_state = UINT;
         break;
+      case parent_id:
+        value_state = UINT;
+        break;
+      case parent_placement_id:
+        value_state = UINT;
+        break;
+      case offset_from_parent_x:
+        value_state = INT;
+        break;
+      case offset_from_parent_y:
+        value_state = INT;
+        break;
       default:
         REPORT_ERROR("Malformed GraphicsCommand control block, invalid key "
                      "character: 0x%x",
@@ -240,6 +256,8 @@ static inline void parse_graphics_code(Screen *screen,
       READ_UINT;
       switch (key) {
         I(z_index);
+        I(offset_from_parent_x);
+        I(offset_from_parent_y);
       default:
         break;
       }
@@ -273,6 +291,8 @@ static inline void parse_graphics_code(Screen *screen,
         U(cell_y_offset);
         U(cursor_movement);
         U(unicode_placement);
+        U(parent_id);
+        U(parent_placement_id);
       default:
         break;
       }
@@ -332,7 +352,7 @@ static inline void parse_graphics_code(Screen *screen,
 
   REPORT_VA_COMMAND(
       "s {sc sc sc sc sI sI sI sI sI sI sI sI sI sI sI sI sI sI sI sI sI sI sI "
-      "sI si sI} y#",
+      "sI sI sI si si si sI} y#",
       "graphics_command", "action", g.action, "delete_action", g.delete_action,
       "transmission_type", g.transmission_type, "compressed", g.compressed,
       "format", (unsigned int)g.format, "more", (unsigned int)g.more, "id",
@@ -347,8 +367,11 @@ static inline void parse_graphics_code(Screen *screen,
       (unsigned int)g.num_lines, "cell_x_offset", (unsigned int)g.cell_x_offset,
       "cell_y_offset", (unsigned int)g.cell_y_offset, "cursor_movement",
       (unsigned int)g.cursor_movement, "unicode_placement",
-      (unsigned int)g.unicode_placement, "z_index", (int)g.z_index,
+      (unsigned int)g.unicode_placement, "parent_id", (unsigned int)g.parent_id,
-      "payload_sz", g.payload_sz, payload, g.payload_sz);
+      "parent_placement_id", (unsigned int)g.parent_placement_id, "z_index",
+      (int)g.z_index, "offset_from_parent_x", (int)g.offset_from_parent_x,
+      "offset_from_parent_y", (int)g.offset_from_parent_y, "payload_sz",
+      g.payload_sz, payload, g.payload_sz);
 
   screen_handle_graphics_command(screen, &g, payload);
 }

kitty_tests/graphics.py

@@ -123,13 +123,15 @@ def put_helpers(self, cw, ch, cols=10, lines=5):
         return s, 2 / s.columns, 2 / s.lines
 
     def put_cmd(
-            z=0, num_cols=0, num_lines=0, x_off=0, y_off=0, width=0,
+        z=0, num_cols=0, num_lines=0, x_off=0, y_off=0, width=0, height=0, cell_x_off=0,
-            height=0, cell_x_off=0, cell_y_off=0, placement_id=0,
+        cell_y_off=0, placement_id=0, cursor_movement=0, unicode_placeholder=0, parent_id=0,
-            cursor_movement=0, unicode_placeholder=0
+        parent_placement_id=0, offset_from_parent_x=0, offset_from_parent_y=0,
     ):
-        return 'z=%d,c=%d,r=%d,x=%d,y=%d,w=%d,h=%d,X=%d,Y=%d,p=%d,C=%d,U=%d' % (
+        return (
-            z, num_cols, num_lines, x_off, y_off, width, height, cell_x_off,
+            f'z={z},c={num_cols},r={num_lines},x={x_off},y={y_off},w={width},h={height},'
-            cell_y_off, placement_id, cursor_movement, unicode_placeholder
+            f'X={cell_x_off},Y={cell_y_off},p={placement_id},C={cursor_movement},'
+            f'U={unicode_placeholder},P={parent_id},Q={parent_placement_id},'
+            f'H={offset_from_parent_x},V={offset_from_parent_y}'
         )
 
     def put_image(screen, w, h, **kw):
@@ -540,6 +542,105 @@ class TestGraphics(BaseTest):
         self.ae(put_image(s, 8, 16, id=2, z=-1)[1], 'OK')
         self.ae(group_counts(), (2, 1, 1, 2, 1))
 
+    def test_image_parents(self):
+        cw, ch = 10, 20
+        iw, ih = 10, 20
+        s, dx, dy, put_image, put_ref, layers, rect_eq = put_helpers(self, cw, ch)
+
+        def positions():
+            ans = {}
+            def x(x):
+                return round(((x + 1)/2) * s.columns)
+            def y(y):
+                return int(((-y + 1)/2) * s.lines)
+
+            for i in layers(s):
+                d = i['dest_rect']
+                ans[(i['image_id'], i['ref_id'])] = {'x': x(d['left']), 'y': y(d['top'])}
+            return ans
+
+        def p(x, y=0):
+            return {'x':x, 'y': y}
+
+        self.ae(put_image(s, iw, ih, id=1)[1], 'OK')
+        self.ae(put_ref(s, id=1, placement_id=1), (1, ('OK', 'i=1,p=1')))
+        pos = {(1, 1): p(0), (1, 2): p(1)}
+        self.ae(positions(), pos)
+        # check that adding a reference to a non-existent parent fails
+        self.ae(put_ref(s, id=1, placement_id=33, parent_id=1, parent_placement_id=2), (1, ('ENOPARENT', 'i=1,p=33')))
+        self.ae(put_ref(s, id=1, placement_id=33, parent_id=33), (1, ('ENOPARENT', 'i=1,p=33')))
+        # check that we cannot add a reference that is its own parent
+        self.ae(put_ref(s, id=1, placement_id=1, parent_id=1, parent_placement_id=1), (1, ('EINVAL', 'i=1,p=1')))
+
+        self.ae(put_image(s, iw, ih, id=2)[1], 'OK')
+        pos[(2,1)] = p(2)
+        self.ae(positions(), pos)
+        # Add two children to the first placement of img2
+        before = s.cursor.x, s.cursor.y
+        self.ae(put_ref(s, id=1, placement_id=2, parent_id=2, offset_from_parent_y=3), (1, ('OK', 'i=1,p=2')))
+        self.ae(before, (s.cursor.x, s.cursor.y), 'Cursor must not move for child image')
+        pos[(1,3)] = p(2, 3)
+        self.ae(positions(), pos)
+        self.ae(put_ref(s, id=2, placement_id=3, parent_id=2, offset_from_parent_y=4), (2, ('OK', 'i=2,p=3')))
+        pos[(2,2)] = p(2, 4)
+        self.ae(positions(), pos)
+        # Add a grand child to the second child of img2
+        self.ae(put_ref(s, id=2, placement_id=4, parent_id=2, parent_placement_id=3, offset_from_parent_x=-1), (2, ('OK', 'i=2,p=4')))
+        pos[(2,3)] = p(pos[(2,2)]['x']-1, pos[(2,2)]['y'])
+        self.ae(positions(), pos)
+        # Check that creating a cycle is prevented
+        self.ae(put_ref(s, id=2, placement_id=3, parent_id=2, parent_placement_id=4), (2, ('ECYCLE', 'i=2,p=3')))
+        self.ae(positions(), pos)
+        # Check that depth is limited
+        for i in range(5, 12):
+            q = put_ref(s, id=2, placement_id=i, parent_id=2, parent_placement_id=i-1, offset_from_parent_x=-1)[1][0]
+            if q == 'ETOODEEP':
+                break
+            self.ae(q, 'OK')
+        else:
+            self.assertTrue(False, 'Failed to limit reference chain depth')
+        # Check that deleting a parent removes all descendants
+        send_command(s, 'a=d,d=i,i=2,p=3')
+        pos.pop((2,3)), pos.pop((2,2))
+        self.ae(positions(), pos)
+        # Check that deleting a parent deletes all descendants and also removes
+        # images with no remaining placements
+        self.ae(put_ref(s, id=2, placement_id=3, parent_id=2, offset_from_parent_y=4), (2, ('OK', 'i=2,p=3')))
+        pos[(2,11)] = p(2, 4)
+        self.ae(positions(), pos)
+        self.ae(put_image(s, iw, ih, id=3, placement_id=97, parent_id=2, parent_placement_id=3)[1], 'OK')
+        pos[(3,1)] = p(2, 4)
+        self.ae(positions(), pos)
+        send_command(s, 'a=d,d=i,i=2')
+        pos.pop((3,1)), pos.pop((2,11)), pos.pop((2,1)), pos.pop((1,3))
+        self.ae(positions(), pos)
+        # Check that virtual placements that try to be relative are rejected
+        self.ae(put_ref(s, id=1, placement_id=11, parent_id=1, unicode_placeholder=1), (1, ('EINVAL', 'i=1,p=11')))
+        # Check creation of children of a unicode placeholder based image
+        s, dx, dy, put_image, put_ref, layers, rect_eq = put_helpers(self, cw, ch)
+        put_image(s, 20, 20, num_cols=4, num_lines=2, unicode_placeholder=1, id=42)
+        s.update_only_line_graphics_data()
+        self.assertFalse(positions())  # the reference is virtual
+        self.ae(put_ref(s, id=42, placement_id=11, parent_id=42, offset_from_parent_y=2, offset_from_parent_x=1), (42, ('OK', 'i=42,p=11')))
+        self.assertFalse(positions())  # the reference is virtual without any cell images so the child is invisible
+        s.apply_sgr("38;5;42")
+        # These two characters will become one 2x1 ref.
+        s.cursor.x = s.cursor.y = 1
+        s.draw("\U0010EEEE\u0305\u0305\U0010EEEE\u0305\u030D")
+        s.cursor.x = s.cursor.y = 0
+        s.draw("\U0010EEEE\u0305\u0305\U0010EEEE\u0305\u030D")
+        s.update_only_line_graphics_data()
+        pos = {(1, 2): p(1, 2), (1, 3): p(0), (1, 4): p(1)}
+        self.ae(positions(), pos)
+        s.cursor.x = s.cursor.y = 0
+        s.erase_in_display(0, False)
+        s.update_only_line_graphics_data()
+        self.assertFalse(positions())  # the reference is virtual without any cell images so the child is invisible
+        s.cursor.x = s.cursor.y = 2
+        s.draw("\U0010EEEE\u0305\u0305\U0010EEEE\u0305\u030D")
+        s.update_only_line_graphics_data()
+        self.ae(positions(), {(1, 5): {'x': 2, 'y': 2}, (1, 2): {'x': 3, 'y': 4}})
+
     def test_unicode_placeholders(self):
         # This test tests basic image placement using using unicode placeholders
         cw, ch = 10, 20

kitty_tests/parser.py

@@ -420,7 +420,9 @@ class TestParser(BaseTest):
             for f in 'action delete_action transmission_type compressed'.split():
                 k.setdefault(f, b'\0')
             for f in ('format more id data_sz data_offset width height x_offset y_offset data_height data_width cursor_movement'
-                      ' num_cells num_lines cell_x_offset cell_y_offset z_index placement_id image_number quiet unicode_placement').split():
+                      ' num_cells num_lines cell_x_offset cell_y_offset z_index placement_id image_number quiet unicode_placement'
+                      ' parent_id parent_placement_id offset_from_parent_x offset_from_parent_y'
+            ).split():
                 k.setdefault(f, 0)
             p = k.pop('payload', '').encode('utf-8')
             k['payload_sz'] = len(p)