Embed images

Returns one embedding per image and, for discoverSniglets(), embeddings of local regions of each image.

Usage

embedImages(
  model,
  paths,
  region = NULL,
  patches = TRUE,
  tiles = 4,
  minDetail = 8,
  minFill = 0.5
)

Arguments

model

A visionModel().

paths

Image files, or an imageSet() (its path column is embedded and its id column names the rows).

region

Optional crop c(y0, y1, x0, x1) as fractions, applied to every image before embedding.

patches

If TRUE, also return the local embeddings.

tiles

Size of the grid of tiles: 4 cuts 4 x 4 windows that overlap their neighbours by half. Several sizes give several scales (c(2, 4)). 0 uses the model's patch tokens.

minDetail

Tiles with less detail than this (standard deviation of the grey levels, 0 to 255) are treated as empty background and match nothing.

minFill

In a photo with a plain background (one colour around most of its border), tiles in which the subject covers less than this share are treated as empty too, so that features are about the subject and not about its backdrop. It has no effect on photos without a plain background.

Value

A list: image (images x d matrix, rows named by image id: the file name without extension, or the id of an imageSet()) and patches (images x regions x d array, or NULL), which remembers the tiling in its tiles attribute.

Details

By default the regions are tiles: overlapping windows cut out of the image and embedded one by one, as if each were a whole image. A tile's embedding describes only what is inside it, so tiles find parts and surfaces that different images have in common (a stem end, a dimpled peel). Tiles that are mostly plain background are left out. tiles = 0 uses the model's own patch tokens instead, which is faster (one pass per image) and is what fordera does, but in the last layer of a CLIP model each token carries much of the whole image, so the features found are less local.