Evaluation of AI-based Image Search: Excire vs. Adobe, Apple & Google

Motivation

Adobe has recently released the new Lightroom CC and has enhanced it with Adobe Sensei technology. Sensei offers several intelligent cloud services including semantic image analysis and keyword-based image search (so-called AI search). Adobe´s AI search, however, is cloud based and therefore available in Lightroom CC only.

Fortunately, users of Adobe Lightroom Classic CC can use the Excire Search plugins for AI search. Excire Search runs locally with no cloud usage: no uploads, no downloads and the AI machine is running on the local computer. So, if you´re a classic user, Excire Search Plugin is the perfect add-on for optimizing the workflow with AI search, the benefits of which are now becoming more obvious with the new workflows that are possible in Lightroom CC.

In addition to the features available in Lightroom CC, Excire Search offers additional features such as a very useful similarity search and more specific search functions that can find, for example, a group foto with smiling ladies at the beach.

But how well does Excire Search perform compared to Adobe and other competitors? To answer this question we performed a comprehensive evaluation and compared Excire Search with Adobe Lightroom CC, Apple Photos and Google’s Vision Api.

Test Dataset

The dataset used for testing consists of 1500 images in JPG format belonging to fifteen different categories with 100 images for each category. The test images have been chosen randomly from a large database containing mainly images downloaded from flickr. The categories have been chosen randomly from the 500 categories that Excire Search can handle.

None of the test images was used for training (something we can know for sure only for Excire). It would have been nice to test with more classes and images but obtaining the Adobe and Apple labels required time-consuming manual work. Overall, we have been careful to design a representative and unbiased test. The following example images depict the 15 categories (semantic classes) of our test dataset (in alphabetical order from upper left to lower right):

Main Features

The following table summarizes the main features of the evaluated search engines. The given runtime duration denotes the time it takes to upload (Adobe, Google) or import (Apple, Excire) and analyze the 1500 images.

CompanySoftwareCloud vs. LocalNr. of CategoriesRuntime*
AdobeAdobe Lightroom CCcloudunknown13:01 min
ApplePhotoslocal4432several hours**
PRCExcire Search Lr v1.3local5006:13 min**
GoogleGoogle Vision Apicloudunknown22:24 min

*50Mbit/s internet connection with an upload speed of 20Mbit/s. WLAN: 5GHz 110-225 Mbit/s (only used for Lightroom CC)

**on a MacBook Pro 2,6GHz, 8GB Ram, SSD and macOS Sierra 10.12.6.

Cloud vs. Local

Cloud computing is becoming increasingly popular (at least with providers) and a growing number of cloud services are becoming available. An obvious benefit is that powerful servers and computing architectures can be used and can be scaled to match the needs for storage and computational power.

For AI services, an important benefit is that one can use large deep networks that would be too complex to run on a local computer.

Therefore, designing a system that runs locally on simple computers with different architectures is much more challenging and these challenges are likely to limit performance.

Then again, an obvious drawback of a cloud-based workflow for photographers is that one needs to upload and download images. While this might be acceptable for those who use the most popular cameras today (cell phones), photographers who shoot large image files for maximum quality might be more reluctant.

For others, privacy might be an issue and, after all, nobody really likes to lose control.

Excire Search has been designed such that all computations are done locally on the user’s computer. One would thus expect that it cannot match the performance levels of more powerful cloud-based solutions. We were surprised to find that this is not the case: Excire Search performs better and is faster than its competitors.

Test Procedure

For each of the 4 search engines, we performed the same tests to evaluate the results of AI-search. We searched with the 15 possible keywords corresponding to the 15 categories that we evaluated, for example ‘beach’, ‘butterfly’, ‘cat’, etc.

Only single keywords were used, no combinations of keywords. Performance was then quantified by determining the quantities TP, FP, FN and TN.:

An image is considered to be relevant for a particular keyword if it depicts the corresponding content, for example if we search with the keyword ‘cat’, all images depicting a cat are relevant.

Given the dataset described above, for each search we have 100 relevant images (P) and 1400 non-relevant images (N).

  • TP = True Positives: the number of relevant images that were found (nr. of cats that were found when searching for cats)

  • FP = False Positives: the number of relevant images that were found (nr. of cats that were found when searching for cats)

  • FN = False Negatives: the number of relevant-images that were missed (nr. of missed cats when searching for cat)

  • TN = True Negatives: True Negatives: the number of non-relevant images, that were not found (nr. of dogs etc. that were correctly not found when searching for cats)

Results

Finally, we are using the following rates for evaluation:

  • Sensitivity (True Positive Rate or Hit Rate): TPR = TP / (TP + FN)
  • Specificity (True Negative Rate) TNR = TN / (FP + TN)
  • Accuracy: ACC = (TP + TN) / (P + N)


The following figure depicts the average results for the 4 engines plotted as sensitivity vs. specificity. The bars indicate the variance of the result obtained for the different keywords. The ideal result would be a dot with very short bars and placed in the upper right corner of the plot.

Discussion and Verdict

The results clearly show that Apple Photos performs the most restrictive search, meaning that it is tuned for high specificity and low sensitivity. This strategy makes sure we get only few dogs if we search for cats but it also leads, in this case, to the drawback that we miss quite a few cats.

Adobe has obviously chosen the opposite strategy of trying to not miss any cats and give us quite a few false dogs.

Excire and Google are striking a good compromise between sensitivity and specificity and Excire is the best performer in this test with a specificity that is somewhat better than Google´s and a clearly better sensitivity. Regarding runtime, Excire Search is the fastest engine from the user’s perspective.

Class Specific Results

For those interested in more details, the following plots show the class-specific results:

Beach
Butterfly
Car
Castle
Cat
Dog
Flower
Horse
House
Lion
Mountain
Skyscraper
Snow
Soccer
Windsurfing

Hinweis für Excire Foto Bestandkunden: Excire Foto Nutzer müssen beim Testen beachten, dass die bisherige Excire Datenbank beim ersten Programmstart modifiziert wird und somit ein Rückwechsel auf die Version 1.3 ohne Backup nicht möglich ist. Wir empfehlen daher vor der Nutzung von Excire Foto 2022, eine Sicherung zu erstellen. Wie dies geht, steht in folgenden Forum-Beiträgen: LINK 1 and LINK 2

Excire Foto Excire Foto Testversion
Mehrere Datenbanken
JPG, PNG & BMP
RAW Formate
EXIF Daten einlesen
IPTC Metadaten einlesen
IPTC Manager & Editor
Sterne, Bewertungen, Label einlesen
Stichwörter einlesen
Metadaten speichern
Fotos exportieren
Teilen via Dropbox
Teilen via Google Drive
Find similar photos
Mit Stichwort finden
Find faces
Find people
Ähnliche Fotos mit externem Foto finden
Personen mit externem Foto finden
GPS Position eines Fotos in Google Maps anzeigen
Manuelle Stichwörter vergeben
Sammlungen anlegen
Sterne, Flaggen & Farblabel vergeben und danach suchen
Supported languages Deutsch & Englisch
Current version 2.1.1
Processor Mehrkernprozessor mit 64-bit und AVX UnterstützungÄltere AMD Prozessoren wie AMD Phenom(tm) II X6 1100T und AMD Phenom(tm) II X (auch unter der Bezeichnung AMD Athlon II X4 640 bekannt) werden nicht unterstützt.

Intel Core 2 Duo processors are not supported.
Operating system macOS 10.14 (or newer) or Windows 10 (64-bit) or Windows 11 (64bit)
Memory Min. 8GB RAM. However, 16GB or more are recommended
Hard disk Die Excire Datenbanken benötigen für 100.000 Fotos ca. 250MB. Der Vorschau-Speicher umfasst dann ca. 25GB bei höchster Qualität, Raw-Formaten und wenn für jedes Foto eine Vorschau erzeugt wird.
Supported languages Deutsch & Englisch
Processor Mehrkernprozessor mit 64-bit und AVX UnterstützungÄltere AMD Prozessoren wie AMD Phenom(tm) II X6 1100T und AMD Phenom(tm) II X (auch unter der Bezeichnung AMD Athlon II X4 640 bekannt) werden nicht unterstützt.

Intel Core 2 Duo processors are not supported.
Operating system macOS 10.14 (or newer) or Windows 10 (64-bit) or Windows 11 (64bit)
Memory Min. 8GB RAM. However, 16GB or more are recommended
Hard disk Die Excire Datenbanken benötigen für 100.000 Fotos ca. 250MB. Der Vorschau-Speicher umfasst dann ca. 25GB bei höchster Qualität, Raw-Formaten und wenn für jedes Foto eine Vorschau erzeugt wird.

Für alle Adobe Lightroom Classic Nutzer ist Excire Search die perfekte Lösung zum Durchsuchen ihres Lightroom Katalogs. Das clevere Lightroom-Plugin analysiert und verschlagwortet Fotos automatisch und erweitert Lightroom um leistungsfähige KI Suchfunktionen. Mit wenigen Klicks und in Sekundenschnelle findet Excire Search genau die Fotos, die Sie suchen. So macht Bildverwaltung wieder Spaß und stundenlange Bildsuchen gehören der Vergangenheit an. 

Für Windows und macOS und in der 2022er Version mit integriertem Duplikatefinder.

Supported languages Deutsch, Englisch, Französisch, Italienisch & Spanisch
Current version 2.1.1
Processor Multi-core processor with 64-bit and AVX support
Ältere AMD Prozessoren wie AMD Phenom(tm) II X6 1100T und AMD Phenom(tm) II X (auch unter der Bezeichnung AMD Athlon II X4 640 bekannt) werden nicht unterstützt.

Intel Core 2 Duo processors are not supported.
Operating system macOS 10.14 (or newer) or Windows 10 (64-bit) or Windows 11 (64-bit)
Lightroom (Classic) Version 6 (or newer)
Memory 8GB RAM. Bei großen Katalogen (> 150.000 Fotos) empfehlen wir jedoch 16GB oder mehr
Hard disk 375 MB of free hard disk space for the base installation and additional memory for image signatures. For example, an Adobe Lightroom catalog of 100,000 images requires about another 800 MB of additional memory.

Excire Analytics is an innovative extension for Excire Foto that provides valuable insights into your photographic work.

Use the numerous functions and settings of Excire Analytics to make better purchasing decisions for future photo equipment, learn from the evaluations and improve your photo skills.

Excire Analytics erfordert mindestens Excire Foto Version 2.0 und kann durch Eingabe eines Aktivierungscodes freigeschaltet werden. Geben sie einen solchen Code dazu einfach in ihrer Excire Foto Version in den Einstellungen im Reiter „Lizenz“ ein. Entsprechende Funktionen werden dann freigeschaltet und benötigte Komponenten nachinstalliert.

Supported languages Deutsch & Englisch
Current version 2.1.1
Processor Mehrkernprozessor mit 64-bit und AVX UnterstützungÄltere AMD Prozessoren wie AMD Phenom(tm) II X6 1100T und AMD Phenom(tm) II X (auch unter der Bezeichnung AMD Athlon II X4 640 bekannt) werden nicht unterstützt.

Intel Core 2 Duo processors are not supported.
Operating system macOS 10.14 (or newer) or Windows 10 (64-bit) or Windows 11 (64bit)
Memory Min. 8GB RAM. However, 16GB or more are recommended
Hard disk Die Excire Datenbanken benötigen für 100.000 Fotos ca. 250MB. Der Vorschau-Speicher umfasst dann ca. 25GB bei höchster Qualität, Raw-Formaten und wenn für jedes Foto eine Vorschau erzeugt wird.
Excire Foto 2022 ist eine leistungsstarke und innovative Software für die einfache Fotoverwaltung und das schnelle Durchsuchen von Fotosammlungen. Zahlreiche intelligente Funktionen helfen Ihnen dabei, Ordnung zu halten und die gesuchten Fotos zu finden.

Das sind die Highlights der 2022er Version:

  • Duplikatefinder mit zahlreichen Einstellungsmöglichkeiten
  • support for PSD files
  • match accuracy for similarity searches is now adjustable
  • the maximum number of search results has been increased to 50,000
  • support for Windows network paths

Wir empfehlen Windows Nutzern die Raw Erweiterung zu installieren: Download
Supported languages Deutsch & Englisch
Current version 2.1.1
Processor Mehrkernprozessor mit 64-bit und AVX UnterstützungÄltere AMD Prozessoren wie AMD Phenom(tm) II X6 1100T und AMD Phenom(tm) II X (auch unter der Bezeichnung AMD Athlon II X4 640 bekannt) werden nicht unterstützt.

Intel Core 2 Duo processors are not supported.
Operating system macOS 10.14 (or newer) or Windows 10 (64-bit) or Windows 11 (64bit)
Memory Min. 8GB RAM. However, 16GB or more are recommended
Hard disk Die Excire Datenbanken benötigen für 100.000 Fotos ca. 250MB. Der Vorschau-Speicher umfasst dann ca. 25GB bei höchster Qualität, Raw-Formaten und wenn für jedes Foto eine Vorschau erzeugt wird.