photos

Follow-on to DFIR Summit Talk: Lucky (iOS) 13: Time To Press Your Bets (via @bizzybarney)

in ,

Facial Recognition in Photos

One facet of my DFIR Summit talk I want to expand upon is a look into the Photos application, and a few of the derivative pieces of that endeavor.  While trying to focus on the topic of facial recognition, it seemed prudent to include a brief progression from snapping a photo thru to a persons name being placed beside their face in the Photos application.  

When you use the Native camera and snap a photo, depending on user options, at least a few standard things occur.  It ultimately writes the newly taken photo to /private/var/mobile/Media/DCIM/1**APPLE/IMG_0001.HEIC / .JPG.  As the photo is taken, the Photos.sqlite database is updated to reflect a lot of the metadata about the photo,  which will be covered a bit later.  Additionally, the “PreviewWellImage.tiff” is created.  The “PreviewWellImage.tiff” represents the photo you see when you open your Photos application and see a preview of the most recent image, which is the photo just taken by the camera in this instance.

The beginning of the user’s photos reside in the ../100APPLE/ directory, but this directory iterates upwards (101APPLE, 102APPLE, etc) as more and more photos and videos are saved.  If iCloud syncing is turned on by the user, then several others behaviors occur - but that is for another time.

Let’s focus on the analysis and intelligence built into the Photos application.  I’m able to type text strings and my photos are immediately searched for matching objects within them.  There is a section of my Photos dedicated to “People” where a name has been associated with a face that Apple has analyzed.  

Some of the analysis pieces occurring with the Photos happens in the mediaanalysis.db file.  This file is analyzing and scoring the media files and producing results that seems to feed into other pieces of the analysis.  Some scoring results to highlight are ones that focus on Humans, Faces, and Pets.  

Path: /private/var/mobile/Media/MediaAnalysis/mediaanalysis.db

The ‘Results’ table of the mediaanalysis.db file contains the ‘assetId’ which represents a media file, a ‘resultsType’ which is the specific analytical type and score value found in that media file, and a BLOB (binary large object) which is a binary .plist (bplist).  You can see in the image below, the ‘assetId’ 3 has numerous ‘resultsType’ associated with it and the BLOB for ‘resultsType’ of 1 is selected and on the right you can see the bplist.  

That bplist can be printed to a text file by saving the bplist as a file and then using ‘plutil’ to print it to a text file. As you can see below beside the red star, the printed text is a clean presentation of that .bplist and it tells us that ‘resultsType’ of 1 is associated with Faces based on the scoring.  

I repeated that process for the remaining pieces of data and wrote a brief description of the results type for each, although my device still had a few types that did not have any results.

After running a SQL Query against this database, you can sort the results to potentially see just files that have the results type for ‘humanBounds’ and ‘humanConfidence’.  The ‘localIdentifier’ column from the “assets’ table is a UUID which matches up to the ZGENERICASSETS table of the Photos.sqlite.  

Here is the query for the mediaanalysis.db file.  It’s big and ugly but please test it out if you’re interested, but this piece just seems to build into what we will see later in the Photos.sqlite where this all comes together.

select
	a.id,
	a.localIdentifier as "Local Identifier",
	a.analysisTypes as "Analysis Types",
	datetime(a.dateModified+978307200, 'unixepoch') as "Date Modified (UTC)",
	datetime(a.dateAnalyzed+978307200, 'unixepoch') as "Date Analyzed (UTC)",
CASE
	when results.resultsType = 1 then "Face Bounds / Position / Quality"
	when results.resultsType = 2 then "Shot Type"
	when results.resultsType = 3 then "Duration / Quality / Start"
	when results.resultsType = 4 then "Duration / Quality / Start"
	when results.resultsType = 5 then "Duration / Quality / Start"
	when results.resultsType = 6 then "Duration / Flags / Start"
	when results.resultsType = 7 then "Duration / Flags / Start"
	when results.resultsType = 15 then "Duration / Quality / Start"
	when results.resultsType = 19 then "Duration / Quality / Start"
	when results.resultsType = 22 then "Duration / Quality / Start"
	when results.resultsType = 23 then "Duration / Quality / Start"
	when results.resultsType = 24 then "Duration / Quality / Start"
	when results.resultsType = 25 then "Duration / Quality / Start"
	when results.resultsType = 27 then "Duration / Quality / Start"
	when results.resultsType = 36 then "Duration / Quality / Start"
	when results.resultsType = 37 then "Duration / Quality / Start"
	when results.resultsType = 38 then "Duration / Quality / Start"
	when results.resultsType = 39 then "Duration / Quality / Start"
	when results.resultsType = 48 then "Duration / Quality / Start"
	when results.resultsType = 8 then "UNK"
	when results.resultsType = 11 then "UNK"
	when results.resultsType = 13 then "UNK"
 	when results.resultsType = 21 then "UNK” 
	when results.resultsType = 26 then "UNK"
	when results.resultsType = 31 then "UNK"
	when results.resultsType = 42 then "UNK"
	when results.resultsType = 45 then "UNK"
	when results.resultsType = 49 then "UNK"
	when results.resultsType = 9 then "Attributes - junk"
	when results.resultsType = 10 then 'Attributes - sharpness'
	when results.resultsType = 12 then "Attributes - featureVector"
	when results.resultsType = 14 then "Attributes - Data"
	when results.resultsType = 16 then "Attributes - orientation"
	when results.resultsType = 17 then 'Quality'
	when results.resultsType = 18 then "Attributes - objectBounds"
	when results.resultsType = 20 then "Saliency Bounds and Confidence"
	when results.resultsType = 28 then "Attributes - faceId / facePrint"
	when results.resultsType = 29 then "Attributes - petsBounds and Confidence"
	when results.resultsType = 30 then "Various Scoring Values"
	when results.resultsType = 32 then "Attributes - bestPlaybackCrop"
	when results.resultsType = 33 then "Attributes - keyFrameScore / keyFrameTime"
	when results.resultsType = 34 then "Attributes - underExpose"
	when results.resultsType = 35 then "Attributes - longExposureSuggestionState / loopSuggestionState"
	when results.resultsType = 40 then "Attributes - petBounds and Confidence"
	when results.resultsType = 41 then "Attributes - humanBounds and Confidence"
	when results.resultsType = 43 then "Attributes - absoluteScore/ humanScore/ relativeScore"
	when results.resultsType = 44 then "Attributes - energyValues/ peakValues"
	when results.resultsType = 46 then "Attributes - sceneprint/ EspressoModelImagePrint"
	when results.resultsType = 47 then "Attributes - flashFired, sharpness, stillTime, texture"
end as "Results Type",
	hex(results.results) as "Results BLOB"
from assets a
left join results on results.assetId=a.id

Before diving into the Photos.sqlite file, I want to first point out the file recording the text strings as an apparent result of Apple’s object analysis of the photos.  This file stores the text results which empower our ability to search text strings in the Photos application and return results.  The text strings are not necessarily a result of any specific user activity, but instead an output from analysis automatically being deployed by Apple against the user’s media files.  

Path: /private/var/mobile/Media/PhotoData/Caches/search/psi.sqlite

The ‘word_embedding’ table within psi.sqlite contains columns ‘word’ and ‘extended_word’ which are just strings stored in BLOB’s.  Using DB Browser for SQLite you can export the table to a CSV and it prints the strings from the BLOB’s pretty cleanly.  Separately there is also a table named ‘collections’ that has ‘title’ and ‘subtitle’ columns that appear to be a history of the Memories and Categories that have been used or ones that will be used.

The last table in psi.sqlite to mention for this piece is the ‘groups’ table.  Within the groups table the ‘content_string’ contains some really interesting data.  I initially set out to find just the words “Green Bay” as it was something populated in my text search for the letter “g”.  What I found was far more interesting.  I did find “Green Bay” but additionally found in one of the other BLOB’s, “Green Bay Packers vs. Miami Dolphins”.  That BLOB has a little extra flavor added by Apple for me.  Whether they simply used my geo coordinates baked into my photos from being at Lambeau Field, or analyzed the content of the photos and found the various Miami Dolphins jerseys - I’m not sure.  But a very interesting artifact that is absolutely accurate, dropped in there for me.  Thanks Apple!

Now let’s tackle Photos.sqlite, but only as it pertains to facial recognition and associating photos of people to an actual name.  Because quite honestly this singular file is nearly a full time job if someone wanted to parse every inch of it, and maintain that support.

Path: /private/var/mobile/Media/PhotoData/Photos.sqlite

My instance of Photos.sqlite is a beast, weighing in at over 300MB and containing 67 tables packed full of data about my Photos.  We are going to focus on two tables - ZDETECTEDFACE and ZPERSON.  

ZDETECTEDFACE 

This table contains values that indicate features about faces to include an estimate of age, hair color, baldness, gender, eye glasses, and facial hair.  Additionally there are indicators for if the left or right eyes were closed, and X and Y axis measurements for the left eye, right eye, mouth and center.  So the data in this table is extremely granular, and was quite fun to work through.  Who doesn’t like looking at old photos?

ZPERSON

This table contains a count for the number of times Apple has been able to identify a certain face from the media files.  So in my device, I am recognized by name for hundreds of photos, but there are also photos of me where it hasn’t associated my name with my face.   For each face identified, a UUID (Unique Identifier) is assigned.  So although the analytics piece may not be able to connect a face with a name, it can group all identified instances of the unknown faces as being the same person. 

If there is an association made between the person’s face and a saved contact, the ZCONTACTMATCHINGDICTIONARY column’s BLOB data can possibly reveal a full name and the phone number.  This again can be achieved by printing the bplist to a .txt file.

select
	zga.z_pk,
	zga.ZDIRECTORY as "Directory",
	zga.ZFILENAME as "File Name",
CASE
	when zga.ZFACEAREAPOINTS > 0 then "Yes"
	else "N/A"
	end as "Face Detected in Photo",
CASE 
	when zdf.ZAGETYPE = 1 then "Baby / Toddler"
	when zdf.ZAGETYPE = 2 then "Baby / Toddler"
	when zdf.ZAGETYPE = 3 then "Child / Young Adult"
	when zdf.ZAGETYPE = 4 then "Young Adult / Adult"
	when zdf.ZAGETYPE = 5 then "Adult"
end as "Age Type Estimate",
case
	when zdf.ZGENDERTYPE = 1 then "Male"
	when zdf.ZGENDERTYPE = 2 then "Female"
	else "UNK"
end as "Gender",
	zp.ZDISPLAYNAME as "Display Name", 
	zp.ZFULLNAME as "Full Name",
	zp.ZFACECOUNT as "Face Count",
CASE	
	when zdf.ZGLASSESTYPE = 3 then "None"
	when zdf.ZGLASSESTYPE = 2 then "Sun"
	when zdf.ZGLASSESTYPE = 1 then "Eye"
	else "UNK"
end as "Glasses Type",
CASE
	when zdf.ZFACIALHAIRTYPE = 1 then "None"
	when zdf.ZFACIALHAIRTYPE = 2 then "Beard / Mustache"
	when zdf.ZFACIALHAIRTYPE = 3 then "Goatee"
	when zdf.ZFACIALHAIRTYPE = 5 then "Stubble"
	else "UNK"
end as "Facial Hair Type",
CASE	
	when zdf.ZBALDTYPE = 2 then "Bald"
	when zdf.ZBALDTYPE = 3 then "Not Bald"
end as "Baldness",
CASE
	when zga.zlatitude = -180
	then 'N/A'
	else zga.ZLATITUDE
end as "Latitude",
CASE
	when zga.ZLONGITUDE = -180 
	then 'N/A' 
	else zga.ZLONGITUDE
end as "Longitude",
	datetime(zga.zaddeddate+978307200, 'unixepoch') as "Date Added (UTC)",
	ZMOMENT.ztitle as "Location Title"
from zgenericasset zga
left join zmoment on zmoment.Z_PK=zga.ZMOMENT
left join ZDETECTEDFACE zdf on zdf.ZASSET=zga.Z_PK
left join ZPERSON zp on zp.Z_PK=zdf.ZPERSON
where zga.ZFACEAREAPOINTS > 0

Below is a sample of the output of this analysis, paired with the photo the metadata came from.  You can see it is able to identify me and my two daughters by name, and accurately assess our genders, my sunglasses and facial hair.  

Please test, verify, and give me a shout on Twitter @bizzybarney with any questions or concerns.  

Socially Distant but Still Interacting! New and Improved Updates to macOS/iOS CoreDuet interactionC.db APOLLO Modules

in , , ,

The interactionC.db database certainly does not get as much as attention as its CoreDuet partner in crime, knowledgeC.db. However, I think it has quite a bit of investigative potential. I’ve written about it before in a prior blog, however I’d like to give it more attention here.

I spent this weekend updating the APOLLO modules to have more contextual support and better backwards compatibility with older iOS versions. This database was also introduced to the macOS side with 10.15. 

I’ve added a new query for this database for the ZKEYWORDS table. This tables appears to capture keywords that are contained in various Calendar (com.apple.mobilecal) events. It seems only select certain events as not all my calendar event has keywords in this table.

The main interactionsC.db query has many new updates including attachments and sender/recipient correlation. In general, this database is keeps track of “recent” contact interactions. As an example, I used this query on my iOS database that I copied off of my iPhone a few days ago and it shows ~28,000 entries going all the way back to January of this year, 6+ months! 

In the screenshot below is a Messages (com.apple.MobileSMS) conversation between Heather Mahalik and I. Items that are blurred are our phone numbers. The GUIDs are our Contact Person IDs, these can be correlated with information in the iOS Address Book database. Why some recipient information is blank, I’m not sure. I scrolled way back in our conversation history and the timestamps are spot on. The content of this conversation could (and should) be correlated with the Messages database (sms.db).

This data is not just for Messages and may include other application bundle IDs. Some that I’ve seen in my data include:

  • com.apple.InCallService – Phone Calls

  • com.apple.MobileSMS - Messages

  • com.apple.Preferences - Settings

  • com.apple.ScreenshotServicesService - Screenshots

  • com.apple.mobilecal - Calendar

  • com.apple.mobilemail - Mail

  • com.apple.mobilesafari - Safari

  • com.apple.mobileslideshow - Photos

Contact Interactions & Attachments

For another example, let’s take a look at some interactions for the Photos app (com.apple.mobileslideshow). Not every interaction will have attachments associated with them. The screenshot below contains some AirDrop activity from this device. Some images were AirDropped from the Photos app itself, while one image was AirDropped from within the Messages application as shown in the Target Bundle ID column. It also shows the contact information to whom it was sent - helpful! 

Some of these attachments have a type associates with them in the UTI column (HEIC/PNG), the other has an associated attachment ID, show in hex (0x DCCB49C2FC74461EAD90DAB0C537DBF7). This is a hex representation of the UUID for the image. It seems not every image will have this attachment ID.

This image UUID can be searched for in the Photos database, as shown below to determine which image exactly was AirDropped. 

This might be another good artifact to look for in addition to some of the unified logs I’ve highlighted previously in AirDropping some Knowledge. This is what Elwood’s Phone looks like after lots of AirDropping of various items for that blog. 

Disassociated but not Forgotten Attachments

It seems some attachments are no longer associated with those in the ZINTERACTIONS Table but are not necessarily removed from the ZATTACHMENTS Table. This has potential forensic use as well. APOLLO will not extract these as there doesn’t appear to be timestamps associated with it however, I HIGHLY recommend at least taking a brief look at this table when doing an investigation.

The example below shows may attachment files of different types were associated with a contact interaction.  

  • PDF

  • Images (PNG, HEIC, JPG)

  • Contact VCard

  • Text

  • Archives

  • Movies

  • Documents

Unfortunately, we lose the contact context here. However, we can still find useful tidbits of information like text, file types, URLs, image UUIDs, etc.

Other examples include text that was copy/pasted, or URLs like a Google search link or a YouTube video that was sent.

A Note about Accounts & Recipient Counts

One more example of Mail (com.apple.mobilemail) shows the ACCOUNT column. This GUID can be used to tie interactions to a specific email account in the Accounts databases (Accounts3.sqlite/Accounts4.sqlite).

The Recipient Count columns can be a bit misleading. The column I’ve labeled ‘Recipient Count’ is the amount of recipients on an interaction. This examples shows 2, however that does not include myself. This is an example of an email thread between Heather, Lee, Phil, and myself. I would have thought there would be at least a 3 in this column however that doesn’t appear to be the case. A good example to not make assumptions!

The incoming/outgoing sender/recipient count columns are even more misleading – I haven’t quite figured those out but I believe this might be the total amount of interactions versus the amount of “recipients” on those interactions.

InteractionsC.db can really be a useful database, especially when it is used along with all the other data that APOLLO extracts – context can be everything!

AirDrop Analysis of the UDP (Unsolicited Dick Pic)

in , ,

I saw this article “NYC plans to make AirDropping dick pics a crime” on Friday and it got me thinking. What exactly are the cops going to find if they do an analysis of a device, either the sender or the receiver? 

I’ve already done my fair share of analysis when it comes to the Continuity technology with Apple devices with Heather Mahalik at the SANS DFIR Summit in 2017. This article gave me a good reason to do a revisit on how AirDrop artifacts between mobile devices.

Testing Scenario

I’m using the scenario of sending of dick pics in the middle of Times Square as an example. I have two test phones, an iPhone 7 and an iPhone 6. The iPhone 7 belongs to “David Lightman” and the iPhone 6 belongs to “Jen Mack”.  Each device is on iOS 11 for this test. Instead of UDPs, Jen is going to be sending David USP’s instead, or Unsolicited Squirrel Pics – I suppose I need to keep this family friendly right?

AirDrop works over ad-hoc Peer-to-Peer networking over Bluetooth and Wi-Fi. When a user attempts to AirDrop someone a file it will determine who is within range and who is currently receiving AirDrop connections. This may be set to Contacts Only or Everyone (or Receiving Off for those who don’t want UDP’s, smart choice). Poor David has the setting to Everyone, he’ll get some surprises!

To send the squirrel photo to David, Jen used the native Photos application’s sharing feature to select a device to AirDrop the USP to. The only device within range that has AirDrop turned on and in a receiving mode is David Lightman’s iPhone.

Jen selects his phone and sends the picture. On David’s device he receives a pop-up, he can choose to Decline or Accept. 

AirDrop ID

Important to this analysis is an AirDrop ID. On devices this can be found in /private/var/mobile/Library/Preferences/com.apple.sharingd.plist. The keyword ‘sharingd’ is very important in this scenario as much of the Continuity artifacts can be found by searching for that process name. This plist file should be available in an iTunes backup and from any commercial forensic utility. 

Jen’s AirDrop ID is 3DAA769F9F23.

David’s AirDrop ID is E7D713098E3B, also take note of that the DiscoverableMode is set to Everyone. (Jen does not have this key, hers was set Contacts only.)

Artifacts from the Receiver’s (David Lightman) Device

Assuming that the receiver is the individual who would likely make the complaint to the cops, we will look at this iPhone first to determine what artifacts will persist and if they are show attribution to a certain device and/or owner.

Very few artifacts on the devices will actually show this connection happening. Let’s take a look at some of the main artifacts that I would explore if I had the case.

Unified Logs  

Logs can show lots of data and in great verbose detail. Unfortunately, these Unified Logs do not get backed up on iOS devices. The only way to extract them from the device is to get a physical file dump of the device (Jailbreak/Cellebrite CAS/GrayKey). (UPDATE: 12/04/18 - It has come to my attention from a commentor that these unified log files can be collected using the sysdiagnose process on iOS devices. I’ve tested this and they are in fact there. Funny enough, per Apple Documentation, it wants you to copy this archive off the device using AirDrop. 🤷🏻‍♀️ Not entirely the most forensically sound method but hey, data is data - get it with whatever means your legal team will allow. Thanks ‘G’!)

In this example on David’s iPhone, you can see the sharingd process start to scan, attempting to find peers (Continuity connections). During this process you will also see lots of wirelessproxd and bluetoothd activity as well since AirDrop uses Bluetooth and Wi-Fi services. You will also see references to AWDL or Apple Wireless Direct Link.

Follow the logs down a bit and you’ll start to run into some potentially identifying information. We are now starting to see records containing Jen’s device name “Jen Mack’s iPhone”. Pay close attention to the first record highlighted and you’ll see what looks like a MAC address, however it is neither the Bluetooth nor the Wi-Fi addresses. This address is generated different every time a connection is made, therefore not an ideal data point for attribution. 

Going back to the device’s name. This may lead you in the right direction, however anyone can name their device anything they want. I can call my iPhone X “Samsung S9” for instance, no identifying information and frankly a device that doesn’t even do AirDrop.

The next couple of highlighted sections (in red), show the start of the AirDrop connection. We can see an incoming request and start seeing records that include the AirDrop ID of Jen’s iPhone, 3DAA769F9F23. This is where I think attribution may be possible. This ID seems consistent across connections and different devices in my experience. It may be possible to tie this to an Apple ID or specific device. I have yet to find this connection however – it’s not part of the Serial, UDID, or various MAC addresses that I can tell. More research is needed here.

Next, in purple, is more metadata about the file transfer to include transfer status, media type, originating device name, source application, and associated transfer GUID.

In between these metadata records, it shows that it is transferring a file to /var/mobile/Downloads/com.apple.AirDrop/BA40D8CF-54E6-4B09-8F2F-717FB638174E/Files. Whether the user chooses Accept or Decline, the photo still gets transferred to the device.

Finally, the user receives an alert about the AirDrop’ed photo. Following this record is more details on how the user is alerted to the connection audibly and physically with alert tones and vibrations.

This particular AirDrop connect was “Declined” by David. This can be seen in the records below where the ‘selectedAction’ now shows Decline and a clean-up process has started. Highlighted in teal is the AirDrop connection closing.

If the user Accepted the AirDrop’ed photo, the logs would look like the following. The file would have been ‘accepted’ in the metadata records. Finally, since it is a photo – the default application Photos wants to import it into its files and databases. The AirDrop connection is also closed at this time.

Photos Database

Since the photo gets transferred into the user’s Photos database we can look there for hints. This is one file that does get backed up by iTunes and commercial forensic utilities. The Photos database is located /private/var/mobile/Media/PhotoData/Photos.sqlite on the physical device. 

The filename on Jen’s device was IMG_0007.JPG before it got renamed to IMG_0110.JPG on David’s phone. The original filename can be found in the ZADDITIONALASSETATTRIBUTES table in the ZORIGINALFILENAME column.

It is worth noting that the imported photo will carry the same EXIF data as the original on Jen’s device, in fact it is exactly the same photo (hashes should match). The file size and some timestamps get carried over into the Photos database. Other metadata items can be used to determine a photo from another device are the Height/Width, assuming it is from a different family of devices the pixels may be different.

In the ZGENERICASSET table, we have the same Height/Width, however some timestamps are updated to match the time of import via AirDrop. The ZDATECREATED timestamp matches the original creation of the photo. The ZCLOUDASSETGUID matches the GUID seen in the last few entries in the logs above. There does not appear to be any attribution data in this database.

Artifacts from the Sender’s (Jen Mack) Device

In the rare event that the Sender’s device was acquired, the related artifacts are below.

Unified Logs

Much of the logs look similar to the receiver. The sample below shows the sharingd process connecting to David’s AirDrop ID, E7D713098E3B and shows his iPhone device name. Again, the MAC address does not appear to be consistent and will change on every connection.

A few more lines down we can see some file conversion for the photo, IMG_0007.JPG (which apparently wasn’t needed). This is followed by an AirDrop transaction with the AirDrop ID of David’s iPhone.

Conclusion

The lack of attribution artifacts at this time (additional research pending) is going to make it very difficult to attribute AirDrop misuse. At best, if the cops are provided each device, they can pair the connections up – however this will require access to file system dumping services like Cellebrite CAS, GrayKey from GrayShift or performing a jailbreak to get the most accurate analysis. If the devices are named appropriately (ie: If Jen Mack’s iPhone was actually named ‘Jen Mack’s iPhone’) this may make an analysis easy, however I can see an instance where this can be abused to imitate someone else.

Curious about the USP, here you go. This was an advertisement in the DC Metro system, that for some reason whenever I saw it I giggled. 🤷🏻‍♀️