DPP 4.18.10 always crashes during batch RAW processing

[dpp4user]

I'm using DPP4 since its first version and most of the time it worked as expected.

But since I recently upgraded to Windows 11 and a new core i5 13500 system I cannot use it anymore since batch raw processing always crashes after about 20-30 files. When restarting with the remaining files, then another chunk of about 20-30 files work until it crashes again. So it is not a particular file that makes it crash.

Ths crash is essentially the same as I reported here https://community.usa.canon.com/t5/Camera-Software/DPP-4-16-11-0-frequently-crashes/m-p/388864 (access violation 0xc0000005 in Dpp4Batch.exe i.e. DppCoreSubD.dll) which occured on a different system (Windows 10, core i5 4500) and a different DPP version. Looks like Canon did not yet fix the bug that causes the access violation. Will they ever do so?

So for now DPP4 is unusable for me and I'm at a loss as to how to proceed with all my .CR3 raw files. Of course there are thrird party raw processors, but I'd like to use the vendors (Canon) raw processor for best results.

Is there any workaround or a chance that Canon will fix this, probably very old, bug?

[wq9nsc]

The crash after a certain number of files sounds a lot like you could have a system memory issue.  DPP grows in size as it operates and it may be hitting the same corrupt section of system memory after a few RAW files have been processed.  I would start by running a memory diagnostics program to ensure that there isn't a problem.

I primarily run DPP (4.18.10.0) on a pair of HP twin Xeon CPU workstations but I am also giving it more limited use on a pair of HP Z series laptops running Windows 11 on I7 series CPUs and none have had this issue.  I use DPP multiple times per week and recently have been shooting around 4,000 images a week using three 1DX series bodies and I shoot only RAW files.

Rodger

[dpp4user]

Well, I already run memory diagnostics without any error and Dpp4 is not the ony application I'm running with high CPU and memory load. Other applications didn't crash so far.

Maybe the issue occurs with (compressed) .CR3 files only. I'm not sure whether your experience on HP Xeaon is with CR3?

[wq9nsc]

The CR3 files work fine, the batch process time is a little slower than with CR2 files but otherwise they work perfectly.

My usual sports setup is a pair of 1DX III bodies with EF 400 or 300 f2.8 glass on one and a 70-200 f2.8 on the other and a 1DX II with 24-70 f2.8.  The III bodies produce CR3 files and they account for an overwhelming share of images captured.

In batch mode, CR2 files are around 2 seconds per image RAW to JPG while CR3 run from 4 to 6.  There is also a slight lag before editing is possible with CR3 files when switching between main and crop tools tabs unlike CR2 which is nearly instantaneous.  The lag is definitely annoying and if cropping were possible from the main (exposure, contrast, etc.) tools tab it would make things faster and more convenient.

Rodger

[johnrmoyer]

It seems to me that DPP crashes when there are too many files cached. Putting a smaller number of images in one file system directory seems to me to help. I have several times let macOS send DPP crash reports to Apple, but I do not know whether Apple shares those reports with Canon. The crash report seems to be a stack trace with some symbol names. I would try to capture one of these and send it to Canon if they need it.

Good Luck.

[wq9nsc]

John,

I remember that happening with earlier iterations of DPP 4 and it may still be an issue with IOS but it hasn't happened on my Windows system in several years.  My typical events are 2000+ RAW images using 3 camera bodies with around 300 images selected for processing to jpg.

For processing, I dump them all into the same directory on the "Z drive" in my workstation which is a fast SSD located on one of the Xeon CPU buses.  Other than exiting about midway through the processing session to get rid of the ever present DPP "memory bloat" that slows it down, DPP has been very well behaved. 

With DPP, it doesn't matter how much memory you have available it still becomes lethargic as it grows in size.  The main workstation I use DPP with has 256 GB per CPU and when just photo processing it never goes over the single digits in memory utilization but it still gets sluggish as DPP will climb beyond 10 GB in memory utilization from its under 500 MB starting size if you let it run long enough and process enough photos.

Rodger

[dpp4user]

ah yes, there are many Dpp4Batch.exe.xyz.dmp file in AppData\Local\CrashDumps
lets see whether I can send them to Canon

[dpp4user · Accepted solution]

Ok, here is the workaround I found that seems to avoid the crash.

When I'm starting Dpp4 with affinity mask 0x01 or any other mask that assigns a single core, the batch job finishes without crash. Of course the whole thing is horribly slow. But it seems to work. So I'm going to work with Dpp4 UI without affinity mask and then restart with a affinity mask of a single bit to start the batch. Only one core working, 15 idle. Quite sad in the year 2024.

This is a very(!) strong indication of a race condition in the program code, which in the end is "just" UB, since race conditions naturally render UB. The Canon guys should do their homework and fix it. It is as simple (and as sad) as that.

[dpp4user]

Since re-starting DPP4 with different affinity is quite cumbersome I found a way to do batch processing on a single core (to avoid crash) without restarting DPP4. I'm sharing the steps since maybe someone else has similar problems.

(On Windows) DPP4 starts Dpp4Batch.exe and communicates via .Net Remoting IPC channel, where Dpp4Batch.exe behaves as server. When I start Dpp4Batch.exe manually with affinity for a single core, the main form will connect to the existing process rather than starting a new one. And, voila, batch processing runs on a single core. To avoid shutdown of Dpp4Batch.exe uncheck the "Close dialog when finished" option in the lower left of the batch dialog. I put it togther by a small .bat script:

cd "%ProgramFiles%\Canon\Digital Photo Professional 4"
start /Affinity 0x02 Dpp4Batch.exe /P CanonDPPExBatch /FF "Microsoft Sans Serif" /FS "8,25"
Dpp4.exe

which starts the usual DPP4 UI as well as the batch server. As described above, I can now start batch processing from DPP4 just as usual. It will connect to the running Dpp4Btach that is limited to a single core (and terribly slow of course).

[johnrmoyer]

Instead of single core, will two cores work? Most Windows programs work best with at least 2 real cores or 2 hyper-threads. Intel hyper-threads have two sets of registers for the same core and were invented to fool Windows into thinking there were more cores. When Intel first shipped CPUs with hyper-threads, it sped up Windows much more than Linux or Unix. When I ran DPP in a Windows virtual machine on my Debian desktop, I always gave the virtual machine 2 cores and 4 threads because after experimenting that seemed to me optimal.

Some of what I write might be wrong because it has been several years since I have used Windows. Intel now seems to put the easy to find architecture descriptions in a "gaming" section.

https://www.intel.com/content/www/us/en/gaming/resources/hyper-threading.html