DPP really needs a major rewrite

[wq9nsc]

I just batch processed close to 400 files from a photo shoot at a musical dress rehearsal.  Later Canon sensors (1DX III and the later mirrorless sensors) are far more processing intensive but Canon has yet to provide a version of DPP that properly utilizes computer resources and GPUs (graphics processor unit cores) in particular.

I use a HP Z series work station with twin 8 core CPUs and a pair of Nvidia workstation graphics cards with significant GPU power available.  Batch processing these files took over 25 minutes which would be OK except DPP CPU utilization was pretty low and it didn't utilize the Nvidia GPU resources at all. 

Other image processing software, both still and video, has done much better for many years and it is past time for Canon to catch up.  Screen grab from Task Manager attached to show how little DPP makes use of available resources.

Rodger

[rs-eos]

Yea, performance is abysmal. On my Mac Studio (M2 Ultra, 24 CPU cores, 76 GPU cores, 192 GB RAM), it would take around 137 minutes total to process 400 files (I stopped it early).

I used 400 .CR3 files taken with an EOS R5 II.  To rule out any impact from file I/O, the original RAWs were on one SSD and destination was set to a second SSD.  Read/write speed around 2,000 MBps.

Zero GPU was used:

 

For CPU, it sometimes would shortly peak to 91%, but average was much lower.  Lot's of pockets where things weren't utilitzed:

Canon isn't known for writing software, but hopefully they can step up their game.  I'd like to use DPP (heard good things about it over the years), but I'll stick with Adobe for now.

[JamesHarvey]

Notably the Canon Neural Network processors & Upscaler are graphics processor intensive, hopefully that is a sign where DPP may go.  

Having said that, I note that the CPU utilization in the two responses above seems quite low compared to what I see on my workstation when processing in DPP (I7-14700 20 core / 28 thread + RTX4070ti + 96GB DDR5), so I wonder if the real issue is that DPP is written to be optimum with a "conventional" PC (single Intel processor)?  My utilization usually stays above 80% continuous. 

[JamesHarvey]

Were the SSDs connected using SATA or USB3x, I have noted considerable reduction in read speeds over USB.

[wq9nsc]

James,

The SSDs used for processing the input and output file are on what HP calls "Z drives" and these are a pair of SSDs directly on the processor bus giving the fastest throughput possible.  I also have two other 1 gig SS drives and two 8 terabyte mechanical drives internally mounted and they are on the internal SAS bus but are not involved with the DPP software. 

The CR3 files from the 1DX III have much more processing overhead than the older CR2 files from my 1DX II bodies.  400 hundred equal size files from the 1DX II take less than 1/3 of the time to process. 

Rodger

[johnrmoyer]

On my 2019 iMac with 96GB RAM and fusion hard drive and i5 9600K CPU (6 cores), sometimes DPP uses 400% CPU if I start a save of 16bit TIFF and 2 separately scaled JPG at about the same time. The DPP batch process uses more than 300% CPU if in the foreground, but less than 50% CPU in the background. It was very slow when I had only 32 GB RAM.

If one knows in advance how one wants to process 400 files, why not just load a picture style into the camera and set the camera to save both CR3 and JPG files.

If one wanted to apply the same crop to 400 files, graphicsmagick can do that in a loop from a shell prompt.

I have not used any Adobe software in decades. I sometimes use rawtherapee and I have used darktable in the past. I also use gimp, GMIC, hugin, and graphicsmagick.

The 2 biggest advantages to DPP for me are the "digital lens optimizer" and DPRAW tool. It is also convenient to be able to use the same settings both in camera and in DPP.

The improvements I would like to see in DPP include:

• ability to edit IPTC metadata
• ability to edit Metadata Working Group standard metadata
• an info popup that will name the algorithm being used
• using DPRAW data to upscale an image
• better noise reduction
• an info screen describing the gamma curve associated with a "picture style"
• ability to save a 16 bit TIFF or 16 bit png with the HDR PQ color curve and dynamic range
• offer a greater variety of algorithms for combining the the 2 DPRAW images

[wq9nsc]

John,

I shoot in all sorts of varying lighting conditions with a variety of mixed color temperatures while shooting sports so most of the files require a lot of post processing.  And each image is going to have an individual crop and because of "uniform malfunction" sometimes it is necessary to clone in a little material or darken part of the image.  So although the jpg in camera approach would work for a lot of people, it won't work for me.

Overall I like DPP but there is no good reason not to upgrade it to take advantage of hardware since pretty much every PC has at least some GPU resources and fast dedicated boxes are built solely around GPU arrays.  Software doesn't seem to be Canon's strong point and it would be nice if they would bring in an outside group to bring DPP up to date.  It is pathetic in performance compared to the video editing software I use.

Rodger

[rs-eos]

Thunderbolt 3/4.  The Mac Studio has 6 independent controllers, so each port offers full bandwidth.

[fruitfly]

I completely agree - I have been on a quest to speed up DPP editing process. Batch processing is OK since it can run unattended. Here are some observations that may be useful for someone. 

After I upgraded from EOS R to R6mkII I noticed a drop in processing speed in DPP. As an experiment, I ran a few tests by downloading RAW files of the same still life scene from Imaging Resource (trying to keep things as uniform as possible), then duplicated each file to get 5 identical copies, and ran batch JPG conversion in DPP. Here are the times (on a PC with i9-11900 processor):

Camera                Time for 5 files, sec         Time for 1 file, sec           Mpixels

RP                           35                                           7                                  26.2

R                             44                                           8.8                               30.3

R6                           70                                           14                                 20

R5                           148                                         29.6                              45

R6 mark II               87                                           17.4                              24

The main observation is that "something happened" when Canon moved from R/RP to R6, R5 and later cameras - processing times more than doubled for similar sensor size! (I tend to blame the new Clarity slider that wasn't there before - maybe there is more to it than that).

But my main quest has been to speed up the dreaded spinning wheel at the bottom right. (By the way, related to the results above, the wheel times are much faster for the R files compared to R6mkII, and still faster for older models) One surprising discovery is the following: I recently started using the Neural Network Processing tool (it runs on the GPU!), and found that screen updates/wheel times when working with .CRN files are ~3 times faster than when the corresponding .CR3 files are processed! This is despite the fact that the .CRN files are ~5 times larger. Now my workflow is the following:

I first cull the RAW files with FastStone (I find that this is faster than using DPP for the same task). Then I batch-process all .CR3 files into .CRN files using the Neural Network Processing tool in DPP, and only then make edits/adjustments. Neural Network processing is slow (for R6mkII it's about 45 seconds per file on my computer using 4060 Ti GPU), but I just start the batch and walk away (can do overnight), and then my editing after that is much faster. I also like the output of this tool (had to reduce sharpness, otherwise it overdoes it).