WriteSample takes long time to finish

I have a very interesting issue.

Basically Sink Writer WriteSample takes long time to finish. Sometimes 3-4 seconds.

Let me explain:

I am copying one MP4 file to another, like mfcopy. When I do a straight up copy it finishes up fast, just like expected.

But if I just read MF_SOURCE_READER_FIRST_VIDEO_STREAM from source reader and write that sample to the new mp4 files, the writes take very long time. It starts off fast, but as it reads along the file, the write takes very long. Each write takes 3-4 seconds.

Can anybody provide some clues as to why this might be happening?