Part 2- demystifying the file upload process in asp.net

This is a continuation of the blog post http://blogs.msdn.com/b/rohithrajan/archive/2013/01/22/large-file-uploads-in-asp-net-part-1.aspx

The default option for uploading files in asp.net is using Fileupload controls SaveAs method

http://msdn.microsoft.com/en-us/library/system.web.ui.webcontrols.fileupload.saveas.aspx

when we use the FileUpload control’s save as method,it is internally using a class HttpRawUploadedContent .This is in turn using TempFile helper class .This class creates a temp file for posted data. This temp file stores the uploaded content to a folder called  “uploads” inside C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files 

coming back to HttpRawUploadedContent class,this class contains the uploaded content in byte array field _data.This contains uploaded data (either all of it or part read from temp file)

Let’s have a look at the memory after the upload process has been finished(I have used windbg to look at this.For more details refer this ,this and this)

• memory usage in IIS 6 +asp.net 2.0

Scenario Tested:

uploading a file of size 450 MB, maxRequestLength is set to 2097151(maximum allowed) , using fileupload control’s saveas method uploadcontrol.SaveAs

http://msdn.microsoft.com/en-us/library/system.web.ui.webcontrols.fileupload.saveas(v=VS.80).aspx

looking at the memory after the uploading a 431 Mb file using windbg

Number of GC Heaps: 1
generation 0 starts at 0x02671fd0
generation 1 starts at 0x02671e00
generation 2 starts at 0x025b1000
ephemeral segment allocation context: (0x02674b18, 0x02675fdc)
segment    begin allocated     size            reserved
000ef780 7a721784  7a74248c 0x00020d08(134,408) 00001000
000e90a0 790d6358  790f5800 0x0001f4a8(128,168) 00001000
025b0000 025b1000  02675fdc 0x000c4fdc(806,876) 00bc2000
Large object heap starts at 0x035b1000
segment    begin allocated     size            reserved
035b0000 035b1000  035ed0c0 0x0003c0c0(245,952) 00fbe000
04730000 04731000  10f31010 0x0c800010(209,715,216) 007fe000
11b70000 11b71000  1e371010 0x0c800010(209,715,216) 007fe000
Total Size  0x1914126c(420,745,836)
——————————
GC Heap Size  0x1914126c(420,745,836)

objects in the large heap

!dumpheap -min 85000

  Address         MT     Size  Gen
0x035bd980 0x000dea98  172,976    3      Free
0x04731000 0x79124418 209,715,212    3 System.Byte[]
0x11b71000 0x79124418 209,715,212    3 System.Byte[]

main class used for uploading is System.Web.HttpRawUploadedContent .This class has a field named _data of type System.byte[] which holds the uploaded content in the memory.

!do 0x0265088c
Name: System.Web.HttpRawUploadedContent
MethodTable: 68a8ed20
EEClass: 68a8ecb0
Size: 40(0x28) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
     MT    Field   Offset                 Type VT     Attr    Value Name
790fed1c  4000fa8        c         System.Int32  1 instance 209715200 _fileThreshold
790fed1c  4000fa9       10         System.Int32  1 instance 452644402 _expectedLength
79104f64  4000faa       20       System.Boolean  1 instance        1 _completed
790fed1c  4000fab       14         System.Int32  1 instance 452644402 _length
79124418  4000fac        4        System.Byte[]  0 instance 04731000 _data
68a870e0  4000fad        8 …dContent+TempFile  0 instance 026501c4 _file
790fed1c  4000fae       18         System.Int32  1 instance 452644196 _chunkOffset
790fed1c  4000faf       1c         System.Int32  1 instance      206 _chunkLength
0:000> !do 04731000
Name: System.Byte[]
MethodTable: 79124418
EEClass: 791244d0
Size: 209715212(0xc80000c) bytes
GC Generation: 3
Array: Rank 1, Number of elements 209715200, Type Byte
Element Type: System.Byte
Fields:
None

There are two System.byte[] in the memory.And this is yet to be garbage collected.It will get garbage collected but if you are having lot of large file uploads ,it will definitely create problem in memory.lot of objects in large object heap creates heap fragmentation. This issue is fixed to some extent on .net framework 4.5.You can refer following blog posts for large object heap details

http://msdn.microsoft.com/en-us/magazine/cc534993.aspx

http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/

• IIS7+asp.net 3.5 on dual core machine 2 GB

Number of GC Heaps: 2
——————————
Heap 0 (01050d00)
generation 0 starts at 0x01ac0050
generation 1 starts at 0x01ac0044
generation 2 starts at 0x01ac0038
ephemeral segment allocation context: none
segment    begin allocated     size
01ac0000 01ac0038  01bf7024 0x00136fec(1273836)
Large object heap starts at 0x09ac0038
segment    begin allocated     size
09ac0000 09ac0038  09ad0b78 0x00010b40(68416)
Heap Size  0x147b2c(1342252)
——————————
Heap 1 (010519a0)
generation 0 starts at 0x05ac0050
generation 1 starts at 0x05ac0044
generation 2 starts at 0x05ac0038
ephemeral segment allocation context: none
segment    begin allocated     size
05ac0000 05ac0038  05be6ff4 0x00126fbc(1208252)
Large object heap starts at 0x0bac0038
segment    begin allocated     size
0bac0000 0bac0038  0bac0048 0x00000010(16)
Heap Size  0x126fcc(1208268)
——————————
GC Heap Size  0x26eaf8(2550520)

checking the large objects

!dumpheap -min 85000
——————————
Heap 0
Address       MT     Size
total 0 objects
——————————
Heap 1
Address       MT     Size
total 0 objects
——————————
total 0 objects
Statistics:
     MT    Count    TotalSize Class Name
Total 0 objects

!dumpheap -type System.Web.HttpRawUploadedContent
——————————
Heap 0
Address       MT     Size
01bdbe60 53c76b58       40     
01bf1ea0 53c6119c       20     
total 2 objects
——————————
Heap 1
Address       MT     Size
total 0 objects
——————————
total 2 objects
Statistics:
     MT    Count    TotalSize Class Name
53c6119c        1           20 System.Web.HttpRawUploadedContent+TempFile
53c76b58        1           40 System.Web.HttpRawUploadedContent
Total 2 objects

!do 01bdbe60
Name: System.Web.HttpRawUploadedContent
MethodTable: 53c76b58
EEClass: 53a7426c
Size: 40(0x28) bytes
(C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
     MT    Field   Offset                 Type VT     Attr    Value Name
54fa2dbc  4001049        c         System.Int32  1 instance    81920 _fileThreshold
54fa2dbc  400104a       10         System.Int32  1 instance 843205071 _expectedLength
54f7463c  400104b       20       System.Boolean  1 instance        1 _completed
54fa2dbc  400104c       14         System.Int32  1 instance 843205071 _length
54fa35e0  400104d        4        System.Byte[]  0 instance 01bdbe88 _data
53c6119c  400104e        8 …dContent+TempFile  0 instance 01bf1ea0 _file
54fa2dbc  400104f       18         System.Int32  1 instance       96 _chunkOffset
54fa2dbc  4001050       1c         System.Int32  1 instance    81920 _chunkLength

!do 01bdbe88
Name: System.Byte[]
MethodTable: 54fa35e0
EEClass: 54d5eb8c
Size: 81932(0x1400c) bytes
Array: Rank 1, Number of elements 81920, Type Byte
Element Type: System.Byte
Fields:
None

So on asp.net 3.5 +IIS7 ,we can see a better memory usage.But still one problem,the file gets uploaded to temp files first.

C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\fileupload\c60361cf\38faed7d\uploads\aqiib-zc.post

!do 01bf1ea0
Name: System.Web.HttpRawUploadedContent+TempFile
MethodTable: 53c6119c
EEClass: 53a681b4
Size: 20(0x14) bytes
(C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
     MT    Field   Offset                 Type VT     Attr    Value Name
54736e60  4001051        4 …empFileCollection  0 instance 01bf2838 _tempFiles
54fa0b70  4001052        8        System.String  0 instance 01bf3414 _filename
54f9e86c  4001053        c     System.IO.Stream  0 instance 01bf3524 _filestream

!do 01bf3414
Name: System.String
MethodTable: 54fa0b70
EEClass: 54d5d66c
Size: 258(0x102) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\fileupload\c60361cf\38faed7d\uploads\aqiib-zc.post
Fields:
     MT    Field   Offset                 Type VT     Attr    Value Name
54fa2dbc  4000096        4         System.Int32  1 instance      121 m_arrayLength
54fa2dbc  4000097        8         System.Int32  1 instance      120 m_stringLength
54fa1850  4000098        c          System.Char  1 instance       43 m_firstChar
54fa0b70  4000099       10        System.String  0   shared   static Empty
    >> Domain:Value  0103b168:05ac01d0 01074788:05ac01d0 <<
54fa17a0  400009a       14        System.Char[]  0   shared   static WhitespaceChars
    >> Domain:Value  0103b168:05ac095c 01074788:05acae2c <<

So SaveAs is merely copying the file content from temp file to the location you provide.So you can never implement a proper upload progressbar because,by the time you call the saveas method ,it would have put all the content to the temp file.Similarly you cannot use Request.Files in any global module ,because this is internally using the HttpRawUploadedContent class.

The file gets uploaded to temp directory when the System.Web.HttpRequest.GetEntireRawContent() is called.And this is called from FillInFormCollection() which is again called from the getter Request.Form.

So if any of the code(even the built in asp.net framework code) refers a Request.Form element,the file gets uploaded to temp directory.