Can SHA-1 algorithm be computed on a stream? With low memory footprint?

Posted by raoulsson on Stack Overflow See other posts from Stack Overflow or by raoulsson
Published on 2010-03-22T21:44:22Z Indexed on 2010/03/22 21:51 UTC
Read the original article Hit count: 268

Filed under:

sha1

|

memory

|

stream

|

algorithm

|

java

I am looking for a way to compute SHA-1 checksums of very large files without having to fully load them into memory at once.

I don't know the details of the SHA-1 implementation and therefore would like to know if it is even possible to do that.

If you know the SAX XML parser, then what I look for would be something similar: Computing the SHA-1 checksum by only always loading a small part into memory at a time.

All the examples I found, at least in Java, always depend on fully loading the file/byte array/string into memory.

If you even know implementations (any language), then please let me know!

© Stack Overflow or respective owner

Related posts about sha1

PBKDF2-HMAC-SHA1

as seen on Stack Overflow - Search for 'Stack Overflow'
To generate a valid pairwise master key for a WPA2 network a router uses the PBKDF2-HMAC-SHA1 algorithm. I understand that the sha1 function is performed 4096 times to derive the PMK, however I have two questions about the process. Excuse the pseudo code. 1) How is the input to the first instance… >>> More
Is it possible to get identical SHA1 hash?

as seen on Stack Overflow - Search for 'Stack Overflow'
Given two different strings S1 and S2 (S1 != S2) is it possible that: SHA1(S1) == SHA1(S2) is True? If yes - with what probability? Is there a upper bound on the length of a string, for which probably of getting duplicates is 0? If not - why not? Thanks >>> More
Is it possible to convert a 40-character SHA1 hash to a 20-character SHA1 hash?

as seen on Stack Overflow - Search for 'Stack Overflow'
My problem is a bit hairy, and I may be asking the wrong questions, so please bear with me... I have a legacy MySQL database which stores the user passwords & salts for a membership system. Both of these values have been hashed using the Ruby framework - roughly like this: hashedsalt = … >>> More
Is there any difference between md5 and sha1 in this situation?

as seen on Stack Overflow - Search for 'Stack Overflow'
It is known that 1. if ( md5(a) == md5(b) ) 2. then ( md5(a.z) == md5(b.z) ) 3. but ( md5(z.a) != md5(z.b) ) where the dots concatenate the strings. What happens in the second row if we change all the md5 to sha1? So: 1. if ( sha1(c) == sha1(d) ) 2. then ( sha1(c.z)… >>> More
iPhone: convert string using HMAC-SHA1

as seen on Stack Overflow - Search for 'Stack Overflow'
hi all, i want to generate HMAC-SHA1 of NSString type variable. I see the post but this method give me error in CCHmac(kCCHmacAlgSHA256, cKey, strlen(cKey), cData, strlen(cData), cHMAC); and NSString *hash = [HMAC base64Encoding]; line. Please suggest how can I generate the HMAC-SHA1 of any string… >>> More

Related posts about memory

SQL Server and Hyper-V Dynamic Memory Part 2

as seen on SQL Blog - Search for 'SQL Blog'
Part 1 of this series was an introduction and overview of Hyper-V Dynamic Memory. This part looks at SQL Server memory management and how the SQL engine responds to changing OS memory conditions. Part 2: SQL Server Memory Management As with any Windows process, sqlserver.exe has a virtual… >>> More
SQL Server and Hyper-V Dynamic Memory - Part 1

as seen on SQL Blog - Search for 'SQL Blog'
SQL and Dynamic Memory Blog Post Series Hyper-V Dynamic Memory is a new feature in Windows Server 2008 R2 SP1 that allows the memory assigned to guest virtual machines to vary according to demand. Using this feature with SQL Server is supported, but how well does it work in… >>> More
SQL Server and Hyper-V Dynamic Memory - Part 1

as seen on SQL Blog - Search for 'SQL Blog'
SQL and Dynamic Memory Blog Post Series Hyper-V Dynamic Memory is a new feature in Windows Server 2008 R2 SP1 that allows the memory assigned to guest virtual machines to vary according to demand. Using this feature with SQL Server is supported, but how well does it work in… >>> More
Python Memory leak - Solved, but still puzzled

as seen on Stack Overflow - Search for 'Stack Overflow'
Dear everyone, I have successfully debugged my own memory leak problems. However, I have noticed some very strange occurence. for fid, fv in freqDic.iteritems(): outf.write(fid+"\t") #ID for i, term in enumerate(domain): #Vector tfidf = self.tf(term… >>> More
SQL Server and Hyper-V Dynamic Memory Part 3

as seen on SQL Blog - Search for 'SQL Blog'
In parts 1 and 2 of this series we looked at the basics of Hyper-V Dynamic Memory and SQL Server memory management. In this part Serdar looks at configuration guidelines for SQL Server memory management. Part 3: Configuration Guidelines for Hyper-V Dynamic Memory and SQL Server Now that… >>> More