Search Results

Search found 22 results on 1 pages for 'dft'.

Page 1/1 | 1 

• Hitachi DFT hangs - Causes and workarounds?

  - by Ssvarc

  Hitachi DFT hangs (it gets all the way to "Loading DFT" and hangs with a blinking cursor) when I use it on various computers (the latest computer to hang being a Dell Vostro 1000 laptop) and works on others. What determines when it will hang and when it won't? What similar program can I use (boot CD) that can determine HD health reliably? Thanks!

  Read the article

• DSP - Filtering frequencies using DFT

  - by Trap

  I'm trying to implement a DFT-based 8-band equalizer for the sole purpose of learning. To prove that my DFT implementation works I fed an audio signal, analyzed it and then resynthesized it again with no modifications made to the frequency spectrum. So far so good. I'm using the so-called 'standard way of calculating the DFT' which is by correlation. This method calculates the real and imaginary parts both N/2 + 1 samples in length. To attenuate a frequency I'm just doing: float atnFactor = 0.6; Re[k] *= atnFactor; Im[k] *= atnFactor; where 'k' is an index in the range 0 to N/2, but what I get after resynthesis is a slighty distorted signal, especially at low frequencies. The input signal sample rate is 44.1 khz and since I just want a 8-band equalizer I'm feeding the DFT 16 samples at a time so I have 8 frequency bins to play with. Can someone show me what I'm doing wrong? I tried to find info on this subject on the internet but couldn't find any. Thanks in advance.

  Read the article

• DSP - How are frequency amplitudes modified using DFT?

  - by Trap

  I'm trying to implement a DFT-based equalizer (not FFT) for the sole purpose of learning. To check if it works I took an audio signal, analyzed it and then resynthesized it again with no modifications made to the frequency spectrum. So far so good. Now I tried to silence some frequency bands, just by setting their amplitudes to zero before resynthesis, but definitely it's not the way to go. What I get is a rather distorted signal. I'm using the so-called 'standard way of calculating the DFT' which is by correlation. I first tried to modify the real part amplitudes only, then modifying both the real and imaginary part amplitudes. I also tried to convert the DFT output to polar notation, then modifying the magnitude and convert back to rectangular notation, but none of this is working. Can someone show me what I'm doing wrong? I tried to find info on this subject in the internet but couldn't find any. Thanks in advance.

  Read the article

• Simple in-place discrete fourier transform ( DFT )

  - by Adam

  I'm writing a very simple in-place DFT. I am using the formula shown here: http://en.wikipedia.org/wiki/Discrete_Fourier_transform#Definition along with Euler's formula to avoid having to use a complex number class just for this. So far I have this: private void fft(double[] data) { double[] real = new double[256]; double[] imag = new double[256]; double pi_div_128 = -1 * Math.PI / 128; for (int k = 0; k < 256; k++) { for (int n = 0; n < 256; n++) { real[k] += data[k] * Math.Cos(pi_div_128 * k * n); imag[k] += data[k] * Math.Sin(pi_div_128 * k * n); } data[k] = Math.Sqrt(real[k] * real[k] + imag[k] * imag[k]); } } But the Math.Cos and Math.Sin terms eventually go both positive and negative, so as I'm adding those terms multiplied with data[k], they cancel out and I just get some obscenely small value. I see how it is happening, but I can't make sense of how my code is perhaps mis-representing the mathematics. Any help is appreciated. FYI, I do have to write my own, I realize I can get off-the shelf FFT's.

  Read the article

• DSP - Problems using the inverse Fast Fourier Transform

  - by Trap

  I've been playing around a little with the Exocortex implementation of the FFT, but I'm having some problems. First, after calculating the inverse FFT of an unchanged frequency spectrum obtained by a previous forward FFT, one would expect to get the original signal back, but this is not the case. I had to figure out that I needed to scale the FFT output to about 1 / fftLength to get the amplitudes ok. Why is this? Second, whenever I modify the amplitudes of the frequency bins before calling the iFFT the signal gets distorted at low frequencies. However, this does not happen if I attenuate all the bins by the same factor. Let me put a very simplified example of the output buffer of a 4-sample FFT: // Bin 0 (DC) FFTOut[0] = 0.0000610351563 FFTOut[1] = 0.0 // Bin 1 FFTOut[2] = 0.000331878662 FFTOut[3] = 0.000629425049 // Central bin FFTOut[4] = -0.0000381469727 FFTOut[5] = 0.0 // Bin 3, this is a negative frequency bin. FFTOut[6] = 0.000331878662 FFTOut[7] = -0.000629425049 The output is composed of pairs of floats, each representing the real and imaginay parts of a single bin. So, bin 0 (array indexes 0, 1) would represent the real and imaginary parts of the DC frequency. As you can see, bins 1 and 3 both have the same values, (except for the sign of the Im part), so I guess these are the negative frequency values, and finally indexes (4, 5) would be the central frequency bin. To attenuate the frequency bin 1 this is what I do: // Attenuate the 'positive' bin FFTOut[2] *= 0.5; FFTOut[3] *= 0.5; // Attenuate its corresponding negative bin. FFTOut[6] *= 0.5; FFTOut[7] *= 0.5; For the actual tests I'm using a 1024-length FFT and I always provide all the samples so no 0-padding is needed. // Attenuate var halfSize = fftWindowLength / 2; float leftFreq = 0f; float rightFreq = 22050f; for( var c = 1; c < halfSize; c++ ) { var freq = c * (44100d / halfSize); // Calc. positive and negative frequency locations. var k = c * 2; var nk = (fftWindowLength - c) * 2; // This kind of attenuation corresponds to a high-pass filter. // The attenuation at the transition band is linearly applied, could // this be the cause of the distortion of low frequencies? var attn = (freq < leftFreq) ? 0 : (freq < rightFreq) ? ((freq - leftFreq) / (rightFreq - leftFreq)) : 1; mFFTOut[ k ] *= (float)attn; mFFTOut[ k + 1 ] *= (float)attn; mFFTOut[ nk ] *= (float)attn; mFFTOut[ nk + 1 ] *= (float)attn; } Obviously I'm doing something wrong but can't figure out what or where.

  Read the article

• DSP - Filtering in the frequency domain via FFT

  - by Trap

  I've been playing around a little with the Exocortex implementation of the FFT, but I'm having some problems. Whenever I modify the amplitudes of the frequency bins before calling the iFFT the resulting signal contains some clicks and pops, especially when low frequencies are present in the signal (like drums or basses). However, this does not happen if I attenuate all the bins by the same factor. Let me put an example of the output buffer of a 4-sample FFT: // Bin 0 (DC) FFTOut[0] = 0.0000610351563 FFTOut[1] = 0.0 // Bin 1 FFTOut[2] = 0.000331878662 FFTOut[3] = 0.000629425049 // Bin 2 FFTOut[4] = -0.0000381469727 FFTOut[5] = 0.0 // Bin 3, this is the first and only negative frequency bin. FFTOut[6] = 0.000331878662 FFTOut[7] = -0.000629425049 The output is composed of pairs of floats, each representing the real and imaginay parts of a single bin. So, bin 0 (array indexes 0, 1) would represent the real and imaginary parts of the DC frequency. As you can see, bins 1 and 3 both have the same values, (except for the sign of the Im part), so I guess bin 3 is the first negative frequency, and finally indexes (4, 5) would be the last positive frequency bin. Then to attenuate the frequency bin 1 this is what I do: // Attenuate the 'positive' bin FFTOut[2] *= 0.5; FFTOut[3] *= 0.5; // Attenuate its corresponding negative bin. FFTOut[6] *= 0.5; FFTOut[7] *= 0.5; For the actual tests I'm using a 1024-length FFT and I always provide all the samples so no 0-padding is needed. // Attenuate var halfSize = fftWindowLength / 2; float leftFreq = 0f; float rightFreq = 22050f; for( var c = 1; c < halfSize; c++ ) { var freq = c * (44100d / halfSize); // Calc. positive and negative frequency indexes. var k = c * 2; var nk = (fftWindowLength - c) * 2; // This kind of attenuation corresponds to a high-pass filter. // The attenuation at the transition band is linearly applied, could // this be the cause of the distortion of low frequencies? var attn = (freq < leftFreq) ? 0 : (freq < rightFreq) ? ((freq - leftFreq) / (rightFreq - leftFreq)) : 1; // Attenuate positive and negative bins. mFFTOut[ k ] *= (float)attn; mFFTOut[ k + 1 ] *= (float)attn; mFFTOut[ nk ] *= (float)attn; mFFTOut[ nk + 1 ] *= (float)attn; } Obviously I'm doing something wrong but can't figure out what. I don't want to use the FFT output as a means to generate a set of FIR coefficients since I'm trying to implement a very basic dynamic equalizer. What's the correct way to filter in the frequency domain? what I'm missing? Thanks in advance.

  Read the article

• Loading XML file containing leading zeros with SSIS preserving the zeros

  - by Compudicted

  Visiting the MSDN SQL Server Integration Services Forum oftentimes I could see that people would pop up asking this question: “why I am not able to load an element from an XML file that contains zeros so the leading/trailing zeros would remain intact?”. I started to suspect that such a trivial and often-required operation perhaps is being misunderstood by the developer community. I would also like to add that the whole state of affairs surrounding the XML today is probably also going to be increasingly affected by a motion of people who dislike XML in general and many aspects of it as XSD and XSLT invoke a negative reaction at best. Nevertheless, XML is in wide use today and its importance as a bridge between diverse systems is ever increasing. Therefore, I deiced to write up an example of loading an arbitrary XML file that contains leading zeros in one of its elements using SSIS so the leading zeros would be preserved keeping in mind the goal on simplicity into a table in SQL Server database. To start off bring up your BIDS (running as admin) and add a new Data Flow Task (DFT). This DFT will serve as container to adding our XML processing elements (besides, the XML Source is not available anywhere else other than from within the DFT). Double-click your DFT and drag and drop the XMS Source component from the Tool Box’s Data Flow Sources. Now, let the fun begin! Being inspired by the upcoming Christmas I created a simple XML file with one set of data that contains an imaginary SSN number of Rudolph containing several leading zeros like 0000003. This file can be viewed here. To configure the XML Source of course it is quite intuitive to point it to our XML file, next what the XML source needs is either an embedded schema (XSD) or it can generate one for us. In lack of the one I opted to auto-generate it for me and I ended up with an XSD that looked like: <?xml version="1.0"?> <xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:element name="XMasEvent"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="CaseInfo"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="ID" type="xs:unsignedByte" /> <xs:element minOccurs="0" name="CreatedDate" type="xs:unsignedInt" /> <xs:element minOccurs="0" name="LastName" type="xs:string" /> <xs:element minOccurs="0" name="FirstName" type="xs:string" /> <xs:element minOccurs="0" name="SSN" type="xs:unsignedByte" /> <!-- Becomes string -- > <xs:element minOccurs="0" name="DOB" type="xs:unsignedInt" /> <xs:element minOccurs="0" name="Event" type="xs:string" /> <xs:element minOccurs="0" name="ClosedDate" /> </xs:sequence> </xs:complexType> </xs:element> </xs:sequence> </xs:complexType> </xs:element> </xs:schema> As an aside on the XML file: if your XML file does not contain the outer node (<XMasEvent>) then you may end up in a situation where you see just one field in the output. Now please note that the SSN element’s data type was chosen to be of unsignedByte (and this is for a reason). The reason is stemming from the fact all our figures in the element are digits, this is good, but this is not exactly what we need, because if we will attempt to load the data with this XSD then we are going to either get errors on the destination or most typically lose the leading zeros. So the next intuitive choice is to change the data type to string. Besides, if a SSIS package was already created based on this XSD and the data type change was done thereafter, one should re-set the metadata by right-clicking the XML Source and choosing “Advanced Editor” in which there is a refresh button at the bottom left which will do the trick. So far so good, we are ready to load our XML file, well actually yes, and no, in my experience typically some data conversion may be required. So depending on your data destination you may need to tweak the data types targeted. Let’s add a Data Conversion Task to our DFT. Your package should look like: To make the story short I only will cover the SSN field, so in my data source the target SQL Table has it as nchar(10) and we chose string in our XSD (yes, this is a big difference), under such circumstances the SSIS will complain. So will go and manipulate on the data type of SSN by making it Unicode String (DT_WSTR), World String per se. The conversion should look like: The peek at the Metadata: We are almost there, now all we need is to configure the destination. For simplicity I chose SQL Server Destination. The mapping is a breeze, F5 and I am able to insert my data into SQL Server now! Checking the zeros – they are all intact!

  Read the article

• How to capture a Header or Trailer Count Value in a Flat File and Assign to a Variable

  - by Compudicted

  Recently I had several questions concerning how to process files that carry a header and trailer in them. Typically those files are a product of data extract from non Microsoft products e.g. Oracle database encompassing various tables data where every row starts with an identifier. For example such a file data record could look like: HDR,INTF_01,OUT,TEST,3/9/2011 11:23 B1,121156789,DATA TEST DATA,2011-03-09 10:00:00,Y,TEST 18 10:00:44,2011-07-18 10:00:44,Y B2,TEST DATA,2011-03-18 10:00:44,Y B3,LEG 1 TEST DATA,TRAN TEST,N B4,LEG 2 TEST DATA,TRAN TEST,Y FTR,4,TEST END,3/9/2011 11:27 A developer is normally able to break the records using a Conditional Split Transformation component by employing an expression similar to Output1 -- SUBSTRING(Output1,1,2) == "B1" and so on, but often a verification is required after this step to check if the number of data records read corresponds to the number specified in the trailer record of the file. This portion sometimes stumbles some people so I decided to share what I came up with. As an aside, I want to mention that the approach I use is slightly more portable than some others I saw because I use a separate DFT that can be copied and pasted into a new SSIS package designer surface or re-used within the same package again and it can survive several trailer/footer records (!). See how a ready DFT can look: The first step is to create a Flat File Connection Manager and make sure you get the row split into columns like this: After you are done with the Flat File connection, move onto adding an aggregate which is in use to simply assign a value to a variable (here the aggregate is used to handle the possibility of multiple footers/headers): The next step is adding a Script Transformation as destination that requires very little coding. First, some variable setup: and finally the code: As you can see it is important to place your code into the appropriate routine in the script, otherwise the end result may not be as expected. As the last step you would use the regular Script Component to compare the variable value obtained from the DFT above to a package variable value obtained say via a Row Count component to determine if the file being processed has the right number of rows.

  Read the article

• SSIS Technique to Remove/Skip Trailer and/or Bad Data Row in a Flat File

  - by Compudicted

  I noticed that the question on how to skip or bypass a trailer record or a badly formatted/empty row in a SSIS package keeps coming back on the MSDN SSIS Forum. I tried to figure out the reason why and after an extensive search inside the forum and outside it on the entire Web (using several search engines) I indeed found that it seems even thought there is a number of posts and articles on the topic none of them are employing the simplest and the most efficient technique. When I say efficient I mean the shortest time to solution for the fellow developers. OK, enough talk. Let’s face the problem: Typically a flat file (e.g. a comma delimited/CSV) needs to be processed (loaded into a database in most cases really). Oftentimes, such an input file is produced by some sort of an out of control, 3-rd party solution and would come in with some garbage characters and/or even malformed/miss-formatted rows. One such example could be this imaginary file: As you can see several rows have no data and there is an occasional garbage character (1, in this example on row #7). Our task is to produce a clean file that will only capture the meaningful data rows. As an aside, our output/target may be a database table, but for the purpose of this exercise we will simply re-format the source. Let’s outline our course of action to start off: Will use SSIS 2005 to create a DFT; The DFT will use a Flat File Source to our input [bad] flat file; We will use a Conditional Split to process the bad input file; and finally Dump the resulting data to a new [clean] file. Well, only four steps, let’s see if it is too much of work. 1: Start the BIDS and add a DFT to the Control Flow designer (I named it Process Dirty File DFT): 2, and 3: I had added the data viewer to just see what I am getting, alas, surprisingly the data issues were not seen it:   What really is the key in the approach it is to properly set the Conditional Split Transformation. Visually it is: and specifically its SSIS Expression LEN([After CS Column 0]) > 1 The point is to employ the right Boolean expression (yes, the Conditional Split accepts only Boolean conditions). For the sake of this post I re-named the Output Name “No Empty Rows”, but by default it will be named Case 1 (remember to drag your first column into the expression area)! You can close your Conditional Split now. The next part will be crucial – consuming the output of our Conditional Split. Last step - #4: Add a Flat File Destination or any other one you need. Click on the Conditional Split and choose the green arrow to drop onto the target. When you do so make sure you choose the No Empty Rows output and NOT the Conditional Split Default Output. Make the necessary mappings. At this point your package must look like: As the last step will run our package to examine the produced output file. F5: and… it looks great!

  Read the article

• Tool to check if XML is valid in my VS2012 comments

  - by davidjr

  I am writing the documentation for our companies software developed with vs2012. I need to add xml examples to the summary of each class, due to xml instantiation of objects. We are using sandcastle to create the documentation (company choice), and I want to be able to review my xml comments without building the help file every time. Is there an application that anyone would recommend where I can view how the xml renders before I build the help file? Here is my example: /// <summary> /// Performs DFT on a data array, writes output in a CSV file. /// </summary> /// <example> /// <para>XML declaration</para> /// <code lang="xml" xml:space="preserve"> /// %lt;DataProvider name="DftDP" description="Computes DFT" etc... I want to check the XML to make sure it is valid, maybe by copy and pasting it into a tool of some sort?

  Read the article

• Displaying Fourier transforms in OpenCV

  - by Simonw

  Hi, I'm just learning to use OpenCV and am having a problem with using DFT. I've done a signal processing class which used MatLab, so I'm trying to go through some of the exercises we did in that class. I'm trying to get and display the FT of an image, so I can mask some of the frequencies. I'd like to be able to see the FT, so I know how big to make the mask, but when I tried, I got an image like this: rather than like one of these Am I forgetting a step somewhere? I'm loading the image, converting its type to CV_32FC1, getting the matrix of it, getting the DFT, and then getting turning the resulting matrix back into an image. I'll post the code I'm using if it will be of any help? Or if someone has a link to an example of displaying the FT? I could only find ones which used it for the convolution. EDIT: Did I get the Phase of the image?

  Read the article

• Difficulty creating classes and arrays of those classes C#

  - by Lucifer Fayte

  I'm trying to implement a Discrete Fourier Transformation algorithm for a project I'm doing in school. But creating a class is seeming to be difficult(which it shouldn't be). I'm using Visual Studio 2012. Basically I need a class called Complex to store the two values I get from a DFT; The real portion and the imaginary portion. This is what I have so far for that: using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace SoundEditor_V3 { public class Complex { public double real; public double im; public Complex() { real = 0; im = 0; } } } The problem is that it doesn't recognize the constructor as a constructor, now I'm just learning C#, but I looked it up online and this is how it's supposed to look apparently. It recognizes my constructor as a method. Why is that? Am I creating the class wrong? It's doing the same thing for my Fourier class as well. So each time I try to create a Fourier object and then use it's method...there is no such thing. example, I do this: Fourier fou = new Fourier(); fou.DFT(s, N, amp, 0); and it tells me fou is a 'field' but is used like a 'type' why is it saying that? Here is the code for my Fourier class as well: using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace SoundEditor_V3 { public class Fourier { //FOURIER //N = number of samples //s is the array of samples(data) //amp is the array where the complex result will be written to //start is the where in the array to start public void DFT(byte[] s, int N, ref Complex[] amp, int start) { Complex tem = new Complex(); int f; int t; for (f = 0; f < N; f++) { tem.real = 0; tem.im = 0; for (t = 0; t < N; t++) { tem.real += s[t + start] * Math.Cos(2 * Math.PI * t * f / N); tem.im -= s[t + start] * Math.Sin(2 * Math.PI * t * f / N); } amp[f].real = tem.real; amp[f].im = tem.im; } } //INVERSE FOURIER public void IDFT(Complex[] A, ref int[] s) { int N = A.Length; int t, f; double result; for (t = 0; t < N; t++) { result = 0; for (f = 0; f < N; f++) { result += A[f].real * Math.Cos(2 * Math.PI * t * f / N) - A[f].im * Math.Sin(2 * Math.PI * t * f / N); } s[t] = (int)Math.Round(result); } } } } I'm very much stuck at the moment, any and all help would be appreciated. Thank you.

  Read the article

• Flash computeSpectrum() unsynchronized with audio

  - by sold

  I am using Flash's (CS4, AS3) SoundMixer.computeSpectrum to visualize a DFT of what supposed to be, according to the docs, whatever is currently being played. However, there is a considerable delay between the audio and the visualization (audio comes later). It seems that computeSpectrum captures whatever is on it's way to the buffer, and not to the speakers. Any cure for this?

  Read the article

• Computer POST's and draws the BIOS screens very slowly - Motherboard issue?

  - by Ssvarc

  I have a desktop that refuses to boot into Windows. I used Hitachi DFT and the HD came back OK. I then used Memtest86+ and it took hours for the test to run. After 8+ hours it was up to test number 6. I aborted and ran Memtest86. It ran at basically the same speed. I aborted and went to look at the BIOS settings. The computer is running slow at POST. It takes a long time for the keyboard to be recognized, etc. The BIOS settings takes time to be (slowly) drawn on the screen. What could be causing such behavior? EDIT: I gave back the computer a while back without ever discovering the cause so I'm closing the question.

  Read the article

• Dell Vostro 1000 w/fresh XP Pro installation running very slow

  - by Ssvarc

  I'm working on a Dell Vostro 1000 laptop. It was running slowly and hanging, antivirus scans turned up nothing and the same with the Dell diagnostics (DFT hung - but Dell's diagnostics gave the HD a pass) so I imaged the HD and installed XP Pro on it. The laptop is still slow! (It takes a noticeable delay to enter words into the Google search box in IE and the computer hangs every time it enters the XP screensaver.) What are the likely possibilities at this point and how would I go about checking for them?

  Read the article

• Dell Vostro 1000 w/fresh XP Pro installation running very slow

  - by ssvarc

  I'm working on a Dell Vostro 1000 laptop. It was running slowly and hanging, antivirus scans turned up nothing and the same with the Dell diagnostics (DFT hung - but Dell's diagnostics gave the HD a pass) so I imaged the HD and installed XP Pro on it. The laptop is still slow! (It takes a noticeable delay to enter words into the Google search box in IE and the computer hangs every time it enters the XP screensaver.) What are the likely possibilities at this point and how would I go about checking for them?

  Read the article

• "Winamp style" spectrum analyzer

  - by cvb

  I have a program that plots the spectrum analysis (Amp/Freq) of a signal, which is preety much the DFT converted to polar. However, this is not exactly the sort of graph that, say, winamp (right at the top-left corner), or effectively any other audio software plots. I am not really sure what is this sort of graph called (if it has a distinct name at all), so I am not sure what to look for. I am preety positive about the frequency axis being base two exponential, the amplitude axis puzzles me though. Any pointers?

  Read the article

• Merge Join component sorted outputs [SSIS]

  - by jamiet

  One question that I have been asked a few times of late in regard to performance tuning SSIS data flows is this: Why isn’t the Merge Join output sorted (i.e.IsSorted=True)? This is a fair question. After all both of the Merge Join inputs are sorted, hence why wouldn’t the output be sorted as well? Well here’s a little secret, the Merge Join output IS sorted! There’s a caveat though – it is only under certain circumstances and SSIS itself doesn’t do a good job of informing you of it. Let’s take a look at an example. Here we have a dataflow that consumes data from the [AdventureWorks2008].[Sales].[SalesOrderHeader] & [AdventureWorks2008].[Sales].[SalesOrderDetail] tables then joins them using a Merge Join component: Let’s take a look inside the editor of the Merge Join: We are joining on the [SalesOrderId] field (which is what the two inputs just happen to be sorted upon). We are also putting [SalesOrderHeader].[SalesOrderId] into the output. Believe it or not the output from this Merge Join component is sorted (i.e. has IsSorted=True) but unfortunately the Merge Join component does not have an Advanced Editor hence it is hidden away from us. There are a couple of ways to prove to you that is the case; I could open up the package XML inside the .dtsx file and show you the metadata but there is an easier way than that – I can attach a Sort component to the output. Take a look: Notice that the Sort component is attempting to sort on the [SalesOrderId] column. This gives us the following warning: Validation warning. DFT Get raw data: {992B7C9A-35AD-47B9-A0B0-637F7DDF93EB}: The data is already sorted as specified so the transform can be removed. The warning proves that the output from the Merge Join is sorted! It must be noted that the Merge Join output will only have IsSorted=True if at least one of the join columns is included in the output. So there you go, the Merge Join component can indeed produce a sorted output and that’s very useful in order to avoid unnecessary expensive Sort operations downstream. Hope this is useful to someone out there! @Jamiet  P.S. Thank you to Bob Bojanic on the SSIS product team who pointed this out to me!

  Read the article

• Merge Join component sorted outputs [SSIS]

  - by jamiet

  One question that I have been asked a few times of late in regard to performance tuning SSIS data flows is this: Why isn’t the Merge Join output sorted (i.e.IsSorted=True)? This is a fair question. After all both of the Merge Join inputs are sorted, hence why wouldn’t the output be sorted as well? Well here’s a little secret, the Merge Join output IS sorted! There’s a caveat though – it is only under certain circumstances and SSIS itself doesn’t do a good job of informing you of it. Let’s take a look at an example. Here we have a dataflow that consumes data from the [AdventureWorks2008].[Sales].[SalesOrderHeader] & [AdventureWorks2008].[Sales].[SalesOrderDetail] tables then joins them using a Merge Join component: Let’s take a look inside the editor of the Merge Join: We are joining on the [SalesOrderId] field (which is what the two inputs just happen to be sorted upon). We are also putting [SalesOrderHeader].[SalesOrderId] into the output. Believe it or not the output from this Merge Join component is sorted (i.e. has IsSorted=True) but unfortunately the Merge Join component does not have an Advanced Editor hence it is hidden away from us. There are a couple of ways to prove to you that is the case; I could open up the package XML inside the .dtsx file and show you the metadata but there is an easier way than that – I can attach a Sort component to the output. Take a look: Notice that the Sort component is attempting to sort on the [SalesOrderId] column. This gives us the following warning: Validation warning. DFT Get raw data: {992B7C9A-35AD-47B9-A0B0-637F7DDF93EB}: The data is already sorted as specified so the transform can be removed. The warning proves that the output from the Merge Join is sorted! It must be noted that the Merge Join output will only have IsSorted=True if at least one of the join columns is included in the output. So there you go, the Merge Join component can indeed produce a sorted output and that’s very useful in order to avoid unnecessary expensive Sort operations downstream. Hope this is useful to someone out there! @Jamiet  P.S. Thank you to Bob Bojanic on the SSIS product team who pointed this out to me!

  Read the article

• SSIS and StreamInsight Working Together.

  I have been thinking a lot recently about what it would be like to have StreamInsight and SSIS working together.  Well the CAT team have produced a paper on some of our options here. Here are some of my thoughts. There is of course a slight mismatch in their types of usage.  StreamInsight is an Event Stream processing engine capable of operating on new data in the sub second timeframe.  The engine allows you to do real time analytics and take decisions on events that have potentially only just happened.  SSIS on the other hand is a batch processing engine.  In general I do not like having to invoke the same package more than once every 90 seconds or so as it can start to get expensive.  Usually when doing batch processing we have an hour or longer of grace before we have to move data from A –> B. StreamInsight operates on streams of data.  Before anyone mentions it yes I know StreamInsight is equally adept at using the IEnumerable interface, but I would argue live streaming and real-time analytics is a primary goal of the product.  SSIS does not have an “Always On” button I do not like the idea of embedding StreamInsight inside SSIS using a transform particularly.  It means StreamInsight becomes a batch processing engine because it can only operate when the SSIS package is running and SSIS is in charge of when that happens. If I am to have StreamInsight within SSIS then I prefer to have StreamInsight on the adapters.  This way you can force the adapters to stay open and introduce events into your Pipeline.   SSIS has a much richer set of transforms out of the box than StreamInsight.  Although “Always On” was not a design goal of SSIS I have used it like this and it works just fine. SSIS being called from within StreamInsight, now that excites me.  see below   For a while now I have been thinking what it would be like to decouple the Data Flow task from the SSIS package and expose it as something with which you can interact.  Anything can instantiate this version of a DFT as it would expose one or more  input interfaces and one or more output interfaces.  I can imagine that this would be a big hit when moving to “The Cloud” as well.  I could see the Data Flow task maybe being hosted in Azure Appfabric or some such layer. StreamInsight would be able to take advantage of this as well.   I am interested to see where this goes and will be pressing for more meat around the subject when I visit Redmond soon.

  Read the article

• Reduce Multiple Errors logging in sysssislog

  - by Akshay

  Need help. I am trying to automate error notifications to be sent in mailers. For that I am querying the sysssislog table. I have pasted an "Execute SQl task" on the package event handler "On error". For testing purpose, I am deliberately trying to load duplicate keys in a table which consists of a Primary key column(so as to get an error). But instead of having just one error, "Violation of primary key constraint", SSIS records 3 in the table. PFA the screenshot as well. How can i restrict the tool to log only one error and not multiple ??? Package Structure. Package ("On error Event handler") - DFT - Oledb Source - Oledb Destination SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "The statement has been terminated.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Violation of PRIMARY KEY constraint 'PK_SalesPerson_SalesPersonID'. Cannot insert duplicate key in object 'dbo.SalesPerson'.". SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "OLE DB Destination Input" (56)" failed because error code 0xC020907B occurred, and the error row disposition on "input "OLE DB Destination Input" (56)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure. SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "OLE DB Destination" (43) failed with error code 0xC0209029 while processing input "OLE DB Destination Input" (56). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure. Please guide me. Your help is very much appreciated. Thanks

  Read the article

• Android library to get pitch from WAV file

  - by Sakura

  I have a list of sampled data from the WAV file. I would like to pass in these values into a library and get the frequency of the music played in the WAV file. For now, I will have 1 frequency in the WAV file and I would like to find a library that is compatible with Android. I understand that I need to use FFT to get the frequency domain. Is there any good libraries for that? I found that [KissFFT][1] is quite popular but I am not very sure how compatible it is on Android. Is there an easier and good library that can perform the task I want? EDIT: I tried to use JTransforms to get the FFT of the WAV file but always failed at getting the correct frequency of the file. Currently, the WAV file contains sine curve of 440Hz, music note A4. However, I got the result as 441. Then I tried to get the frequency of G4, I got the result as 882Hz which is incorrect. The frequency of G4 is supposed to be 783Hz. Could it be due to not enough samples? If yes, how much samples should I take? //DFT DoubleFFT_1D fft = new DoubleFFT_1D(numOfFrames); double max_fftval = -1; int max_i = -1; double[] fftData = new double[numOfFrames * 2]; for (int i = 0; i < numOfFrames; i++) { // copying audio data to the fft data buffer, imaginary part is 0 fftData[2 * i] = buffer[i]; fftData[2 * i + 1] = 0; } fft.complexForward(fftData); for (int i = 0; i < fftData.length; i += 2) { // complex numbers -> vectors, so we compute the length of the vector, which is sqrt(realpart^2+imaginarypart^2) double vlen = Math.sqrt((fftData[i] * fftData[i]) + (fftData[i + 1] * fftData[i + 1])); //fd.append(Double.toString(vlen)); // fd.append(","); if (max_fftval < vlen) { // if this length is bigger than our stored biggest length max_fftval = vlen; max_i = i; } } //double dominantFreq = ((double)max_i / fftData.length) * sampleRate; double dominantFreq = (max_i/2.0) * sampleRate / numOfFrames; fd.append(Double.toString(dominantFreq)); Can someone help me out? EDIT2: I manage to fix the problem mentioned above by increasing the number of samples to 100000, however, sometimes I am getting the overtones as the frequency. Any idea how to fix it? Should I use Harmonic Product Frequency or Autocorrelation algorithms?

  Read the article

1