# Search Results

• ### Unexpected performance curve from CPython merge sort

##### - by vkazanov
I have implemented a naive merge sorting algorithm in Python. Algorithm and test code is below: import time import random import matplotlib.pyplot as plt import math from collections import deque def sort(unsorted): if len(unsorted) <= 1: return unsorted to_merge = deque(deque([elem]) for elem in unsorted) while len(to_merge) > 1: left = to_merge.popleft() right = to_merge.popleft() to_merge.append(merge(left, right)) return to_merge.pop() def merge(left, right): result = deque() while left or right: if left and right: elem = left.popleft() if left[0] > right[0] else right.popleft() elif not left and right: elem = right.popleft() elif not right and left: elem = left.popleft() result.append(elem) return result LOOP_COUNT = 100 START_N = 1 END_N = 1000 def test(fun, test_data): start = time.clock() for _ in xrange(LOOP_COUNT): fun(test_data) return time.clock() - start def run_test(): timings, elem_nums = [], [] test_data = random.sample(xrange(100000), END_N) for i in xrange(START_N, END_N): loop_test_data = test_data[:i] elapsed = test(sort, loop_test_data) timings.append(elapsed) elem_nums.append(len(loop_test_data)) print "%f s --- %d elems" % (elapsed, len(loop_test_data)) plt.plot(elem_nums, timings) plt.show() run_test() As much as I can see everything is OK and I should get a nice N*logN curve as a result. But the picture differs a bit: Things I've tried to investigate the issue: PyPy. The curve is ok. Disabled the GC using the gc module. Wrong guess. Debug output showed that it doesn't even run until the end of the test. Memory profiling using meliae - nothing special or suspicious. ` I had another implementation (a recursive one using the same merge function), it acts the similar way. The more full test cycles I create - the more "jumps" there are in the curve. So how can this behaviour be explained and - hopefully - fixed? UPD: changed lists to collections.deque UPD2: added the full test code UPD3: I use Python 2.7.1 on a Ubuntu 11.04 OS, using a quad-core 2Hz notebook. I tried to turn of most of all other processes: the number of spikes went down but at least one of them was still there.

• ### SQL Server: preventing dirty reads in a stored procedure

##### - by pcampbell
Consider a SQL Server database and its two stored procs: *1. A proc that performs 3 important things in a transaction: Create a customer, call a sproc to perform another insert, and conditionally insert a third record with the new identity. BEGIN TRAN INSERT INTO Customer(CustName) (@CustomerName) SELECT @NewID = SCOPE_IDENTITY() EXEC CreateNewCustomerAccount @NewID, @CustomerPhoneNumber IF @InvoiceTotal > 100000 INSERT INTO PreferredCust(InvoiceTotal, CustID) VALUES (@InvoiceTotal, @NewID) COMMIT TRAN *2. A stored proc which polls the Customer table for new entries that don't have a related PreferredCust entry. The client app performs the polling by calling this stored proc every 500ms. A problem has arisen where the polling stored procedure has found an entry in the Customer table, and returned it as part of its results. The problem was that it has picked up that record, I am assuming, as part of a dirty read. The record ended up having an entry in PreferredCust later, and ended up creating a problem downstream. Question How can you explicitly prevent dirty reads by that second stored proc? The environment is SQL Server 2005 with the default configuration out of the box. No other locking hits are given in either of these stored procedures.

• ### How to allow users to define financial formulas in a C# app

##### - by Peter Morris
I need to allow my users to be able to define formulas which will calculate values based on data. For example //Example 1 return GetMonetaryAmountFromDatabase("Amount due") * 1.2; //Example 2 return GetMonetaryAmountFromDatabase("Amount due") * GetFactorFromDatabase("Discount"); I will need to allow / * + - operations, also to assign local variables and execute IF statements, like so var amountDue = GetMonetaryAmountFromDatabase("Amount due"); if (amountDue > 100000) return amountDue * 0.75; if (amountDue > 50000) return amountDue * 0.9; return amountDue; The scenario is complicated because I have the following structure.. Customer (a few hundred) Configuration (about 10 per customer) Item (about 10,000 per customer configuration) So I will perform a 3 level loop. At each "Configuration" level I will start a DB transaction and compile the forumlas, each "Item" will use the same transaction + compiled formulas (there are about 20 formulas per configuration, each item will use all of them). This further complicates things because I can't just use the compiler services as it would result in continued memory usage growth. I can't use a new AppDomain per each "Configuration" loop level because some of the references I need to pass cannot be marshalled. Any suggestions?

• ### Java Performance measurement

##### - by portoalet
Hi, I am doing some Java performance comparison between my classes, and wondering if there is some sort of Java Performance Framework to make writing performance measurement code easier? I.e, what I am doing now is trying to measure what effect does it have having a method as "synchronized" as in PseudoRandomUsingSynch.nextInt() compared to using an AtomicInteger as my "synchronizer". So I am trying to measure how long it takes to generate random integers using 3 threads accessing a synchronized method looping for say 10000 times. I am sure there is a much better way doing this. Can you please enlighten me? :) public static void main( String [] args ) throws InterruptedException, ExecutionException { PseudoRandomUsingSynch rand1 = new PseudoRandomUsingSynch((int)System.currentTimeMillis()); int n = 3; ExecutorService execService = Executors.newFixedThreadPool(n); long timeBefore = System.currentTimeMillis(); for(int idx=0; idx<100000; ++idx) { Future<Integer> future = execService.submit(rand1); Future<Integer> future1 = execService.submit(rand1); Future<Integer> future2 = execService.submit(rand1); int random1 = future.get(); int random2 = future1.get(); int random3 = future2.get(); } long timeAfter = System.currentTimeMillis(); long elapsed = timeAfter - timeBefore; out.println("elapsed:" + elapsed); } the class public class PseudoRandomUsingSynch implements Callable<Integer> { private int seed; public PseudoRandomUsingSynch(int s) { seed = s; } public synchronized int nextInt(int n) { byte [] s = DonsUtil.intToByteArray(seed); SecureRandom secureRandom = new SecureRandom(s); return ( secureRandom.nextInt() % n ); } @Override public Integer call() throws Exception { return nextInt((int)System.currentTimeMillis()); } } Regards

• ### Disable second dropdown menu before first is populated

##### - by johnny-kessel
I need to grey out the second jump box (Select a Subcategory) before the first (Choose a Category) has a valid selection ..Here is the current code.. thanks guys <script type="text/javascript"> \\$j(document).ready(function() { \\$j('.subf_dropdown').html(\$j('.subf_dropdown').html()); }); function chooseForum(f, name) { \\$j.ajax({ url: 'index.php?autocom=cats&root=' + f, type: 'GET', timeout: 100000, error: function(){ alert('Oops something went wrong. Please try again'); }, success: function(xml){ \\$j('.subf_dropdown').html("<optgroup label='Subcategories'> " + xml + " </optgroup>"); } }); } function newPostInForum(f) { if (f != "") { window.location = "http://www.xxx.co.za/?act=post&do=new_post&f=" + f; } } </script> <select name='f' class='f_dropdown' onchange="chooseForum(this.value, this); return false;"> <optgroup label="Choose a Category"> {\$data} </optgroup> </select> <br /><br /> <select name='subf' class='subf_dropdown' onchange="newPostInForum(this.value); return false;"> <optgroup label="Subcategories"> <option value="" selected="selected">Select a Subcategory</option> </optgroup> </select>

• ### Implementing java FixedTreadPool status listener

##### - by InsertNickHere
Hi there, it's about an application which is supposed to process (VAD, Loudness, Clipping) a lot of soundfiles (e.g. 100k). At this time, I create as many worker threads (callables) as I can put into memory, and then run all with a threadPool.invokeAll(), write results to file system, unload processed files and continue at step 1. Due to the fact it's an app with a GUI, i don't want to user to feel like the app "is not responding" while processing all soundfiles. (which it does at this time cause invokeAll is blocking). Im not sure what is a "good" way to fix this. It shall not be possible for the user to do other things while processing, but I'd like to show a progress bar like "10 of 100000 soundfiles are done". So how do I get there? Do I have to create a "watcher thread", so that every worker hold a callback on it? I'm quite new to multi threading, and don't get the idea of such a mechanisem.. If you need to know: I'm using SWT/JFace. Regards, InsertNickHere

• ### How can I store large amount of data from a database to XML (memory problem)?

##### - by Andrija
First, I had a problem with getting the data from the Database, it took too much memory and failed. I've set -Xmx1500M and I'm using scrolling ResultSet so that was taken care of. Now I need to make an XML from the data, but I can't put it in one file. At the moment, I'm doing it like this: while(rs.next()){ i++; xmlStringBuilder.append("\n\t<row>"); xmlStringBuilder.append("\n\t\t<ID>" + Util.transformToHTML(rs.getInt("id")) + "</ID>"); xmlStringBuilder.append("\n\t\t<JED_ID>" + Util.transformToHTML(rs.getInt("jed_id")) + "</JED_ID>"); xmlStringBuilder.append("\n\t\t<IME_PJ>" + Util.transformToHTML(rs.getString("ime_pj")) + "</IME_PJ>"); //etc. xmlStringBuilder.append("\n\t</row>"); if (i%100000 == 0){ //stores the data to a file with the name i.xml storeKBR(xmlStringBuilder.toString(),i); xmlStringBuilder= null; xmlStringBuilder= new StringBuilder(); } and it works; I get 12 100 MB files. Now, what I'd like to do is to do is have all that data in one file (which I then compress) but if just remove the if part, I go out of memory. I thought about trying to write to a file, closing it, then opening, but that wouldn't get me much since I'd have to load the file to memory when I open it. P.S. If there's a better way to release the Builder, do let me know :)

• ### iphone encode problem with ffmpeg

##### - by samantha
Hi, I need to encode a video from image. I use ffmpeg and compiling rigth. My problem is that when i try to opne video with quicktime on iphone, this give me a message "this movie format is not supported". I create a file mp4 with this parameter on context: context-time_base.num = 1; context-time_base.den = 15; context-codec_type = CODEC_TYPE_VIDEO; context-codec_id = CODEC_ID_H264; context-bit_rate = 1000000; context-width = width; context-height = height; context-keyint_min = 10; context-i_quant_factor = 0.71; context-bit_rate_tolerance = 20000; context-rc_max_rate = 100000; context-rc_buffer_size = 8835000; context-qcompress = 0.6; context-qmin = 10; context-qmax = 30; context-max_qdiff = 4; context-gop_size = 30; context->time_base.num = 1; context-time_base.den = 30; context-sample_aspect_ratio = av_d2q(1, 255); context-profile = 30; context-pix_fmt = PIX_FMT_YUV420P; context-flags |= CODEC_FLAG_LOOP_FILTER; where is my mistake?? thanks

• ### Full Text Search like Google

##### - by Eduardo
I would like to implement full-text-search in my off-line (android) application to search the user generated list of notes. I would like it to behave just like Google (since most people are already used to querying to Google) My initial requirements are: Fast: like Google or as fast as possible, having 100000 documents with 200 hundred words each. Searching for two words should only return documents that contain both words (not just one word) (unless the OR operator is used) Case insensitive (aka: normalization): If I have the word 'Hello' and I search for 'hello' it should match. Diacritical mark insensitive: If I have the word 'así' a search for 'asi' should match. In Spanish, many people, incorrectly, either do not put diacritical marks or fail in correctly putting them. Stop word elimination: To not have a huge index meaningless words like 'and', 'the' or 'for' should not be indexed at all. Dictionary substitution (aka: stem words): Similar words should be indexed as one. For example, instances of 'hungrily' and 'hungry' should be replaced with 'hunger'. Phrase search: If I have the text 'Hello world!' a search of '"world hello"' should not match it but a search of '"hello world"' should match. Search all fields (in multifield documents) if no field specified (not just a default field) Auto-completion in search results while typing to give popular searches. (just like Google Suggest) How may I configure a full-text-search engine to behave as much as possible as Google? (I am mostly interested in Open Source, Java and in particular Lucene)

• ### (outofmemoryerror: java heap space) when iterating through oracle records...

##### - by rockit
hello fellow java developers. I'm having a bit of an issue here. I have code that gets a resultset from an oracle database, prints each row to a file, then gets the next row - and continues till the end of the resultset. Only this isn't what happens. What happens is that it gets the resultset, starts iterating through the rows, printing to file as it goes, until it runs out of memory - claiming it needs more space on the java heap. The app is currently running with 2g of memory on the heap and the code breaks at about the 150000th row. I'm using jodbc6.jar and java 6 Here is an idea of what my code is doing: Connection conn = DriverManager.getConnection(url,"name","pwd"); conn.setAutoCommit(false); Statement stmt = conn.createStatement(); ResultSet rset = stmt.executeQuery(strSql); String strVar_1 = null; long lCount = 0; while(rset.next()){ lCount++; if (lCount % 100000 == 0){ System.out.println(lCount + " rows completed"); } strVar_1 = rset.getString("StringID"); /// breaks here!!!!!!!!! if (strVar_1 == null){ strVar_1 = ""; } if (!strQuery_1.equals("")){ out.write(strVar_1 + "\n"); } } out.close();

• ### Is there really such a thing as a char or short in modern programming?

##### - by Dean P
Howdy all, I've been learning to program for a Mac over the past few months (I have experience in other languages). Obviously that has meant learning the Objective C language and thus the plainer C it is predicated on. So I have stumbles on this quote, which refers to the C/C++ language in general, not just the Mac platform. With C and C++ prefer use of int over char and short. The main reason behind this is that C and C++ perform arithmetic operations and parameter passing at integer level, If you have an integer value that can fit in a byte, you should still consider using an int to hold the number. If you use a char, the compiler will first convert the values into integer, perform the operations and then convert back the result to char. So my question, is this the case in the Mac Desktop and IPhone OS environments? I understand when talking about theses environments we're actually talking about 3-4 different architectures (PPC, i386, Arm and the A4 Arm variant) so there may not be a single answer. Nevertheless does the general principle hold that in modern 32 bit / 64 bit systems using 1-2 byte variables that don't align with the machine's natural 4 byte words doesn't provide much of the efficiency we may expect. For instance, a plain old C-Array of 100,000 chars is smaller than the same 100,000 ints by a factor of four, but if during an enumeration, reading out each index involves a cast/boxing/unboxing of sorts, will we see overall lower 'performance' despite the saved memory overhead?

• ### Efficient algorithm for Next button on a MySQL result set

##### - by David Grayson
I have a website that lets people view rows in a table (each row is a picture). There are more than 100,000 rows. You can view different subsets of the rows, and you can view them with different sort orders. While you are viewing one of the rows, you can click the "Next" or "Previous" buttons to go the next/previous row in the list. How would you implement the "Next" and "Previous" features of the website? More specifically, if you have an arbitrary query that returns a list of up to 100,000+ rows, and you know some information about the current row someone is viewing, how do you determine the NEXT row efficiently? Here is the pseudo-code of the solution I came up with when the website was young, and it worked well when there were only 1000 rows, but now that there are 100,000 rows I think it is eating up too much memory. int nextRowId(string query, int currentRowId) { array allRowIds = mysql_query(query); // Takes up a lot of memory! int currentIndex = (index of currentRowId in allRowIds); // Takes time! return allRowIds[currentIndex+1]; } While you are thinking about this problem, remember that the website can store more information about the current row than just its ID (for example, the position of the current row in the result set), and this information can be used as a hint to help determine the ID of the next row. Edit: Sorry for not mentioning this earlier, but this isn't just a static website: rows can often be added to the list, and rows can be re-ordered in the list. (Much rarer, rows can be removed from the list.) I think that I should worry about that kind of thing, but maybe you can convince me otherwise.

• ### How can I pass a const array or a variable array to a function in C?

##### - by CSharperWithJava
I have a simple function Bar that uses a set of values from a data set that is passed in in the form of an Array of data structures. The data can come from two sources: a constant initialized array of default values, or a dynamically updated cache. The calling function determines which data is used and should be passed to Bar. Bar doesn't need to edit any of the data and in fact should never do so. How should I declare Bar's data parameter so that I can provide data from either set? union Foo { long _long; int _int; } static const Foo DEFAULTS[8] = {1,10,100,1000,10000,100000,1000000,10000000}; static Foo Cache[8] = {0}; void Bar(Foo* dataSet, int len);//example function prototype Note, this is C, NOT C++ if that makes a difference; Edit Oh, one more thing. When I use the example prototype I get a type qualifier mismatch warning, (because I'm passing a mutable reference to a const array?). What do I have to change for that?

• ### Getting wierd issue with TO_NUMBER function in Oracle

##### - by Fazal
I have been getting an intermittent issue when executing to_number function in the where clause on a varchar2 column if number of records exceed a certain number n. I used n as there is no exact number of records on which it happens. On one DB it happens after n was 1 million on another when it was 0.1. million. E.g. I have a table with 10 million records say Table Country which has field1 varchar2 containing numberic data and Id If I do a query as an example select * from country where to_number(field1) = 23 and id 1 and id < 100000 This works But if i do the query select * from country where to_number(field1) = 23 and id 1 and id < 100001 It fails saying invalid number Next I try the query select * from country where to_number(field1) = 23 and id 2 and id < 100001 It works again As I only got invalid number it was confusing, but in the log file it said Memory Notification: Library Cache Object loaded into SGA Heap size 3823K exceeds notification threshold (2048K) KGL object name :with sqlplan as ( select c006 object_owner, c007 object_type,c008 object_name from htmldb_collections where COLLECTION_NAME='HTMLDB_QUERY_PLAN' and c007 in ('TABLE','INDEX','MATERIALIZED VIEW','INDEX (UNIQUE)')), ws_schemas as( select schema from wwv_flow_company_schemas where security_group_id = :flow_security_group_id), t as( select s.object_owner table_owner,s.object_name table_name, d.OBJECT_ID from sqlplan s,sys.dba_objects d It seems its related to SGA size, but google did not give me much help on this. Does anyone have any idea about this issue with TO_NUMBER or oracle functions for large data?

• ### How to test if a user has SELECTED a file to upload ?

##### - by Tristan
Hello, on a page, i have : if (!empty(\$_FILES['logo']['name'])) { \$dossier = 'upload/'; \$fichier = basename(\$_FILES['logo']['name']); \$taille_maxi = 100000; \$taille = filesize(\$_FILES['logo']['tmp_name']); \$extensions = array('.png', '.jpg', '.jpeg'); \$extension = strrchr(\$_FILES['logo']['name'], '.'); if(!in_array(\$extension, \$extensions)) { \$erreur = 'ERROR you must upload the right type'; } if(\$taille>\$taille_maxi) { \$erreur = 'too heavy'; } if(!empty(\$erreur)) { ....................... } The problem is, if the users wants to edit information WITHOUT uploading a LOGO, it raises an error : 'error you must upload the right type' So, if a user didn't put anything in the inputbox in order to upload it, i don't want to enter in these conditions test. i tested : if (!empty(\$_FILES['logo']['name']) and if (isset(\$_FILES['logo']['name']) but both doesn't seems to work. Any ideas? edit : maybe i wasn't so clear, i don't want to test if he uploaded a logo, i want to test IF he selected a file to upload, because right now, if he doesn't select a file to upload, php raises an error telling he must upload with the right format. thanks.

• ### how to implement a really efficient bitvector sorting in python

##### - by xiao
Hello guys! Actually this is an interesting topic from programming pearls, sorting 10 digits telephone numbers in a limited memory with an efficient algorithm. You can find the whole story here What I am interested in is just how fast the implementation could be in python. I have done a naive implementation with the module bitvector. The code is as following: from BitVector import BitVector import timeit import random import time import sys def sort(input_li): return sorted(input_li) def vec_sort(input_li): bv = BitVector( size = len(input_li) ) for i in input_li: bv[i] = 1 res_li = [] for i in range(len(bv)): if bv[i]: res_li.append(i) return res_li if __name__ == "__main__": test_data = range(int(sys.argv[1])) print 'test_data size is:', sys.argv[1] random.shuffle(test_data) start = time.time() sort(test_data) elapsed = (time.time() - start) print "sort function takes " + str(elapsed) start = time.time() vec_sort(test_data) elapsed = (time.time() - start) print "sort function takes " + str(elapsed) start = time.time() vec_sort(test_data) elapsed = (time.time() - start) print "vec_sort function takes " + str(elapsed) I have tested from array size 100 to 10,000,000 in my macbook(2GHz Intel Core 2 Duo 2GB SDRAM), the result is as following: test_data size is: 1000 sort function takes 0.000274896621704 vec_sort function takes 0.00383687019348 test_data size is: 10000 sort function takes 0.00380706787109 vec_sort function takes 0.0371489524841 test_data size is: 100000 sort function takes 0.0520560741425 vec_sort function takes 0.374383926392 test_data size is: 1000000 sort function takes 0.867373943329 vec_sort function takes 3.80475401878 test_data size is: 10000000 sort function takes 12.9204008579 vec_sort function takes 38.8053860664 What disappoints me is that even when the test_data size is 100,000,000, the sort function is still faster than vec_sort. Is there any way to accelerate the vec_sort function?

• ### Counting total sum of each value in one column w.r.t another in Perl

##### - by sfactor
I have tab delimited data with multiple columns. I have OS names in column 31 and data bytes in columns 6 and 7. What I want to do is count the total volume of each unique OS. So, I did something in Perl like this: #!/usr/bin/perl use warnings; my @hhfilelist = glob "*.txt"; my %count = (); for my \$f (@hhfilelist) { open F, \$f || die "Cannot open \$f: \$!"; while (<F>) { chomp; my @line = split /\t/; # counting volumes in col 6 and 7 for 31 \$count{\$line[30]} = \$line[5] + \$line[6]; } close (F); } my \$w = 0; foreach \$w (sort keys %count) { print "\$w\t\$count{\$w}\n"; } So, the result would be something like Windows 100000 Linux 5000 Mac OSX 15000 Android 2000 But there seems to be some error in this code because the resulting values I get aren't as expected. What am I doing wrong?