Search Results

Search found 6017 results on 241 pages for 'universal records managem'.

Page 186/241 | < Previous Page | 182 183 184 185 186 187 188 189 190 191 192 193  | Next Page >

  • Do something when phone reaches a set of location?

    - by Pentium10
    I have a list of coordinates in the database identified as POI. For a city could be 100 records. I would like to get notified when the phone gets in 150 meters range of one of the location. The location coordinates too has an error/radius, usually 10 to 100meters. Since I don't find it good to add each location(could be hundreds) for a trigger, how can I optimize the wake-up code? Also do I have options to remove a previously setup notification from the queue?

    Read the article

  • MySQL: Select remaining rows

    - by Bjork24
    I've searched everywhere for this, but I can't seem to find a solution. Perhaps I'm using the wrong terms. Either way, I'm turning to good ol' trusty S.O. to help my find the answer. I have two tables, we'll call them 'tools' and 'installs' tools = id, name, version installs = id, tool_id, user_id The 'tools' table records available tools, which are then installed by a user and recorded in the 'installs' table. Selecting the installed tools are simple enough: SELECT tools.name FROM tools LEFT JOIN installs ON tools.id = installs.tool_id WHERE user_id = 99 ; How do I select the remaining tools -- the ones that have yet to be installed by user #99? I'm sorry if this is painfully obvious, but I just can't seem to figure it out! Thanks for the help!

    Read the article

  • Problem with the row count transform

    - by abkl
    Hi, I currently deployed an SSIS package (Developed on the 2005 version) (developed on my local server) in a pre production environment for testing. I have used the Row count transform to get a count of good/bad records. It works fine on my local system . However when i deploy this on the pre prod server, the row count does not work! (as in it does not recognize the vairbales i have assigned to the relevant transofm - no drop down abvaliable in the variables attribute part. tried deleting and adding a new transoform.. no luck. Strangely this does not work for any of the other packages also present/deployed on the same server (tried this out by dropping an rc tramsform onto an existing package... same problem) Any suggestions? Thanks a tonne

    Read the article

  • Hi, i have a c programming doubt? I want to know the difference between the two and where one is use

    - by aks
    Hi, i have a c programming doubt? I want to know the difference between the two and where one is useful over other? suppose i have a struct called employee as below: struct emp{ char first_name[10]; char last_name[10]; char key[10]; }; Now, i want to store the table of employee records, then which method should i use: struct emp e1[100]; // Or struct emp * e1[100]; I know the two are not same but would like to know a use case where second declaration would be of interest and more advantageous to use? Can someone clarify?

    Read the article

  • CakePHP ACL use case(s)

    - by Jonathan
    I have got a simple web app in development, i want to establish a couple of user groups; Admin, Doctors & Patients. Each group would have their access restricted to particular controller actions rather than individual content. So for example, Doctors can view patient records (index & view actions), but cannot delete them. Usually i would create a groups model, and assign the various users to a group. And filter in the beforeFilter() method to determine if the user has access. But if ACL can do the job, why right the code, right? Thanks

    Read the article

  • How to copy tables from one website to another with php?

    - by Lost_in_code
    I have 2 websites, lets say - example.com and example1.com example.com has a database fruits which has a table apple with 7000 records. I exported apple and tried to import it to example1.com but I'm always getting "MYSQL Server has gone away" error. I suspect this is due to some server side restriction. So, how can I copy the tables without having to contact the system admins? Is there a way to do this using PHP? I went through example of copying tables, but that was inside the same database. Both example.com and example1.com are on the same server.

    Read the article

  • Codeigniter: how do I select count when `$query->num_rows()` doesn't work for me?

    - by mOrloff
    I have a query which is returning a sum, so naturally it returns one row. I need to count the number of records in the DB which made that sum. Here's a sample of the type of query I am talking about (MySQL): SELECT i.id, i.vendor_quote_id, i.product_id_requested, SUM(i.quantity_on_hand) AS qty, COUNT(i.quantity_on_hand) AS count FROM vendor_quote_item AS i JOIN vendor_quote_container AS c ON i.vendor_quote_id = c.id LEFT JOIN company_types ON company_types.company_id = c.company_id WHERE company_types.company_type = 'f' AND i.product_id_requested = 12345678 I have found and am now using the select_min(), select_max(), and select_sum() functions, but my COUNT() is still hard-coded in. The main problem is that I am having to specify the table name in a tightly coupled manner with something like $this->$db->select( 'COUNT(myDbPrefix_vendor_quote_item.quantity_on_hand) AS count' ) which kills portability and makes switching environments a PIA. How can/should I get my the count values I am after with CI in an uncoupled way??

    Read the article

  • iOS static Framework crash when animating view

    - by user1439216
    I'm encountering a difficult to debug issue with a static library project when attempting to animate a view. It works fine when debugging (and even when debugging in the release configuration), but throws an error archived as a release: Exception Type: EXC_CRASH (SIGSYS) Exception Codes: 0x00000000, 0x00000000 Crashed Thread: 0 Thread 0 name: Dispatch queue: com.apple.main-thread Thread 0 Crashed: 0 TestApp 0x000d04fc 0x91000 + 259324 1 UIKit 0x336d777e +[UIView(UIViewAnimationWithBlocks) animateWithDuration:animations:] + 42 2 TestApp 0x000d04de 0x91000 + 259294 3 TestApp 0x000d0678 0x91000 + 259704 4 Foundation 0x355f04f8 __57-[NSNotificationCenter addObserver:selector:name:object:]_block_invoke_0 + 12 5 CoreFoundation 0x35aae540 ___CFXNotificationPost_block_invoke_0 + 64 6 CoreFoundation 0x35a3a090 _CFXNotificationPost + 1400 7 Foundation 0x355643e4 -[NSNotificationCenter postNotificationName:object:userInfo:] + 60 8 UIKit 0x33599112 -[UIInputViewTransition postNotificationsForTransitionStart] + 846 9 UIKit 0x335988cc -[UIPeripheralHost(UIKitInternal) executeTransition:] + 880 10 UIKit 0x3351bb8c -[UIPeripheralHost(UIKitInternal) setInputViews:animationStyle:] + 304 11 UIKit 0x3351b260 -[UIPeripheralHost(UIKitInternal) _reloadInputViewsForResponder:] + 952 12 UIKit 0x3351ae54 -[UIResponder(UIResponderInputViewAdditions) reloadInputViews] + 160 13 UIKit 0x3351a990 -[UIResponder becomeFirstResponder] + 452 14 UIKit 0x336194a0 -[UITextInteractionAssistant setFirstResponderIfNecessary] + 168 15 UIKit 0x33618d6a -[UITextInteractionAssistant oneFingerTap:] + 1602 16 UIKit 0x33618630 _UIGestureRecognizerSendActions + 100 17 UIKit 0x335a8d5e -[UIGestureRecognizer _updateGestureWithEvent:] + 298 18 UIKit 0x337d9472 ___UIGestureRecognizerUpdate_block_invoke_0541 + 42 19 UIKit 0x33524f4e _UIGestureRecognizerApplyBlocksToArray + 170 20 UIKit 0x33523a9c _UIGestureRecognizerUpdate + 892 21 UIKit 0x335307e2 _UIGestureRecognizerUpdateGesturesFromSendEvent + 22 22 UIKit 0x33530620 -[UIWindow _sendGesturesForEvent:] + 768 23 UIKit 0x335301ee -[UIWindow sendEvent:] + 82 24 UIKit 0x3351668e -[UIApplication sendEvent:] + 350 25 UIKit 0x33515f34 _UIApplicationHandleEvent + 5820 26 GraphicsServices 0x376d5224 PurpleEventCallback + 876 27 CoreFoundation 0x35ab651c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 32 28 CoreFoundation 0x35ab64be __CFRunLoopDoSource1 + 134 29 CoreFoundation 0x35ab530c __CFRunLoopRun + 1364 30 CoreFoundation 0x35a3849e CFRunLoopRunSpecific + 294 31 CoreFoundation 0x35a38366 CFRunLoopRunInMode + 98 32 GraphicsServices 0x376d4432 GSEventRunModal + 130 33 UIKit 0x33544cce UIApplicationMain + 1074 Thread 0 crashed with ARM Thread State: r0: 0x0000004e r1: 0x000d04f8 r2: 0x338fed47 r3: 0x3f523340 r4: 0x00000000 r5: 0x2fe8da00 r6: 0x00000001 r7: 0x2fe8d9d0 r8: 0x3f54cad0 r9: 0x00000000 r10: 0x3fd00000 r11: 0x3f523310 ip: 0x3f497048 sp: 0x2fe8d988 lr: 0x33539a41 pc: 0x000d04fc cpsr: 0x60000010 To give some background info: The static library is part of an 'iOS fake-framework', built using the templates from here: https://github.com/kstenerud/iOS-Universal-Framework The framework presents a registration UI as a modal view on top of whatever the client application is doing at the time. It pushes these views using a handle to a UIViewController provided by the client application. It doesn't do anything special, but here's the animation code: -(void)keyboardWillShowNotification:(NSNotification *)notification { double animationDuration = [[[notification userInfo] objectForKey:UIKeyboardAnimationDurationUserInfoKey] doubleValue]; dispatch_async(dispatch_get_main_queue(), ^(void) { [self animateViewsToState:kUMAnimationStateKeyboardVisible forIdiom:[UIDevice currentDevice].userInterfaceIdiom forDuration:animationDuration]; }); } -(void)animateViewsToState:(kUMAnimationState)state forIdiom:(UIUserInterfaceIdiom)idiom forDuration:(double)duration { float fieldOffset; if (idiom == UIUserInterfaceIdiomPhone) { if (state == kUMAnimationStateKeyboardVisible) { fieldOffset = -KEYBOARD_HEIGHT_IPHONE_PORTRAIT; } else { fieldOffset = KEYBOARD_HEIGHT_IPHONE_PORTRAIT; } } else { if (state == kUMAnimationStateKeyboardVisible) { fieldOffset = -IPAD_FIELD_OFFSET; } else { fieldOffset = IPAD_FIELD_OFFSET; } } [UIView animateWithDuration:duration animations:^(void) { mUserNameField.frame = CGRectOffset(mUserNameField.frame, 0, fieldOffset); mUserPasswordField.frame = CGRectOffset(mUserPasswordField.frame, 0, fieldOffset); }]; } Further printf-style debugging shows that it crashes whenever I do anything much with UIKit - specifically, it crashes when I replace -animateViewsToState with: if (0 == UIUserInterfaceIdiomPhone) { NSLog(@""); } and [[[[UIAlertView alloc] initWithTitle:@"test" message:@"123" delegate:nil cancelButtonTitle:@"OK" otherButtonTitles:nil] autorelease] show]; To me, this sounds like a linker problem, but I don't understand how such problems would only manifest here, and not beforehand. Any help would be greatly appreciated.

    Read the article

  • XCode linking error when targeting armv7.

    - by Tom
    I've already spent countless hours puzzling over this, utilizing Google searches and other Stack Overflow questions to no avail. I have an iPhone/iPad universal application, which seems to compile fine when the target is armv6. However, when the device is iPad, I get this warning: warning: building for SDK 'Device - iPhone OS 3.2' requires an armv7 architecture. Oddly enough, the app still runs great on iPad in spite of this warning. However, I do want to do things the "right way" what ever that means in this case. When I switch the target architecture to armv7, I get linking errors: "___restore_vfp_d8_d15_regs", referenced from: *redacted* "___save_vfp_d8_d15_regs", referenced from: *redacted* ld: symbol(s) not found collect2: ld returned 1 exit status The "redacted" portions of the errors are references to the static library to which I'm trying to link. Here's what I've tried from the many suggestions online. Each of these were suggested more than once without any explanation, which leads me to believe nobody quite understands this problem: "Never use the drop down menu in the upper left of the XCode window to choose the target. Instead, set this to Base SDK and then the Base SDK to iPhone OS 3.0 in the target configuration. Set the target device to your preferred target (iPad, iPhone OS 3.2 in my situation.)" This yields the error "Library not found for -lcrt1.3.1.o" "Make sure that GCC isn't linking against the wrong version of the standard library. (You'll have to make sure the LIBRARY_SEARCH_PATH doesn't have the wrong path in it.)" My LIBRARY_SEARCH_PATH is already empty, so this doesn't seem relevant. "Try compiling with GCC 4.0 rather than GCC 4.2." I get a syntax error inside a UIKit header file. The error is "Syntax error before 'AT_NAME' token." The line is "UIKIT_EXTERN @interface UILocalizedIndexedCollation : NSObject." Another project compiles just fine with the same target settings, which is really making me question my sanity. Could I be dealing with a corrupt XCode project? If anyone knows what's actually happening and has a reference or doesn't mind explaining it, I would be so very grateful. Cheers!

    Read the article

  • Gateway Page Between ASP and an ASP.NET Page

    - by ajdams
    I'll admit, I am pretty new with ASP .NET programming and I have been asked to take all our gateway pages (written in classic ASP) and make one universal gateway page to the few C# .NET applications we have (that I wrote). I tried searching here and the web and couldn't find much of anything describing a great way to do this and figured I was either not searching properly or was not properly naming what I am trying to do. I decided to to take one of the main gateway pages we had in classic ASP and use that as a base for my new gateway. Without boring you with a bunch of code I will summarize my gateway in steps and then can take advice/critique from there. EDIT: Basically what I am trying to do is go from a classic ASP page to a ASP .NET page and then back again. EDIT2: If my question is still unclear I am asking if what I have an the right start and if anyone has suggestions as to how this could be better. It can be as generic as need-be, not looking for a specific off-the-shelf code answer. My Gateway page: In the first part of the page I grab session variables and determine if the user is leaving or returning through the gateway: Code (in VB): uid = Request.QueryString("GUID") If uid = "" Then direction = "Leaving" End If ' Gather available user information. userid = Session("lnglrnregid") bankid = Session("strBankid") ' Return location. floor = Request.QueryString("Floor") ' The option chosen will determine if the user continues as SSL or not. ' If they are currently SSL, they should remain if chosen. option1 = Application(bankid & "Option1") If MID(option1, 6, 1) = "1" Then sslHttps = "s" End If Next I enter the uid into a database table (SQL-Server 2005) as a uniqueidentifier field called GUID. I omitted the stored procedure call. Lastly, I use the direction variable to determine if the user is leaving or returning and do redirects from there to the different areas of the site. Code (In VB again): If direction = "Leaving" Then Select Case floor Case "sscat", "ssassign" ' A SkillSoft course Response.Redirect("Some site here") Case "lrcat", "lrassign" ' A LawRoom course Response.Redirect("Some site here") Case Else ' Some other SCORM course like MindLeaders or a custom upload. Response.Redirect("Some site here") End Select Session.Abandon Else ' The only other direction is "Returning" ..... That's about it so far - so like I said, not an expert so any suggestions would be greatly appreciated!

    Read the article

  • What's the correct way to use Cakephp urls?

    - by Pichan
    Hello all, it's my first post here :) I'm having some difficulties with dealing with urls and parameters. I've gone through the router class api documentation over and over again and found nothing useful. First of all, I'd like to know if there is any 'universal' format in CakePHP(1.3) for handling urls. I'm currently handling all my urls as simple arrays(in the format that Router::url and $html-link accepts) and it's easy as long as I only need to pass them as arguments to cake's own methods. It usually gets tricky if I need something else. Mainly I'm having problems with converting string urls to the basic array format. Let's say I want to convert $arrayUrl to string and than again into url: $arrayUrl=array('controller'=>'SomeController','action'=>'someAction','someValue'); $url=Router::url($arrayUrl); //$url is now '/path/to/site/someController/someAction/someValue' $url=Router::normalize($url); //remove '/path/to/site' $parsed=Router::parse($url); /*$parsed is now Array( [controller] => someController [action] => someAction [named] => Array() [pass] => Array([0] => someValue) [plugin] => ) */ That seems an awful lot of code to do something as simple as to convert between 2 core formats. Also, note that $parsed is still not in the same as $arrayUrl. Of course I could tweak $parsed manually and actually I've done that a few times as a quick patch but I'd like to get to the bottom of this. I also noticed that when using prefix routing, $this-params in controller has the prefix embedded in the action(i.e. [action] = 'admin_edit') and the result of Router::parse() does not. Both of course have the prefix in it's own key. To summarize, how do I convert an url between any of these 3(or 4, if you include the prefix thing) mentioned formats the right way? Of course it would be easy to hack my way through this, but I'd still like to believe that cake is being developed by a bunch of people who have a lot more experience and insight than me, so I'm guessing there's a good reason for this "perceived misbehavior". I've tried to present my problem as good as I can, but due to my rusty english skills, I had to take a few detours :) I'll explain more if needed.

    Read the article

  • what do i need to do so that mod_wsgi will find libmysqlclient.16.dylib? (osx 10.7 with apache mod_wsgi)

    - by compound eye
    I am trying to run django on osx 10.7 (lion) with apache mod_wsgi and virtualenv. My site works if I use the django testing server: (baseline)otter:hello mathew$ python manage.py runserver but it doesn't work when I run apache. The core of the error seems to be Library not loaded: libmysqlclient.16.dylib I think its to do with the path apache is using to locate libmysqlclient.16.dylib when I run otool in the lib directory it looks good otter:lib mathew$ pwd /usr/local/mysql/lib otter:lib mathew$ otool -L libmysqlclient.16.dylib libmysqlclient.16.dylib: libmysqlclient.16.dylib (compatibility version 16.0.0, current version 16.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1) but from outside it can't find it otter:lib mathew$ cd / otter:/ mathew$ otool -L libmysqlclient.16.dylib otool: can't open file: libmysqlclient.16.dylib (No such file or directory) if i manually set DYLD_LIBRARY_PATH otool works otter:lib mathew$ DYLD_LIBRARY_PATH=/usr/local/mysql/lib otter:lib mathew$ otool -L libmysqlclient.16.dylib libmysqlclient.16.dylib: libmysqlclient.16.dylib (compatibility version 16.0.0, current version 16.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1) When I run the django testing server, my .bash_profile sets up the virtualenv and the path to the mysql dynamic library export DYLD_LIBRARY_PATH=/usr/local/mysql/lib/:$DYLD_LIBRARY_PATH export PATH When i run apache it finds my virtualenv paths, but it doesn't seem to find the dynamic library path. I tried adding this path to /usr/sbin/envvars DYLD_LIBRARY_PATH="/usr/lib:/usr/local/mysql/lib:$DYLD_LIBRARY_PATH" export DYLD_LIBRARY_PATH and to /private/etc/paths.d/libmysql /usr/local/mysql/lib then restarted the machine but that has not changed the error message. Error loading MySQLdb module: dlopen(/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.16.dylib I don't think is a permissions issue: -rwxr-xr-x 1 root wheel 3787328 4 Dec 2010 libmysqlclient.16.dylib drwxr-xr-x 39 root wheel 1394 18 Nov 21:07 / drwxr-xr-x@ 15 root wheel 510 24 Oct 22:10 /usr drwxrwxr-x 20 root admin 680 2 Nov 20:22 /usr/local drwxr-xr-x 20 mathew admin 680 9 Nov 21:58 /usr/local/python_virtualenv drwxr-xr-x 6 mathew admin 204 2 Nov 21:36 /usr/local/python_virtualenv/baseline drwxr-xr-x 4 mathew admin 136 2 Nov 21:26 /usr/local/python_virtualenv/baseline/lib drwxr-xr-x 52 mathew admin 1768 2 Nov 21:26 /usr/local/python_virtualenv/baseline/lib/python2.7 drwxr-xr-x 18 mathew admin 612 4 Nov 21:20 /usr/local/python_virtualenv/baseline/lib/python2.7/site-packages -rwxr-xr-x 1 mathew admin 66076 2 Nov 21:18 /usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/_mysql.so What do i need to do so that mod_wsgi will find libmysqlclient.16.dylib? apache and mysql are both 64 bit: otter:lib mathew$ file /usr/sbin/httpd /usr/sbin/httpd: Mach-O universal binary with 2 architectures /usr/sbin/httpd (for architecture x86_64): Mach-O 64-bit executable x86_64 /usr/sbin/httpd (for architecture i386): Mach-O executable i386 otter:lib mathew$ otter:lib mathew$ file /usr/local/mysql/lib/libmysqlclient.16.dylib /usr/local/mysql/lib/libmysqlclient.16.dylib: Mach-O 64-bit dynamically linked shared library x86_64

    Read the article

  • PNGs alpha transparancy in AS3 - Unknown file-type

    - by WiseDonkey
    Hello there! After whittling down of the options we've encountered a problem with PNG's and ActionScript 3 (AS3). When loading a PNG 8 or PNG 32 with alpha transparancy we're getting the following error reported in Flash:- "Error #2124: Loaded file is an unknown type" Now, we're dealing with some legacy images, and it appears as though this problem isn't universal - some images believed to be 32bit alpha PNG are loading. BUT, some conclusions:- converting one image that was 32 bit alpha (NOT WORKING IN AS3) to PNG 8 index transparency DID work. And converting that same image to PNG 8 alpha DID NOT work. These all worked in AS2 There is no difference between the headers Headers of a Failing Image [0] => HTTP/1.1 200 OK [1] => Date: Tue, 06 Apr 2010 14:17:28 GMT [2] => Server: Apache/2.2.3 (Red Hat) [3] => Last-Modified: Tue, 06 Apr 2010 13:44:05 GMT [4] => ETag: "3700054-11d6-a3983340" [5] => Accept-Ranges: bytes [6] => Content-Length: 4566 [7] => Connection: close [8] => Content-Type: image/png Headers of a Working Image [0] => HTTP/1.1 200 OK [1] => Date: Tue, 06 Apr 2010 14:19:02 GMT [2] => Server: Apache/2.2.3 (Red Hat) [3] => Last-Modified: Fri, 30 Oct 2009 18:38:08 GMT [4] => ETag: "ba8057-65f2-5445c400" [5] => Accept-Ranges: bytes [6] => Content-Length: 26098 [7] => Connection: close [8] => Content-Type: image/png Any thoughts of a direction of further investigation or thoughts on a bewildering problem with little to no documentation; very warmly welcomed. EDIT Now it would appear as though something in the PHP conversion of the images is shafting; I use the following PHP to add alpha layers:- imagealphablending($image_p, false); ImageSaveAlpha($image_p, true); ImageFill($image_p, 0, 0, IMG_COLOR_TRANSPARENT);

    Read the article

  • UINavigationBar unresponsive after canceling a UITableView search in nav controller in tab bar in a popover

    - by Mark
    Ok, this is an odd one and I can reproduce it with a new project easily. Here is the setup: I have a UISplitViewController. In the left side I have a UITabBarController. In this tab bar controller I have two UINavigationControllers. In the navigation controllers I have UITableViewControllers. These table views have search bars on them. Ok, what happens with this setup is that if I'm in portrait mode and bring up this view in the popover and I start a search in one of the table views and cancel it, the navigation bar becomes unresponsive. That is, the "back" button as well as the right side button cannot be clicked. If I do the exact same thing in landscape mode so we are not in a popover, this doesn't happen. The navigation bar stays responsive. So, the problem only seems to happen inside a popover. I've also noticed that if I do the search but click on an item in the search results which ends up loading something into the "detail view" of the split view and dismissing the popover, and then come back to the popover and then click the Cancel button for the search, the navigation bar is responsive. My application is a universal app and uses the same tab bar controller in the iPhone interface and it works there without this issue. As I mentioned above, I can easily reproduce this with a new project. Here are the steps if you want to try it out yourself: start new project - split view create new UITableViewController class (i named TableViewController) uncomment out the viewDidLoad method as well as the rightBarButtonItem line in viewDidLoad (so we will have an Edit button in the navigation bar) enter any values you want to return from numberOfSectioinsInTableView and numberOfRowsInSection methods open MainWindow.xib and do the following: please note that you will need to be viewing the xib in the middle "view mode" so you can expand the contents of the items drag a Tab Bar Controller into the xib to replace the Navigation Controller item drag a Navigation Controller into the xib as another item under the Tab Bar Controller delete the other two view controllers that are under the Tab Bar Controller (so, now our tab bar has just the one navigation controller on it) inside the navigation controller, drag in a Table View Controller and use it to replace the View Controller (Root View Controller) change the class of the new Table View Controller to the class created above (TableViewController for me) double-click on the Table View under the new Table View Controller to open it up (will be displayed in the tab bar inside the split view controller) drag a "Search Bar and Search Display" onto the table view save the xib run the project in simulator while in portrait mode, click on the Root List button to bring up popover notice the Edit button is clickable click in the Search box - we go into search mode click the Cancel button to exit search mode notice the Edit button no longer works So, can anyone help me figure out why this is happening? Thanks, Mark

    Read the article

  • Is it possible to reliably auto-decode user files to Unicode? [C#]

    - by NVRAM
    I have a web application that allows users to upload their content for processing. The processing engine expects UTF8 (and I'm composing XML from multiple users' files), so I need to ensure that I can properly decode the uploaded files. Since I'd be surprised if any of my users knew their files even were encoded, I have very little hope they'd be able to correctly specify the encoding (decoder) to use. And so, my application is left with task of detecting before decoding. This seems like such a universal problem, I'm surprised not to find either a framework capability or general recipe for the solution. Can it be I'm not searching with meaningful search terms? I've implemented BOM-aware detection (http://en.wikipedia.org/wiki/Byte_order_mark) but I'm not sure how often files will be uploaded w/o a BOM to indicate encoding, and this isn't useful for most non-UTF files. My questions boil down to: Is BOM-aware detection sufficient for the vast majority of files? In the case where BOM-detection fails, is it possible to try different decoders and determine if they are "valid"? (My attempts indicate the answer is "no.") Under what circumstances will a "valid" file fail with the C# encoder/decoder framework? Is there a repository anywhere that has a multitude of files with various encodings to use for testing? While I'm specifically asking about C#/.NET, I'd like to know the answer for Java, Python and other languages for the next time I have to do this. So far I've found: A "valid" UTF-16 file with Ctrl-S characters has caused encoding to UTF-8 to throw an exception (Illegal character?) (That was an XML encoding exception.) Decoding a valid UTF-16 file with UTF-8 succeeds but gives text with null characters. Huh? Currently, I only expect UTF-8, UTF-16 and probably ISO-8859-1 files, but I want the solution to be extensible if possible. My existing set of input files isn't nearly broad enough to uncover all the problems that will occur with live files. Although the files I'm trying to decode are "text" I think they are often created w/methods that leave garbage characters in the files. Hence "valid" files may not be "pure". Oh joy. Thanks.

    Read the article

  • Loading jar file using JCL(JarClassLoader ) : classpath in manifest is ignored ..

    - by Xinus
    I am trying to load jar file using JCL using following code FileInputStream fis = new FileInputStream(new File( "C:\\Users\\sunils\\glassfish-tests\\working\\test.jar") ); JarClassLoader jc = new JarClassLoader( ); jc.add(fis); Class main = jc.loadClass( "highmark.test.Main" ); String[] str={}; main.getMethod("test").invoke(null);//.getDeclaredMethod("main",String[].class).invoke(null,str); fis.close(); But when I try to run this program I get Exception as Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at Main.main(Main.java:21) Caused by: java.lang.RuntimeException: Embedded startup not found, classpath is probably incomplete at org.glassfish.api.embedded.Server.<init>(Server.java:292) at org.glassfish.api.embedded.Server.<init>(Server.java:75) at org.glassfish.api.embedded.Server$Builder.build(Server.java:185) at org.glassfish.api.embedded.Server$Builder.build(Server.java:167) at highmark.test.Main.test(Main.java:33) ... 5 more According to this it is not able to locate class, But when I run the jar file explicitly it runs fine. It seems like JCL is ignoring other classes present in the jar file, MANIFEST.MF file in jar file shows: Manifest-Version: 1.0 Class-Path: . Main-Class: highmark.test.Main It seems to be ignoring Class-Path: . , This jar file runs fine when I run it using Java explicitly, This is just a test, in reality this jar file is coming as a InputStream and it cannot be stored in filesystem, How can I overcome this problem , Is there any workaround ? Thanks for any help . UNDATE: Here is a jar Main class : package highmark.test; import org.glassfish.api.embedded.*; import java.io.*; import org.glassfish.api.deployment.*; import com.sun.enterprise.universal.io.FileUtils; public class Main { public static void main(String[] args) throws IOException, LifecycleException, ClassNotFoundException { test(); } public static void test() throws IOException, LifecycleException, ClassNotFoundException{ Server.Builder builder = new Server.Builder("test"); Server server = builder.build(); server.createPort(8080); ContainerBuilder containerBuilder = server.createConfig(ContainerBuilder.Type.web); server.addContainer(containerBuilder); server.start(); File war=new File("C:\\Users\\sunils\\maventests\\simple-webapp\\target\\simple-webapp.war");//(File) inputStream.readObject(); EmbeddedDeployer deployer = server.getDeployer(); DeployCommandParameters params = new DeployCommandParameters(); params.contextroot = "simple"; deployer.deploy(war, params); } }

    Read the article

  • Learn C# now or finish up with Java and then learn C#?

    - by Sahat
    Ok here is my situation. I've studied Java in my college for 2 semesters. But you know they teach you jack in there, just the basics. We skipped half of our textbook and even then our professors don't teach from section to section of each chapter. I don't blame them. It's hard as it is for new students to understand even the basic concepts of programming. Now this is a community college we are talking about and not Stanford, MIT or Berkeley. So like I said I've done 2 semester of Java. I really like our textbook because it has some challenging projects to do at the end of each chapter. This textbook is pretty clear and i have no problem understanding it (although 2-D and 3-D Arrays have given me some trouble). I have tried reading a few C# books such as Pro C# 2008 and .NET 3.5 and C# 4.0 in a Nutshell. I found these books to be dry and overloaded with information that put me to sleep (No offense to the authors of those 2 wonderful, according to amazon ratings, books). Would you suggest I finish my Java textbook, brush up my knowledge of Arrays, Polymorphism, and etc that are universal to most programming languages. And then switch to C#, plus the syntax is very similar so it should be easy to switch. Or should I just start learning C# right now from the very beginning? If it's the latter then could you recommend some free online resources that will keep me engaged and at the same time teach me everything I need to know about C#. Someone has recommended me to learn .NET first, but I found it to be not the brightest idea. .NET is just a big monster full of libraries. How am I going to apply it if I don't even know the C# or VB!? Anyway back to my question: Master Java and switch to C# or just go with C#? DISCLAIMER: I don't want to start .NET vs J2EE or C# vs Java flame war. I am going with C#. I've decided that I want to work in a Microsoft shop in the future. .NET is what I want to learn. Thanks! Will be waiting for the answers.

    Read the article

  • iPhone app works hundreds of times, then crashes from memory error on startup, then never works unti

    - by peter
    I have a Cocos2d/openGL iPhone game. It's a universal app and I'm dealing with an occasional but nasty error on the iPad. We are loading a lot of textures up front (3 2048x2048 textures). I'm working on reducing this up front load, but what worries me is I really don't understand the root cause of this crash that permanently breaks the app. This is the deal: 1. App works fine for hundreds of plays on the iPad 2. Eventually (I'm guessing due to other programs using up some memory and not letting go or whatever) the app starts crashing on startup. It just closes again in the middle of loading. 3. The App will now never work again on that iPad, closing immediately every time, until the iPad is restarted. Obviously my app is demanding too much memory up front to work reliably every time, I get that. What I don't get is why when it fails once, it has failed forever until the iPad is restarted. Can anyone explain what is going on here? EDIT: forgot to add organizer crash lags just say low memory, like this every time (I changed my app name to MyAppName below). Again, I know it's low memory, but why does it stay low memory until restart?: Incident Identifier: E7A2507C-3FB1-4E3B-B315-09F094236541 CrashReporter Key: 0fda9d667f2c6073f20a76809aa25438b6854d15 OS Version: iPhone OS 3.2 (7B367) Date: 2010-04-30 16:59:44 -0400 Free pages: 437 Wired pages: 17228 Purgeable pages: 0 Largest process: MyAppName Processes Name UUID Count resident pages MyAppName <6307ce41802850944baa78d29224fa7f> 22385 (jettisoned) (active) mediaserverd <ea8bac28b06fe3980fdd44b5caceb563> 242 DTMobileIS <a0f651e43881e66f50f8a95abea72921> 5826 notification_pro <4c9a7ee0a5bbe160465991228f2d2f2e> 67 syslog_relay <4ceaed776d2df957fa130712f4ef21d0> 66 notification_pro <4c9a7ee0a5bbe160465991228f2d2f2e> 67 notification_pro <4c9a7ee0a5bbe160465991228f2d2f2e> 67 afcd <4f3c9566e33b4463f05603d990584e5d> 72 ptpd <83de0f774bd6553d513ae9e19b0f9b56> 181 syslogd <66247e305d5c0bf6f1ce1cc950653263> 81 lsd <a4d852c1c8da2b3d231bdc90887b52ba> 130 iapd <a8534cbde4b90ad5915dd26ab03ff3e3> 204 notifyd <5e9d5bee7c3eae1c8b494c79eb11406e> 71 BTServer <64e4a6ea6b1240db2331e05a29caa862> 108 CommCenter <97bf297944ac4bde19bcee96dd23bd5f> 181 SpringBoard <c7a5904c12db7b14334a4edaa4cabaa9> 5339 (active) configd <aca9fa3380322669164fd6b1a3864300> 373 fairplayd.K48 <2d997ffca1a568f9c5400ac32d8f0782> 84 locationd <dd1ea88105c62173908ce767db5c4d37> 599 mDNSResponder <820560222d47a1f2a0dce98a7f8a9721> 108 lockdownd <497fd54c79a680bf29f5d9320f514613> 303 MobileStorageMou <c277b79c2157c4dc5cfc5c3ca35bd5f2> 69 launchd <66972eee4d865c4383b33d985d22994b> 98 **End**

    Read the article

  • How to reliably categorize HTTP sessions in proxy to corresponding browser' windows/tabs user is viewing?

    - by Jehonathan
    I was using the Fiddler core .Net library as a local proxy to record the user activity in web. However I ended up with a problem which seems dirty to solve. I have a web browser say Google Chrome, and the user opened like 10 different tabs each with different web URLs. The problem is that the proxy records all the HTTP session initiated by each pages separately, causing me to figure out using my intelligence the tab which the corresponding HTTP session belonged to. I understand that this is because of the stateless nature of HTTP protocol. However I am just wondering is there an easy way to do this? I ended up with below c# code for that in Fiddler. Still its not a reliable solution due to the heuristics. This is a modification of the sample project bundled with Fiddler core for .NET 4. Basically what it does is filtering HTTP sessions initiated in last few seconds to find the first request or switching to another page made by the same tab in browser. It almost works, but not seems to be a universal solution. Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.Session oS) { //exclude other HTTP methods if (oS.oRequest.headers.HTTPMethod == "GET" || oS.oRequest.headers.HTTPMethod == "POST") //exclude other HTTP Status codes if (oS.oResponse.headers.HTTPResponseStatus == "200 OK" || oS.oResponse.headers.HTTPResponseStatus == "304 Not Modified") { //exclude other MIME responses (allow only text/html) var accept = oS.oRequest.headers.FindAll("Accept"); if (accept != null) { if(accept.Count>0) if (accept[0].Value.Contains("text/html")) { //exclude AJAX if (!oS.oRequest.headers.Exists("X-Requested-With")) { //find the referer for this request var referer = oS.oRequest.headers.FindAll("Referer"); //if no referer then assume this as a new request and display the same if(referer!=null) { //if no referer then assume this as a new request and display the same if (referer.Count > 0) { //lock the sessions Monitor.Enter(oAllSessions); //filter further using the response if (oS.oResponse.MIMEType == string.Empty || oS.oResponse.MIMEType == "text/html") //get all previous sessions with the same process ID this session request if(oAllSessions.FindAll(a=>a.LocalProcessID == oS.LocalProcessID) //get all previous sessions within last second (assuming the new tab opened initiated multiple sessions other than parent) .FindAll(z => (z.Timers.ClientBeginRequest > oS.Timers.ClientBeginRequest.AddSeconds(-1))) //get all previous sessions that belongs to the same port of the current session .FindAll(b=>b.port == oS.port ).FindAll(c=>c.clientIP ==oS.clientIP) //get all previus sessions with the same referrer URL of the current session .FindAll(y => referer[0].Value.Equals(y.fullUrl)) //get all previous sessions with the same host name of the current session .FindAll(m=>m.hostname==oS.hostname).Count==0 ) //if count ==0 that means this is the parent request Console.WriteLine(oS.fullUrl); //unlock sessions Monitor.Exit(oAllSessions); } else Console.WriteLine(oS.fullUrl); } else Console.WriteLine(oS.fullUrl); Console.WriteLine(); } } } } };

    Read the article

  • plane bombing problems- help

    - by peiska
    I'm training code problems, and on this one I am having problems to solve it, can you give me some tips how to solve it please. The problem is something like this: Your task is to find the sequence of points on the map that the bomber is expected to travel such that it hits all vital links. A link from A to B is vital when its absence isolates completely A from B. In other words, the only way to go from A to B (or vice versa) is via that link. Notice that if we destroy for example link (d,e), it becomes impossible to go from d to e,m,l or n in any way. A vital link can be hit at any point that lies in its segment (e.g. a hit close to d is as valid as a hit close to e). Of course, only one hit is enough to neutralize a vital link. Moreover, each bomb affects an exact circle of radius R, i.e., every segment that intersects that circle is considered hit. Due to enemy counter-attack, the plane may have to retreat at any moment, so the plane should follow, at each moment, to the closest vital link possible, even if in the end the total distance grows larger. Given all coordinates (the initial position of the plane and the nodes in the map) and the range R, you have to determine the sequence of positions in which the plane has to drop bombs. This sequence should start (takeoff) and finish (landing) at the initial position. Except for the start and finish, all the other positions have to fall exactly in a segment of the map (i.e. it should correspond to a point in a non-hit vital link segment). The coordinate system used will be UTM (Universal Transverse Mercator) northing and easting, which basically corresponds to a Euclidian perspective of the world (X=Easting; Y=Northing). Input Each input file will start with three floating point numbers indicating the X0 and Y0 coordinates of the airport and the range R. The second line contains an integer, N, indicating the number of nodes in the road network graph. Then, the next N (<10000) lines will each contain a pair of floating point numbers indicating the Xi and Yi coordinates (1 No two links will ever cross with each other. Output The program will print the sequence of coordinates (pairs of floating point numbers with exactly one decimal place), each one at a line, in the order that the plane should visit (starting and ending in the airport). Sample input 1 102.3 553.9 0.2 14 342.2 832.5 596.2 638.5 479.7 991.3 720.4 874.8 744.3 1284.1 1294.6 924.2 1467.5 659.6 1802.6 659.6 1686.2 860.7 1548.6 1111.2 1834.4 1054.8 564.4 1442.8 850.1 1460.5 1294.6 1485.1 17 1 2 1 3 2 4 3 4 4 5 4 6 6 7 7 8 8 9 8 10 9 10 10 11 6 11 5 12 5 13 12 13 13 14 Sample output 1 102.3 553.9 720.4 874.8 850.1 1460.5 102.3 553.9

    Read the article

  • SSIS code smell – Unused columns in the dataflow

    - by jamiet
    A code smell is defined on Wikipedia as being a “symptom in the source code of a program that possibly indicates a deeper problem”. It’s a term commonly used by our code-writing brethren to describe sub-optimal code but I think the term can be applied equally well to SSIS packages too as I shall now explain One of my pet hates about SSIS development is packages that throw warnings of the form: The output column "ColumnName" (1358) on output "OLE DB Source Output" (1289) and component "OLE_SRC Name" (1279) is not subsequently used in the Data Flow task. Removing this unused output column can increase Data Flow task performance.  The warning is fairly self-explanatory – any column that appears in the data flow but doesn’t get used will throw this warning when the data flow is executed. Its not the negligible performance degradation that they cause that bothers me though, it’s the clutter that they cause in your log file/table. Take a look at the following screenshot if you don’t believe me: There are 231409 such warnings in the system that I took this screenshot from, that is 231409 log records that should not be there. The most infuriating thing about this warning is that it is so easily avoidable; eliminating such columns is a very quick and easy thing to do in the SSIS Designer. The only problem I see is that the warnings don’t occur until you execute the package – it would be preferable for the designer to have an unobtrusive way of informing you of them as well. Anyway, I digress… I consider such warnings to be a code smell because, to me, they’re symptomatic of a lack of due care and attention; a lack of developer discipline if you will. What other code smells can you think of when building SSIS packages? If I get a good list in the comments maybe I’ll compile them into a later blog post. @Jamiet Share this post: email it! | bookmark it! | digg it! | reddit! | kick it! | live it!

    Read the article

  • Toorcon14

    - by danx
    Toorcon 2012 Information Security Conference San Diego, CA, http://www.toorcon.org/ Dan Anderson, October 2012 It's almost Halloween, and we all know what that means—yes, of course, it's time for another Toorcon Conference! Toorcon is an annual conference for people interested in computer security. This includes the whole range of hackers, computer hobbyists, professionals, security consultants, press, law enforcement, prosecutors, FBI, etc. We're at Toorcon 14—see earlier blogs for some of the previous Toorcon's I've attended (back to 2003). This year's "con" was held at the Westin on Broadway in downtown San Diego, California. The following are not necessarily my views—I'm just the messenger—although I could have misquoted or misparaphrased the speakers. Also, I only reviewed some of the talks, below, which I attended and interested me. MalAndroid—the Crux of Android Infections, Aditya K. Sood Programming Weird Machines with ELF Metadata, Rebecca "bx" Shapiro Privacy at the Handset: New FCC Rules?, Valkyrie Hacking Measured Boot and UEFI, Dan Griffin You Can't Buy Security: Building the Open Source InfoSec Program, Boris Sverdlik What Journalists Want: The Investigative Reporters' Perspective on Hacking, Dave Maas & Jason Leopold Accessibility and Security, Anna Shubina Stop Patching, for Stronger PCI Compliance, Adam Brand McAfee Secure & Trustmarks — a Hacker's Best Friend, Jay James & Shane MacDougall MalAndroid—the Crux of Android Infections Aditya K. Sood, IOActive, Michigan State PhD candidate Aditya talked about Android smartphone malware. There's a lot of old Android software out there—over 50% Gingerbread (2.3.x)—and most have unpatched vulnerabilities. Of 9 Android vulnerabilities, 8 have known exploits (such as the old Gingerbread Global Object Table exploit). Android protection includes sandboxing, security scanner, app permissions, and screened Android app market. The Android permission checker has fine-grain resource control, policy enforcement. Android static analysis also includes a static analysis app checker (bouncer), and a vulnerablity checker. What security problems does Android have? User-centric security, which depends on the user to grant permission and make smart decisions. But users don't care or think about malware (the're not aware, not paranoid). All they want is functionality, extensibility, mobility Android had no "proper" encryption before Android 3.0 No built-in protection against social engineering and web tricks Alternative Android app markets are unsafe. Simply visiting some markets can infect Android Aditya classified Android Malware types as: Type A—Apps. These interact with the Android app framework. For example, a fake Netflix app. Or Android Gold Dream (game), which uploads user files stealthy manner to a remote location. Type K—Kernel. Exploits underlying Linux libraries or kernel Type H—Hybrid. These use multiple layers (app framework, libraries, kernel). These are most commonly used by Android botnets, which are popular with Chinese botnet authors What are the threats from Android malware? These incude leak info (contacts), banking fraud, corporate network attacks, malware advertising, malware "Hackivism" (the promotion of social causes. For example, promiting specific leaders of the Tunisian or Iranian revolutions. Android malware is frequently "masquerated". That is, repackaged inside a legit app with malware. To avoid detection, the hidden malware is not unwrapped until runtime. The malware payload can be hidden in, for example, PNG files. Less common are Android bootkits—there's not many around. What they do is hijack the Android init framework—alteering system programs and daemons, then deletes itself. For example, the DKF Bootkit (China). Android App Problems: no code signing! all self-signed native code execution permission sandbox — all or none alternate market places no robust Android malware detection at network level delayed patch process Programming Weird Machines with ELF Metadata Rebecca "bx" Shapiro, Dartmouth College, NH https://github.com/bx/elf-bf-tools @bxsays on twitter Definitions. "ELF" is an executable file format used in linking and loading executables (on UNIX/Linux-class machines). "Weird machine" uses undocumented computation sources (I think of them as unintended virtual machines). Some examples of "weird machines" are those that: return to weird location, does SQL injection, corrupts the heap. Bx then talked about using ELF metadata as (an uintended) "weird machine". Some ELF background: A compiler takes source code and generates a ELF object file (hello.o). A static linker makes an ELF executable from the object file. A runtime linker and loader takes ELF executable and loads and relocates it in memory. The ELF file has symbols to relocate functions and variables. ELF has two relocation tables—one at link time and another one at loading time: .rela.dyn (link time) and .dynsym (dynamic table). GOT: Global Offset Table of addresses for dynamically-linked functions. PLT: Procedure Linkage Tables—works with GOT. The memory layout of a process (not the ELF file) is, in order: program (+ heap), dynamic libraries, libc, ld.so, stack (which includes the dynamic table loaded into memory) For ELF, the "weird machine" is found and exploited in the loader. ELF can be crafted for executing viruses, by tricking runtime into executing interpreted "code" in the ELF symbol table. One can inject parasitic "code" without modifying the actual ELF code portions. Think of the ELF symbol table as an "assembly language" interpreter. It has these elements: instructions: Add, move, jump if not 0 (jnz) Think of symbol table entries as "registers" symbol table value is "contents" immediate values are constants direct values are addresses (e.g., 0xdeadbeef) move instruction: is a relocation table entry add instruction: relocation table "addend" entry jnz instruction: takes multiple relocation table entries The ELF weird machine exploits the loader by relocating relocation table entries. The loader will go on forever until told to stop. It stores state on stack at "end" and uses IFUNC table entries (containing function pointer address). The ELF weird machine, called "Brainfu*k" (BF) has: 8 instructions: pointer inc, dec, inc indirect, dec indirect, jump forward, jump backward, print. Three registers - 3 registers Bx showed example BF source code that implemented a Turing machine printing "hello, world". More interesting was the next demo, where bx modified ping. Ping runs suid as root, but quickly drops privilege. BF modified the loader to disable the library function call dropping privilege, so it remained as root. Then BF modified the ping -t argument to execute the -t filename as root. It's best to show what this modified ping does with an example: $ whoami bx $ ping localhost -t backdoor.sh # executes backdoor $ whoami root $ The modified code increased from 285948 bytes to 290209 bytes. A BF tool compiles "executable" by modifying the symbol table in an existing ELF executable. The tool modifies .dynsym and .rela.dyn table, but not code or data. Privacy at the Handset: New FCC Rules? "Valkyrie" (Christie Dudley, Santa Clara Law JD candidate) Valkyrie talked about mobile handset privacy. Some background: Senator Franken (also a comedian) became alarmed about CarrierIQ, where the carriers track their customers. Franken asked the FCC to find out what obligations carriers think they have to protect privacy. The carriers' response was that they are doing just fine with self-regulation—no worries! Carriers need to collect data, such as missed calls, to maintain network quality. But carriers also sell data for marketing. Verizon sells customer data and enables this with a narrow privacy policy (only 1 month to opt out, with difficulties). The data sold is not individually identifiable and is aggregated. But Verizon recommends, as an aggregation workaround to "recollate" data to other databases to identify customers indirectly. The FCC has regulated telephone privacy since 1934 and mobile network privacy since 2007. Also, the carriers say mobile phone privacy is a FTC responsibility (not FCC). FTC is trying to improve mobile app privacy, but FTC has no authority over carrier / customer relationships. As a side note, Apple iPhones are unique as carriers have extra control over iPhones they don't have with other smartphones. As a result iPhones may be more regulated. Who are the consumer advocates? Everyone knows EFF, but EPIC (Electrnic Privacy Info Center), although more obsecure, is more relevant. What to do? Carriers must be accountable. Opt-in and opt-out at any time. Carriers need incentive to grant users control for those who want it, by holding them liable and responsible for breeches on their clock. Location information should be added current CPNI privacy protection, and require "Pen/trap" judicial order to obtain (and would still be a lower standard than 4th Amendment). Politics are on a pro-privacy swing now, with many senators and the Whitehouse. There will probably be new regulation soon, and enforcement will be a problem, but consumers will still have some benefit. Hacking Measured Boot and UEFI Dan Griffin, JWSecure, Inc., Seattle, @JWSdan Dan talked about hacking measured UEFI boot. First some terms: UEFI is a boot technology that is replacing BIOS (has whitelisting and blacklisting). UEFI protects devices against rootkits. TPM - hardware security device to store hashs and hardware-protected keys "secure boot" can control at firmware level what boot images can boot "measured boot" OS feature that tracks hashes (from BIOS, boot loader, krnel, early drivers). "remote attestation" allows remote validation and control based on policy on a remote attestation server. Microsoft pushing TPM (Windows 8 required), but Google is not. Intel TianoCore is the only open source for UEFI. Dan has Measured Boot Tool at http://mbt.codeplex.com/ with a demo where you can also view TPM data. TPM support already on enterprise-class machines. UEFI Weaknesses. UEFI toolkits are evolving rapidly, but UEFI has weaknesses: assume user is an ally trust TPM implicitly, and attached to computer hibernate file is unprotected (disk encryption protects against this) protection migrating from hardware to firmware delays in patching and whitelist updates will UEFI really be adopted by the mainstream (smartphone hardware support, bank support, apathetic consumer support) You Can't Buy Security: Building the Open Source InfoSec Program Boris Sverdlik, ISDPodcast.com co-host Boris talked about problems typical with current security audits. "IT Security" is an oxymoron—IT exists to enable buiness, uptime, utilization, reporting, but don't care about security—IT has conflict of interest. There's no Magic Bullet ("blinky box"), no one-size-fits-all solution (e.g., Intrusion Detection Systems (IDSs)). Regulations don't make you secure. The cloud is not secure (because of shared data and admin access). Defense and pen testing is not sexy. Auditors are not solution (security not a checklist)—what's needed is experience and adaptability—need soft skills. Step 1: First thing is to Google and learn the company end-to-end before you start. Get to know the management team (not IT team), meet as many people as you can. Don't use arbitrary values such as CISSP scores. Quantitive risk assessment is a myth (e.g. AV*EF-SLE). Learn different Business Units, legal/regulatory obligations, learn the business and where the money is made, verify company is protected from script kiddies (easy), learn sensitive information (IP, internal use only), and start with low-hanging fruit (customer service reps and social engineering). Step 2: Policies. Keep policies short and relevant. Generic SANS "security" boilerplate policies don't make sense and are not followed. Focus on acceptable use, data usage, communications, physical security. Step 3: Implementation: keep it simple stupid. Open source, although useful, is not free (implementation cost). Access controls with authentication & authorization for local and remote access. MS Windows has it, otherwise use OpenLDAP, OpenIAM, etc. Application security Everyone tries to reinvent the wheel—use existing static analysis tools. Review high-risk apps and major revisions. Don't run different risk level apps on same system. Assume host/client compromised and use app-level security control. Network security VLAN != segregated because there's too many workarounds. Use explicit firwall rules, active and passive network monitoring (snort is free), disallow end user access to production environment, have a proxy instead of direct Internet access. Also, SSL certificates are not good two-factor auth and SSL does not mean "safe." Operational Controls Have change, patch, asset, & vulnerability management (OSSI is free). For change management, always review code before pushing to production For logging, have centralized security logging for business-critical systems, separate security logging from administrative/IT logging, and lock down log (as it has everything). Monitor with OSSIM (open source). Use intrusion detection, but not just to fulfill a checkbox: build rules from a whitelist perspective (snort). OSSEC has 95% of what you need. Vulnerability management is a QA function when done right: OpenVas and Seccubus are free. Security awareness The reality is users will always click everything. Build real awareness, not compliance driven checkbox, and have it integrated into the culture. Pen test by crowd sourcing—test with logging COSSP http://www.cossp.org/ - Comprehensive Open Source Security Project What Journalists Want: The Investigative Reporters' Perspective on Hacking Dave Maas, San Diego CityBeat Jason Leopold, Truthout.org The difference between hackers and investigative journalists: For hackers, the motivation varies, but method is same, technological specialties. For investigative journalists, it's about one thing—The Story, and they need broad info-gathering skills. J-School in 60 Seconds: Generic formula: Person or issue of pubic interest, new info, or angle. Generic criteria: proximity, prominence, timeliness, human interest, oddity, or consequence. Media awareness of hackers and trends: journalists becoming extremely aware of hackers with congressional debates (privacy, data breaches), demand for data-mining Journalists, use of coding and web development for Journalists, and Journalists busted for hacking (Murdock). Info gathering by investigative journalists include Public records laws. Federal Freedom of Information Act (FOIA) is good, but slow. California Public Records Act is a lot stronger. FOIA takes forever because of foot-dragging—it helps to be specific. Often need to sue (especially FBI). CPRA is faster, and requests can be vague. Dumps and leaks (a la Wikileaks) Journalists want: leads, protecting ourselves, our sources, and adapting tools for news gathering (Google hacking). Anonomity is important to whistleblowers. They want no digital footprint left behind (e.g., email, web log). They don't trust encryption, want to feel safe and secure. Whistleblower laws are very weak—there's no upside for whistleblowers—they have to be very passionate to do it. Accessibility and Security or: How I Learned to Stop Worrying and Love the Halting Problem Anna Shubina, Dartmouth College Anna talked about how accessibility and security are related. Accessibility of digital content (not real world accessibility). mostly refers to blind users and screenreaders, for our purpose. Accessibility is about parsing documents, as are many security issues. "Rich" executable content causes accessibility to fail, and often causes security to fail. For example MS Word has executable format—it's not a document exchange format—more dangerous than PDF or HTML. Accessibility is often the first and maybe only sanity check with parsing. They have no choice because someone may want to read what you write. Google, for example, is very particular about web browser you use and are bad at supporting other browsers. Uses JavaScript instead of links, often requiring mouseover to display content. PDF is a security nightmare. Executible format, embedded flash, JavaScript, etc. 15 million lines of code. Google Chrome doesn't handle PDF correctly, causing several security bugs. PDF has an accessibility checker and PDF tagging, to help with accessibility. But no PDF checker checks for incorrect tags, untagged content, or validates lists or tables. None check executable content at all. The "Halting Problem" is: can one decide whether a program will ever stop? The answer, in general, is no (Rice's theorem). The same holds true for accessibility checkers. Language-theoretic Security says complicated data formats are hard to parse and cannot be solved due to the Halting Problem. W3C Web Accessibility Guidelines: "Perceivable, Operable, Understandable, Robust" Not much help though, except for "Robust", but here's some gems: * all information should be parsable (paraphrasing) * if not parsable, cannot be converted to alternate formats * maximize compatibility in new document formats Executible webpages are bad for security and accessibility. They say it's for a better web experience. But is it necessary to stuff web pages with JavaScript for a better experience? A good example is The Drudge Report—it has hand-written HTML with no JavaScript, yet drives a lot of web traffic due to good content. A bad example is Google News—hidden scrollbars, guessing user input. Solutions: Accessibility and security problems come from same source Expose "better user experience" myth Keep your corner of Internet parsable Remember "Halting Problem"—recognize false solutions (checking and verifying tools) Stop Patching, for Stronger PCI Compliance Adam Brand, protiviti @adamrbrand, http://www.picfun.com/ Adam talked about PCI compliance for retail sales. Take an example: for PCI compliance, 50% of Brian's time (a IT guy), 960 hours/year was spent patching POSs in 850 restaurants. Often applying some patches make no sense (like fixing a browser vulnerability on a server). "Scanner worship" is overuse of vulnerability scanners—it gives a warm and fuzzy and it's simple (red or green results—fix reds). Scanners give a false sense of security. In reality, breeches from missing patches are uncommon—more common problems are: default passwords, cleartext authentication, misconfiguration (firewall ports open). Patching Myths: Myth 1: install within 30 days of patch release (but PCI §6.1 allows a "risk-based approach" instead). Myth 2: vendor decides what's critical (also PCI §6.1). But §6.2 requires user ranking of vulnerabilities instead. Myth 3: scan and rescan until it passes. But PCI §11.2.1b says this applies only to high-risk vulnerabilities. Adam says good recommendations come from NIST 800-40. Instead use sane patching and focus on what's really important. From NIST 800-40: Proactive: Use a proactive vulnerability management process: use change control, configuration management, monitor file integrity. Monitor: start with NVD and other vulnerability alerts, not scanner results. Evaluate: public-facing system? workstation? internal server? (risk rank) Decide:on action and timeline Test: pre-test patches (stability, functionality, rollback) for change control Install: notify, change control, tickets McAfee Secure & Trustmarks — a Hacker's Best Friend Jay James, Shane MacDougall, Tactical Intelligence Inc., Canada "McAfee Secure Trustmark" is a website seal marketed by McAfee. A website gets this badge if they pass their remote scanning. The problem is a removal of trustmarks act as flags that you're vulnerable. Easy to view status change by viewing McAfee list on website or on Google. "Secure TrustGuard" is similar to McAfee. Jay and Shane wrote Perl scripts to gather sites from McAfee and search engines. If their certification image changes to a 1x1 pixel image, then they are longer certified. Their scripts take deltas of scans to see what changed daily. The bottom line is change in TrustGuard status is a flag for hackers to attack your site. Entire idea of seals is silly—you're raising a flag saying if you're vulnerable.

    Read the article

  • NHibernate Pitfalls: Fetch and Paging

    - by Ricardo Peres
    This is part of a series of posts about NHibernate Pitfalls. See the entire collection here. NHibernate allows you to force loading additional references (many to one, one to one) or collections (one to many, many to many) in a query. You must know, however, that this is incompatible with paging. It’s easy to see why. Let’s say you want to get 5 products starting on the fifth, you can issue the following LINQ query: 1: session.Query<Product>().Take(5).Skip(5).ToList(); Will product this SQL in SQL Server: 1: SELECT 2: TOP (@p0) product1_4_, 3: name4_, 4: price4_ 5: FROM 6: (select 7: product0_.product_id as product1_4_, 8: product0_.name as name4_, 9: product0_.price as price4_, 10: ROW_NUMBER() OVER( 11: ORDER BY 12: CURRENT_TIMESTAMP) as __hibernate_sort_row 13: from 14: product product0_) as query 15: WHERE 16: query.__hibernate_sort_row > @p1 17: ORDER BY If, however, you wanted to bring as well the associated order details, you might be tempted to try this: 1: session.Query<Product>().Fetch(x => x.OrderDetails).Take(5).Skip(5).ToList(); Which, in turn, will produce this SQL: 1: SELECT 2: TOP (@p0) product1_4_0_, 3: order1_3_1_, 4: name4_0_, 5: price4_0_, 6: order2_3_1_, 7: product3_3_1_, 8: quantity3_1_, 9: product3_0__, 10: order1_0__ 11: FROM 12: (select 13: product0_.product_id as product1_4_0_, 14: orderdetai1_.order_detail_id as order1_3_1_, 15: product0_.name as name4_0_, 16: product0_.price as price4_0_, 17: orderdetai1_.order_id as order2_3_1_, 18: orderdetai1_.product_id as product3_3_1_, 19: orderdetai1_.quantity as quantity3_1_, 20: orderdetai1_.product_id as product3_0__, 21: orderdetai1_.order_detail_id as order1_0__, 22: ROW_NUMBER() OVER( 23: ORDER BY 24: CURRENT_TIMESTAMP) as __hibernate_sort_row 25: from 26: product product0_ 27: left outer join 28: order_detail orderdetai1_ 29: on product0_.product_id=orderdetai1_.product_id 30: ) as query 31: WHERE 32: query.__hibernate_sort_row > @p1 33: ORDER BY 34: query.__hibernate_sort_row; However, because of the JOIN, what happens is that, if your products have more than one order details, you will get several records – one per order detail – per product, which means that pagination will be broken. There is an workaround, which forces you to write your LINQ query in another way: 1: session.Query<OrderDetail>().Where(x => session.Query<Product>().Select(y => y.ProductId).Take(5).Skip(5).Contains(x.Product.ProductId)).Select(x => x.Product).ToList() Or, using HQL: 1: session.CreateQuery("select od.Product from OrderDetail od where od.Product.ProductId in (select p.ProductId from Product p skip 5 take 5)").List<Product>(); The generated SQL will then be: 1: select 2: product1_.product_id as product1_4_, 3: product1_.name as name4_, 4: product1_.price as price4_ 5: from 6: order_detail orderdetai0_ 7: left outer join 8: product product1_ 9: on orderdetai0_.product_id=product1_.product_id 10: where 11: orderdetai0_.product_id in ( 12: SELECT 13: TOP (@p0) product_id 14: FROM 15: (select 16: product2_.product_id, 17: ROW_NUMBER() OVER( 18: ORDER BY 19: CURRENT_TIMESTAMP) as __hibernate_sort_row 20: from 21: product product2_) as query 22: WHERE 23: query.__hibernate_sort_row > @p1 24: ORDER BY 25: query.__hibernate_sort_row); Which will get you what you want: for 5 products, all of their order details.

    Read the article

  • Advanced TSQL Tuning: Why Internals Knowledge Matters

    - by Paul White
    There is much more to query tuning than reducing logical reads and adding covering nonclustered indexes.  Query tuning is not complete as soon as the query returns results quickly in the development or test environments.  In production, your query will compete for memory, CPU, locks, I/O and other resources on the server.  Today’s entry looks at some tuning considerations that are often overlooked, and shows how deep internals knowledge can help you write better TSQL. As always, we’ll need some example data.  In fact, we are going to use three tables today, each of which is structured like this: Each table has 50,000 rows made up of an INTEGER id column and a padding column containing 3,999 characters in every row.  The only difference between the three tables is in the type of the padding column: the first table uses CHAR(3999), the second uses VARCHAR(MAX), and the third uses the deprecated TEXT type.  A script to create a database with the three tables and load the sample data follows: USE master; GO IF DB_ID('SortTest') IS NOT NULL DROP DATABASE SortTest; GO CREATE DATABASE SortTest COLLATE LATIN1_GENERAL_BIN; GO ALTER DATABASE SortTest MODIFY FILE ( NAME = 'SortTest', SIZE = 3GB, MAXSIZE = 3GB ); GO ALTER DATABASE SortTest MODIFY FILE ( NAME = 'SortTest_log', SIZE = 256MB, MAXSIZE = 1GB, FILEGROWTH = 128MB ); GO ALTER DATABASE SortTest SET ALLOW_SNAPSHOT_ISOLATION OFF ; ALTER DATABASE SortTest SET AUTO_CLOSE OFF ; ALTER DATABASE SortTest SET AUTO_CREATE_STATISTICS ON ; ALTER DATABASE SortTest SET AUTO_SHRINK OFF ; ALTER DATABASE SortTest SET AUTO_UPDATE_STATISTICS ON ; ALTER DATABASE SortTest SET AUTO_UPDATE_STATISTICS_ASYNC ON ; ALTER DATABASE SortTest SET PARAMETERIZATION SIMPLE ; ALTER DATABASE SortTest SET READ_COMMITTED_SNAPSHOT OFF ; ALTER DATABASE SortTest SET MULTI_USER ; ALTER DATABASE SortTest SET RECOVERY SIMPLE ; USE SortTest; GO CREATE TABLE dbo.TestCHAR ( id INTEGER IDENTITY (1,1) NOT NULL, padding CHAR(3999) NOT NULL,   CONSTRAINT [PK dbo.TestCHAR (id)] PRIMARY KEY CLUSTERED (id), ) ; CREATE TABLE dbo.TestMAX ( id INTEGER IDENTITY (1,1) NOT NULL, padding VARCHAR(MAX) NOT NULL,   CONSTRAINT [PK dbo.TestMAX (id)] PRIMARY KEY CLUSTERED (id), ) ; CREATE TABLE dbo.TestTEXT ( id INTEGER IDENTITY (1,1) NOT NULL, padding TEXT NOT NULL,   CONSTRAINT [PK dbo.TestTEXT (id)] PRIMARY KEY CLUSTERED (id), ) ; -- ============= -- Load TestCHAR (about 3s) -- ============= INSERT INTO dbo.TestCHAR WITH (TABLOCKX) ( padding ) SELECT padding = REPLICATE(CHAR(65 + (Data.n % 26)), 3999) FROM ( SELECT TOP (50000) n = ROW_NUMBER() OVER (ORDER BY (SELECT 0)) - 1 FROM master.sys.columns C1, master.sys.columns C2, master.sys.columns C3 ORDER BY n ASC ) AS Data ORDER BY Data.n ASC ; -- ============ -- Load TestMAX (about 3s) -- ============ INSERT INTO dbo.TestMAX WITH (TABLOCKX) ( padding ) SELECT CONVERT(VARCHAR(MAX), padding) FROM dbo.TestCHAR ORDER BY id ; -- ============= -- Load TestTEXT (about 5s) -- ============= INSERT INTO dbo.TestTEXT WITH (TABLOCKX) ( padding ) SELECT CONVERT(TEXT, padding) FROM dbo.TestCHAR ORDER BY id ; -- ========== -- Space used -- ========== -- EXECUTE sys.sp_spaceused @objname = 'dbo.TestCHAR'; EXECUTE sys.sp_spaceused @objname = 'dbo.TestMAX'; EXECUTE sys.sp_spaceused @objname = 'dbo.TestTEXT'; ; CHECKPOINT ; That takes around 15 seconds to run, and shows the space allocated to each table in its output: To illustrate the points I want to make today, the example task we are going to set ourselves is to return a random set of 150 rows from each table.  The basic shape of the test query is the same for each of the three test tables: SELECT TOP (150) T.id, T.padding FROM dbo.Test AS T ORDER BY NEWID() OPTION (MAXDOP 1) ; Test 1 – CHAR(3999) Running the template query shown above using the TestCHAR table as the target, we find that the query takes around 5 seconds to return its results.  This seems slow, considering that the table only has 50,000 rows.  Working on the assumption that generating a GUID for each row is a CPU-intensive operation, we might try enabling parallelism to see if that speeds up the response time.  Running the query again (but without the MAXDOP 1 hint) on a machine with eight logical processors, the query now takes 10 seconds to execute – twice as long as when run serially. Rather than attempting further guesses at the cause of the slowness, let’s go back to serial execution and add some monitoring.  The script below monitors STATISTICS IO output and the amount of tempdb used by the test query.  We will also run a Profiler trace to capture any warnings generated during query execution. DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) TC.id, TC.padding FROM dbo.TestCHAR AS TC ORDER BY NEWID() OPTION (MAXDOP 1) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; Let’s take a closer look at the statistics and query plan generated from this: Following the flow of the data from right to left, we see the expected 50,000 rows emerging from the Clustered Index Scan, with a total estimated size of around 191MB.  The Compute Scalar adds a column containing a random GUID (generated from the NEWID() function call) for each row.  With this extra column in place, the size of the data arriving at the Sort operator is estimated to be 192MB. Sort is a blocking operator – it has to examine all of the rows on its input before it can produce its first row of output (the last row received might sort first).  This characteristic means that Sort requires a memory grant – memory allocated for the query’s use by SQL Server just before execution starts.  In this case, the Sort is the only memory-consuming operator in the plan, so it has access to the full 243MB (248,696KB) of memory reserved by SQL Server for this query execution. Notice that the memory grant is significantly larger than the expected size of the data to be sorted.  SQL Server uses a number of techniques to speed up sorting, some of which sacrifice size for comparison speed.  Sorts typically require a very large number of comparisons, so this is usually a very effective optimization.  One of the drawbacks is that it is not possible to exactly predict the sort space needed, as it depends on the data itself.  SQL Server takes an educated guess based on data types, sizes, and the number of rows expected, but the algorithm is not perfect. In spite of the large memory grant, the Profiler trace shows a Sort Warning event (indicating that the sort ran out of memory), and the tempdb usage monitor shows that 195MB of tempdb space was used – all of that for system use.  The 195MB represents physical write activity on tempdb, because SQL Server strictly enforces memory grants – a query cannot ‘cheat’ and effectively gain extra memory by spilling to tempdb pages that reside in memory.  Anyway, the key point here is that it takes a while to write 195MB to disk, and this is the main reason that the query takes 5 seconds overall. If you are wondering why using parallelism made the problem worse, consider that eight threads of execution result in eight concurrent partial sorts, each receiving one eighth of the memory grant.  The eight sorts all spilled to tempdb, resulting in inefficiencies as the spilled sorts competed for disk resources.  More importantly, there are specific problems at the point where the eight partial results are combined, but I’ll cover that in a future post. CHAR(3999) Performance Summary: 5 seconds elapsed time 243MB memory grant 195MB tempdb usage 192MB estimated sort set 25,043 logical reads Sort Warning Test 2 – VARCHAR(MAX) We’ll now run exactly the same test (with the additional monitoring) on the table using a VARCHAR(MAX) padding column: DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) TM.id, TM.padding FROM dbo.TestMAX AS TM ORDER BY NEWID() OPTION (MAXDOP 1) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; This time the query takes around 8 seconds to complete (3 seconds longer than Test 1).  Notice that the estimated row and data sizes are very slightly larger, and the overall memory grant has also increased very slightly to 245MB.  The most marked difference is in the amount of tempdb space used – this query wrote almost 391MB of sort run data to the physical tempdb file.  Don’t draw any general conclusions about VARCHAR(MAX) versus CHAR from this – I chose the length of the data specifically to expose this edge case.  In most cases, VARCHAR(MAX) performs very similarly to CHAR – I just wanted to make test 2 a bit more exciting. MAX Performance Summary: 8 seconds elapsed time 245MB memory grant 391MB tempdb usage 193MB estimated sort set 25,043 logical reads Sort warning Test 3 – TEXT The same test again, but using the deprecated TEXT data type for the padding column: DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) TT.id, TT.padding FROM dbo.TestTEXT AS TT ORDER BY NEWID() OPTION (MAXDOP 1, RECOMPILE) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; This time the query runs in 500ms.  If you look at the metrics we have been checking so far, it’s not hard to understand why: TEXT Performance Summary: 0.5 seconds elapsed time 9MB memory grant 5MB tempdb usage 5MB estimated sort set 207 logical reads 596 LOB logical reads Sort warning SQL Server’s memory grant algorithm still underestimates the memory needed to perform the sorting operation, but the size of the data to sort is so much smaller (5MB versus 193MB previously) that the spilled sort doesn’t matter very much.  Why is the data size so much smaller?  The query still produces the correct results – including the large amount of data held in the padding column – so what magic is being performed here? TEXT versus MAX Storage The answer lies in how columns of the TEXT data type are stored.  By default, TEXT data is stored off-row in separate LOB pages – which explains why this is the first query we have seen that records LOB logical reads in its STATISTICS IO output.  You may recall from my last post that LOB data leaves an in-row pointer to the separate storage structure holding the LOB data. SQL Server can see that the full LOB value is not required by the query plan until results are returned, so instead of passing the full LOB value down the plan from the Clustered Index Scan, it passes the small in-row structure instead.  SQL Server estimates that each row coming from the scan will be 79 bytes long – 11 bytes for row overhead, 4 bytes for the integer id column, and 64 bytes for the LOB pointer (in fact the pointer is rather smaller – usually 16 bytes – but the details of that don’t really matter right now). OK, so this query is much more efficient because it is sorting a very much smaller data set – SQL Server delays retrieving the LOB data itself until after the Sort starts producing its 150 rows.  The question that normally arises at this point is: Why doesn’t SQL Server use the same trick when the padding column is defined as VARCHAR(MAX)? The answer is connected with the fact that if the actual size of the VARCHAR(MAX) data is 8000 bytes or less, it is usually stored in-row in exactly the same way as for a VARCHAR(8000) column – MAX data only moves off-row into LOB storage when it exceeds 8000 bytes.  The default behaviour of the TEXT type is to be stored off-row by default, unless the ‘text in row’ table option is set suitably and there is room on the page.  There is an analogous (but opposite) setting to control the storage of MAX data – the ‘large value types out of row’ table option.  By enabling this option for a table, MAX data will be stored off-row (in a LOB structure) instead of in-row.  SQL Server Books Online has good coverage of both options in the topic In Row Data. The MAXOOR Table The essential difference, then, is that MAX defaults to in-row storage, and TEXT defaults to off-row (LOB) storage.  You might be thinking that we could get the same benefits seen for the TEXT data type by storing the VARCHAR(MAX) values off row – so let’s look at that option now.  This script creates a fourth table, with the VARCHAR(MAX) data stored off-row in LOB pages: CREATE TABLE dbo.TestMAXOOR ( id INTEGER IDENTITY (1,1) NOT NULL, padding VARCHAR(MAX) NOT NULL,   CONSTRAINT [PK dbo.TestMAXOOR (id)] PRIMARY KEY CLUSTERED (id), ) ; EXECUTE sys.sp_tableoption @TableNamePattern = N'dbo.TestMAXOOR', @OptionName = 'large value types out of row', @OptionValue = 'true' ; SELECT large_value_types_out_of_row FROM sys.tables WHERE [schema_id] = SCHEMA_ID(N'dbo') AND name = N'TestMAXOOR' ; INSERT INTO dbo.TestMAXOOR WITH (TABLOCKX) ( padding ) SELECT SPACE(0) FROM dbo.TestCHAR ORDER BY id ; UPDATE TM WITH (TABLOCK) SET padding.WRITE (TC.padding, NULL, NULL) FROM dbo.TestMAXOOR AS TM JOIN dbo.TestCHAR AS TC ON TC.id = TM.id ; EXECUTE sys.sp_spaceused @objname = 'dbo.TestMAXOOR' ; CHECKPOINT ; Test 4 – MAXOOR We can now re-run our test on the MAXOOR (MAX out of row) table: DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) MO.id, MO.padding FROM dbo.TestMAXOOR AS MO ORDER BY NEWID() OPTION (MAXDOP 1, RECOMPILE) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; TEXT Performance Summary: 0.3 seconds elapsed time 245MB memory grant 0MB tempdb usage 193MB estimated sort set 207 logical reads 446 LOB logical reads No sort warning The query runs very quickly – slightly faster than Test 3, and without spilling the sort to tempdb (there is no sort warning in the trace, and the monitoring query shows zero tempdb usage by this query).  SQL Server is passing the in-row pointer structure down the plan and only looking up the LOB value on the output side of the sort. The Hidden Problem There is still a huge problem with this query though – it requires a 245MB memory grant.  No wonder the sort doesn’t spill to tempdb now – 245MB is about 20 times more memory than this query actually requires to sort 50,000 records containing LOB data pointers.  Notice that the estimated row and data sizes in the plan are the same as in test 2 (where the MAX data was stored in-row). The optimizer assumes that MAX data is stored in-row, regardless of the sp_tableoption setting ‘large value types out of row’.  Why?  Because this option is dynamic – changing it does not immediately force all MAX data in the table in-row or off-row, only when data is added or actually changed.  SQL Server does not keep statistics to show how much MAX or TEXT data is currently in-row, and how much is stored in LOB pages.  This is an annoying limitation, and one which I hope will be addressed in a future version of the product. So why should we worry about this?  Excessive memory grants reduce concurrency and may result in queries waiting on the RESOURCE_SEMAPHORE wait type while they wait for memory they do not need.  245MB is an awful lot of memory, especially on 32-bit versions where memory grants cannot use AWE-mapped memory.  Even on a 64-bit server with plenty of memory, do you really want a single query to consume 0.25GB of memory unnecessarily?  That’s 32,000 8KB pages that might be put to much better use. The Solution The answer is not to use the TEXT data type for the padding column.  That solution happens to have better performance characteristics for this specific query, but it still results in a spilled sort, and it is hard to recommend the use of a data type which is scheduled for removal.  I hope it is clear to you that the fundamental problem here is that SQL Server sorts the whole set arriving at a Sort operator.  Clearly, it is not efficient to sort the whole table in memory just to return 150 rows in a random order. The TEXT example was more efficient because it dramatically reduced the size of the set that needed to be sorted.  We can do the same thing by selecting 150 unique keys from the table at random (sorting by NEWID() for example) and only then retrieving the large padding column values for just the 150 rows we need.  The following script implements that idea for all four tables: SET STATISTICS IO ON ; WITH TestTable AS ( SELECT * FROM dbo.TestCHAR ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id = ANY (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; WITH TestTable AS ( SELECT * FROM dbo.TestMAX ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id IN (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; WITH TestTable AS ( SELECT * FROM dbo.TestTEXT ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id IN (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; WITH TestTable AS ( SELECT * FROM dbo.TestMAXOOR ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id IN (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; SET STATISTICS IO OFF ; All four queries now return results in much less than a second, with memory grants between 6 and 12MB, and without spilling to tempdb.  The small remaining inefficiency is in reading the id column values from the clustered primary key index.  As a clustered index, it contains all the in-row data at its leaf.  The CHAR and VARCHAR(MAX) tables store the padding column in-row, so id values are separated by a 3999-character column, plus row overhead.  The TEXT and MAXOOR tables store the padding values off-row, so id values in the clustered index leaf are separated by the much-smaller off-row pointer structure.  This difference is reflected in the number of logical page reads performed by the four queries: Table 'TestCHAR' logical reads 25511 lob logical reads 000 Table 'TestMAX'. logical reads 25511 lob logical reads 000 Table 'TestTEXT' logical reads 00412 lob logical reads 597 Table 'TestMAXOOR' logical reads 00413 lob logical reads 446 We can increase the density of the id values by creating a separate nonclustered index on the id column only.  This is the same key as the clustered index, of course, but the nonclustered index will not include the rest of the in-row column data. CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestCHAR (id); CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestMAX (id); CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestTEXT (id); CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestMAXOOR (id); The four queries can now use the very dense nonclustered index to quickly scan the id values, sort them by NEWID(), select the 150 ids we want, and then look up the padding data.  The logical reads with the new indexes in place are: Table 'TestCHAR' logical reads 835 lob logical reads 0 Table 'TestMAX' logical reads 835 lob logical reads 0 Table 'TestTEXT' logical reads 686 lob logical reads 597 Table 'TestMAXOOR' logical reads 686 lob logical reads 448 With the new index, all four queries use the same query plan (click to enlarge): Performance Summary: 0.3 seconds elapsed time 6MB memory grant 0MB tempdb usage 1MB sort set 835 logical reads (CHAR, MAX) 686 logical reads (TEXT, MAXOOR) 597 LOB logical reads (TEXT) 448 LOB logical reads (MAXOOR) No sort warning I’ll leave it as an exercise for the reader to work out why trying to eliminate the Key Lookup by adding the padding column to the new nonclustered indexes would be a daft idea Conclusion This post is not about tuning queries that access columns containing big strings.  It isn’t about the internal differences between TEXT and MAX data types either.  It isn’t even about the cool use of UPDATE .WRITE used in the MAXOOR table load.  No, this post is about something else: Many developers might not have tuned our starting example query at all – 5 seconds isn’t that bad, and the original query plan looks reasonable at first glance.  Perhaps the NEWID() function would have been blamed for ‘just being slow’ – who knows.  5 seconds isn’t awful – unless your users expect sub-second responses – but using 250MB of memory and writing 200MB to tempdb certainly is!  If ten sessions ran that query at the same time in production that’s 2.5GB of memory usage and 2GB hitting tempdb.  Of course, not all queries can be rewritten to avoid large memory grants and sort spills using the key-lookup technique in this post, but that’s not the point either. The point of this post is that a basic understanding of execution plans is not enough.  Tuning for logical reads and adding covering indexes is not enough.  If you want to produce high-quality, scalable TSQL that won’t get you paged as soon as it hits production, you need a deep understanding of execution plans, and as much accurate, deep knowledge about SQL Server as you can lay your hands on.  The advanced database developer has a wide range of tools to use in writing queries that perform well in a range of circumstances. By the way, the examples in this post were written for SQL Server 2008.  They will run on 2005 and demonstrate the same principles, but you won’t get the same figures I did because 2005 had a rather nasty bug in the Top N Sort operator.  Fair warning: if you do decide to run the scripts on a 2005 instance (particularly the parallel query) do it before you head out for lunch… This post is dedicated to the people of Christchurch, New Zealand. © 2011 Paul White email: @[email protected] twitter: @SQL_Kiwi

    Read the article

  • Announcing Oracle Enterprise Content Management Suite 11g

    - by [email protected]
    Today Oracle announced Oracle Enterprise Content Management Suite 11g. This is a major release for us, and reinforces our three key themes at Oracle: Complete New in this release - Oracle ECM Suite 11g is built on a single, unified repository. Every piece of content - documents, HTML pages, digital assets, scanned images - is stored and accessbile directly from the repository, whether you are working on websites, creating brand logos, processing accounts payable invoices, or running records and retention functions. It makes complete, end-to-end management of content possible, from the point it enters the organization, through its entire lifecycle. Also new in this release, the installation, access, monitoring and administration of Oracle ECM Suite 11g is centralized. As a complete system, organizations can lower the costs of training and usage by having a centralized source of information that is easily administered. As part of this new unified repository release, Oracle has released a benchmarking white paper that shows the extreme performance and scalability of Oracle ECM Suite. When tested on a two node UCM Server running on Sun Oracle DB Machine Half Rack Hardware with an Exadata storage server, Oracle ECM Suite 11g is able to ingest over 178 million documents per day. Open Oracle ECM Suite 11g is built on a service-oriented architecture. All functions are available through standards-based services calls in Web Services or Java. In this release Oracle unveils Open Web Content Management. Open Web Content Management is a revolutionary approach to web content management that decouples the content management process from the process of creating web applications. One piece of this approach is our one-click web content management. With one click, a web application builder can drag content services into their application, enabling their users to also edit content with just one click. Open Web Content Management is also open because it enables Web developers to add Web content management to new and existing JavaServer Pages (JSP), JavaServer Faces (JSF) and Oracle Application Development Framework (ADF) Faces applications Open content distribution - Oracle ECM Suite 11g offers flexible deployment options with a built-in smart cache so organizations can deliver Web sites or Web applications without requiring Oracle ECM Suite as part of the delivery system Integrated Oracle ECM Suite 11g also offers a series of next generation desktop integrations, providing integrations such as: New MS Office integration with menus to access managed content, insert managed links, and compare managed documents using standard MS Office reviewing tools Automatic identity tagging of documents on download - to help users understand which versions they are viewing and prevent duplicate content items in the content repository. New "smart productivity folders" to show a users workflow inbox, saved searches and checked out content directly from Windows Explorer Drag and drop metadata pop-ups Check in and check out for all file formats with any standard WebDAV server As part of Oracle's Enterprise Application Documents initiative, Oracle Content Management 11g also provides certified application integrations with solution templates You can read the press release here. You can see more assets at the launch center here. You can sign up for the announcement webinar and hear more about the new features here. You can read the benchmarking study here.

    Read the article

< Previous Page | 182 183 184 185 186 187 188 189 190 191 192 193  | Next Page >