Search Results

Search found 282 results on 12 pages for 'aggregation'.

Page 9/12 | < Previous Page | 5 6 7 8 9 10 11 12  | Next Page >

  • Monitoring AWS Systems Behind ElasticBeanStalk

    - by A. Avadis
    So I'm getting a company set up in the Amazon Cloud -- creating IAAS protocol/solutions/standardized implementation, etc while also being the SysAdmin for individual systems, app environments, and day-to-day uptime. One of the biggest issues I'm having is tracking various system/application logs, as well as logging/monitoring/archiving system metrics like memory usage, cpu usage, etc etc In a centralized fashion. E.g. -- Nagios + Urchin. The BIGGEST impediment to my endeavors is the following: The company application is deployed in the form of a Java *.WAR file, uploaded to an Elastic BeanStalk application environment, load balancing and auto-scaling between 3(min) and 10(max) servers, and the EC2's that run the application are fired up and disposed of ad-hoc. That is to say, I can't monitor the individual EC2's for very long because so many are being terminated then auto-provisioned/auto-scaled on the fly -- so I'd constantly be having to "monitor what I'm monitoring", and continuously remove/add EC2 machine addresses to my monitoring lists. IS there some sort of way to use monitoring tools like Zabbix or Nagios to monitor the ElasticBeanStalk, and have it automatically add on new EC2's, and remove terminated/failed EC2's from its monitoring list automatically? Furthermore, is there anything I can do with GrayLog to achieve similar results with the aggregation/centralization of my application logs from multiple EC2 instances into ONE consolidated set of logs/events? If not GrayLog, is there ANYTHING LIKE GrayLog that can automatically detect what EC2 members are being added/removed from the environment, and collect the logs from them automatically? Any and all advice or direction is appreciated. Thanks much, and cheers!!

    Read the article

  • Query Logging in Analysis Services

    - by MikeD
    On a project I work on, we capture the queries that get executed on our Analysis Services instance (SQL Server 2008 R2) and use the table for helping us to build aggregations and also we aggregate the query log daily into a data warehouse of operational data so we can track usage of our Analysis databases by users over time. We've learned a couple of helpful things about this logging that I'd like to share here.First off, the query log table automatically gets cleaned out by SSAS under a few conditions - schema changes to the analysis database and even regular data and aggregation processing can delete rows in the table. We like to keep these logs longer than that, so we have a trigger on the table that copies all rows into another table with the same structure:Here is our trigger code:CREATE TRIGGER [dbo].[SaveQueryLog] on [dbo].[OlapQueryLog] AFTER INSERT AS       INSERT INTO dbo.[OlapQueryLog_History] (MSOLAP_Database, MSOLAP_ObjectPath, MSOLAP_User, Dataset, StartTime, Duration)      SELECT MSOLAP_Database, MSOLAP_ObjectPath, MSOLAP_User, Dataset, StartTime, Duration FROM inserted Second, the query logging process is "best effort" - if SSAS cannot connect to the database listed in the QueryLogConnectionString in the Analysis Server properties, it just stops logging - it doesn't generate any errors to the client at all, which is a good thing. Once it stops logging, it doesn't retry later - an hour, a day, a week, or even a month later, so long as the service doesn't restart.That has burned us a couple of times, when we have made changes to the service account that is used for SSAS, and that account doesn't have access to the database we want to log to. The last time this happened, we noticed a while later that no logging was taking place, and I determined that the service account didn't have sufficient permissions, so I made the necessary changes to give that service account access to the logging database. I first tried just the db_datawriter role and that wasn't enough, so I granted the service account membership in the db_owner role. Yes, that's a much bigger set of permissions, but I didn't want to search out the specific permissions at the time. Once I determined that the service account had the appropriate permissions, I wanted to get query logging restarted from SSAS, and I wondered how to do that? Having just used a larger hammer than necessary with the db_owner role membership, I considered just restarting SSAS to get it logging again. However, this was a production server, and it was in the middle of business hours, and there were active users connecting to that SSAS instance, so I thought better of it.As I considered the options, I remembered that the first time I set up query logging, by putting in a valid connection string to the QueryLogConnectionString server property, logging started immediately after I saved the properties. I wondered if I could make some other change to the connection string so that the query logging would start again without restarting the service. I went into the connection string dialog, went to the All page, and looked at the properties I could change that wouldn't affect the actual connection. Aha! The Application Name property would do just nicely - I set it to "SSAS Query Logging" (it was previously blank) and saved the changes to the server properties. And the query logging started up right away. If I need to get this running again in the future, I could just make a small change in the Application Name property again, save it, and even change it back again if I wanted to.The other nice side effect of setting the Application Name property is that now I can see (and possibly filter for or filter out) the SQL activity in that database that is related to the query logging process in Profiler:  To sum up:The SSAS Query Logging process will automatically delete rows from the QueryLog table, so if you want to keep them longer, put a trigger on the table to copy the rows to another tableThe SSAS service account requires more than db_datawriter role membership (and probably less than db_owner) in the database specified in the QueryLogConnectionString server property to successfully insert log rows to the QueryLog  table.Query logging will stop quietly whenever it encounters an error. Make a change to the QueryLogConnectionString server property (such as the Application Name attribute) to get query logging to restart and you won't have to restart the service.

    Read the article

  • Analysis Services (SSAS) - Unexpected Internal Error when processing (ProcessUpdate). Workaround/Resolution

    - by James Rogers
    Many implementations require the use of ProcessUpdate to support Type 1 slowly changing dimensions. ProcessUpdate drops all of the affected indexes and aggregations in partitions affected by data that changes in the Dimension on which the ProcessUpdate is being performed. Twice now I have had situations where the processing fails with "Internal error: An unexpected exception occurred." Any subsequent ProcessUpdate processing will also fail with the same error. In talking with Microsoft the issue is corrupt indexes for the Dimension(s) being processed in the partitions of the affected measure group. I cannot guarantee that the following will correct your problem but it did in my case and saved us quite a bit of down time.   Workaround: ProcessIndexes on the entire cube that is being processed and throwing the error. This corrected the problem on both 2008 and 2008 R2.   Pros:  Does not require a complete rebuild of the data (ProcessFull) for either the Dimension or Cube. User access can continue while this ProcessIndexes in underway.   Cons: Can take a long time, especially on large cubes with many partitions, dimensions and/or aggregations. Query Performance is usually severely impacted due to the memory and CPU requirements for Aggregation and Index building   <Batch http://schemas.microsoft.com/analysisservices/2003/engine"http://schemas.microsoft.com/analysisservices/2003/engine">  <Parallel>     <Process xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ddl2="http://schemas.microsoft.com/analysisservices/2003/engine/2" xmlns:ddl2_2="http://schemas.microsoft.com/analysisservices/2003/engine/2/2" xmlns:ddl100_100="http://schemas.microsoft.com/analysisservices/2008/engine/100/100" xmlns:ddl200="http://schemas.microsoft.com/analysisservices/2010/engine/200" xmlns:ddl200_200="http://schemas.microsoft.com/analysisservices/2010/engine/200/200">       <Object>         <DatabaseID>MyDatabase</DatabaseID>         <CubeID>MyCube</CubeID>       </Object>       <Type>ProcessIndexes</Type>       <WriteBackTableCreation>UseExisting</WriteBackTableCreation>     </Process>  </Parallel> </Batch>   The cube where the corruption exists can be found by having Profiler running while the ProcessUpdate is executing. The first partition that displays the "The Job has ended in failure." message in the TextData column will be part of the cube/measuregroup that has the corruption. You can try to run ProcessIndexes on just that measure group. This may correct the problem and save additional time if you have other large measure groups in the cube that are not affected by the corruption.   Remember to execute your normal ProcessUpdate batch after the successful completion of the ProcessIndexes. The ProcessIndexes does not pick up data changes.   Things that did not work: ProcessClearIndexes - why this doesn't work and ProcessIndexes does is unclear at this point. ProcessFull on the partition in question. In my latest case, this would clear up the problem for that partition. However, the next partition the ProcessUpdate touched that had data in it would generate and error. This leads me to believe the corruption problem will exist in all partitions in the affected measure group that have data in them.   NOTE: I experience this problem in both a SQL 2008 and SQL 2008 R2 Analysis Services environment, on separate built from the same relational database. This leads me to believe that some data condition in the tables used for the Dimension processing caused the corruption since the two environments were on physically separate hardware. I am waiting on Microsoft to analyze the dumps to give us more insight into what actually caused the corruption and will update this post accordingly.

    Read the article

  • Silverlight Cream for March 08, 2011 -- #1056

    - by Dave Campbell
    In this Issue: Joost van Schaik, Manas Patnaik, Kevin Hoffman, Jesse Liberty, Deborah Kurata, Dhananjay Kumar, Dennis Delimarsky, Samuel Jack, Peter Kuhn, WindowsPhoneGeek, and Jfo. Above the Fold: Silverlight: "How I let the trees grow" Peter Kuhn WP7: "Simple Windows Phone 7 / Silverlight drag/flick behavior" Joost van Schaik Shoutouts: SilverlightShow has their top 5 from last week posted, plus the ECOContest is ready to be voted on: SilverlightShow for Feb 28 - March 06, 2011 Drew DeVault is a young man involved with the Microsoft Student Insiders. He gave a WP7 presentation at RMTT and has posted his material: Post-Session: Windows Phone 7 @ RMTT Rui Marinho has an app in the ECO Contest called Forest Findr. is based on the BIng Map Control for silverlight and Sql Spatial data, and helps you find Forests and get geolocated pictures and wikipedia information, and has a post up with a bunch of info on it here: Forest Findr. my entry on the SilverlightShow EcoContest From SilverlightCream.com: Simple Windows Phone 7 / Silverlight drag/flick behavior Joost van Schaik has a behavior that makes *anything* draggable and 'flickable' in WP7 ... read the intro, scroll to the bottom to watch the demo, and then grab up the code... cool stuff, Joost! Data Aggregation Using Presentation Model in RIA and Silverlight 4 Manas Patnaik sent me a link to his blog, and it appears he's got lots of Silverlight goodness out there so you'll be hearing more about him. This first post is on the Presentation Model in RIA and Silverlight 4... good discussion, diagrams and code... good job, Manas! WP7 for iPhone and Android Developers - Advanced UI Kevin Hoffman has part 3 of an ambitious 12-part tutorial series up on WP7 development ... this go-around is concentrating on Advanced UI - Panorama/Pivot controls, DataBinding, ObservableCollections, and Converters... whew! Sterling DB on top of Isolated Storage – 2 Jesse Liberty has part 2 of his Sterling series up... this time setting up the database in App.xaml so it can be used for dealing with tombstoning. Silverlight Charting: Formatting the Tick Marks Deborah Kurata's next chart tutorial is all about showing you how to continue to dress up your charts.. this time by formatting the tick marks... if you don't know what that is... check out the first image in the post. Stored Procedure in WCF Data Service Dhananjay Kumar has a very nice tutorial up on using a stored proc with WCF Data Services... I happen to know someone working on just that at this time. If you have this in mind, here's a step-by-step guide to getting it done. Windows Phone 7 – Episode 5 – Pages Dennis Delimarsky has part 5 of his WP7 tutorial series up and is discussing Pages in this 17 minute video. Unpacking Simon Squared: My mini framework-independent animation library Samuel Jack has not only Open-Sourced the WP7 game he built and blogged about, but he's now explaining some of the structure of the game in posts such as this one about the animation library he wrote that his game is built on. How I let the trees grow Peter Kuhn shares with us the code he used for the tree animation in his ECO Contest entry. There's a lot to learn in this post about performance ... the fully-animated tree has about 20K elements... 5K branches and 20K leaves... check it out. WP7 ToastPrompt in depth WindowsPhoneGeek takes a deep dive into the ToastPrompt control in the Coding4fun Toolkit... everything you need to completely use the control including sample code. Beware the loaded event Jfo talks about another frustration point she had with WP7 development, and that is around the use of the loaded event... read these tips from someone that's been there. Stay in the 'Light! Twitter SilverlightNews | Twitter WynApse | WynApse.com | Tagged Posts | SilverlightCream Join me @ SilverlightCream | Phoenix Silverlight User Group Technorati Tags: Silverlight    Silverlight 3    Silverlight 4    Windows Phone MIX10

    Read the article

  • Windows Azure Recipe: Mobile Computing

    - by Clint Edmonson
    A while back, mashups were all the rage. The idea was to compose solutions that provided aggregation and integration across applications and services to make information more available, useful, and personal. Mashups ushered in the era of Web 2.0 in all it’s socially connected goodness. They taught us that to be successful, we needed to add web service APIs to our web applications. Web and client based mashups met with great success and have evolved even further with the introduction of the internet connected smartphone. Nothing is more available, useful, or personal than our smartphones. The current generation of cloud connected mobile computing mashups allow our mobilized workforces to receive, process, and react to information from disparate sources faster than ever before. Drivers Integration Reach Time to market Solution Here’s a sketch of a prototypical mobile computing solution using Windows Azure: Ingredients Web Role – with the phone running a dedicated client application, the web role is responsible for serving up backend web services that implement the solution’s core connected functionality. Database – used to store core operational and workflow data for the solution’s web services. Access Control – this service is used to authenticate and manage users identity, roles, and groups, possibly in conjunction with 3rd identity providers such as Windows LiveID, Google, Yahoo!, and Facebook. Worker Role – this role is used to handle the orchestration of long-running, complex, asynchronous operations. While much of the integration and interaction with other services can be handled directly by the mobile client application, it’s possible that the backend may need to integrate with 3rd party services as well. Offloading this work to a worker role better distributes computing resources and keeps the web roles focused on direct client interaction. Queues – these provide reliable, persistent messaging between applications and processes. They are an absolute necessity once asynchronous processing is involved. Queues facilitate the flow of distributed events and allow a solution to send push notifications back to mobile devices at appropriate times. Training & Resources These links point to online Windows Azure training labs and resources where you can learn more about the individual ingredients described above. (Note: The entire Windows Azure Training Kit can also be downloaded for offline use.) Windows Azure (16 labs) Windows Azure is an internet-scale cloud computing and services platform hosted in Microsoft data centers, which provides an operating system and a set of developer services which can be used individually or together. It gives developers the choice to build web applications; applications running on connected devices, PCs, or servers; or hybrid solutions offering the best of both worlds. New or enhanced applications can be built using existing skills with the Visual Studio development environment and the .NET Framework. With its standards-based and interoperable approach, the services platform supports multiple internet protocols, including HTTP, REST, SOAP, and plain XML SQL Azure (7 labs) Microsoft SQL Azure delivers on the Microsoft Data Platform vision of extending the SQL Server capabilities to the cloud as web-based services, enabling you to store structured, semi-structured, and unstructured data. Windows Azure Services (9 labs) As applications collaborate across organizational boundaries, ensuring secure transactions across disparate security domains is crucial but difficult to implement. Windows Azure Services provides hosted authentication and access control using powerful, secure, standards-based infrastructure. Windows Azure Toolkit for Windows Phone The Windows Azure Toolkit for Windows Phone is designed to make it easier for you to build mobile applications that leverage cloud services running in Windows Azure. The toolkit includes Visual Studio project templates for Windows Phone and Windows Azure, class libraries optimized for use on the phone, sample applications, and documentation Windows Azure Toolkit for iOS The Windows Azure Toolkit for iOS is a toolkit for developers to make it easy to access Windows Azure storage services from native iOS applications. The toolkit can be used for both iPhone and iPad applications, developed using Objective-C and XCode. Windows Azure Toolkit for Android The Windows Azure Toolkit for Android is a toolkit for developers to make it easy to work with Windows Azure from native Android applications. The toolkit can be used for native Android applications developed using Eclipse and the Android SDK. See my Windows Azure Resource Guide for more guidance on how to get started, including links web portals, training kits, samples, and blogs related to Windows Azure.

    Read the article

  • October 2013 Fusion Middleware (FMW) Proactive Patches released

    - by Irina
    We are glad to announce that the following Fusion Middleware (FMW) Proactive  patches were released on October 15, 2013.Bundle PatchesBundle patches are collections of controlled, well tested critical bug fixes for a specific product  which may include security contents and occasionally minor enhancements. These are cumulative in nature meaning the latest bundle patch in a particular series includes the contents of the previous bundle patches released.  A suite bundle patch is an aggregation of multiple product  bundle patches that are part of a product suite. Oracle Identity Management Suite Bundle Patch 11.1.1.5.5 consisting of Oracle Identity Manager (OIM) 11.1.1.5.9 bundle patch Oracle Access Manager (OAM) 11.1.1.5.6 bundle patch. Oracle Adaptive Access Manager (OAAM) 11.1.1.5.2 bundle patch. Oracle Entitlement Server (OES) 11.1.1.5.4 bundle patch. Oracle Identity Management Suite Bundle Patch 11.1.2.0.4 consisting of Oracle Access Manager (OAM) 11.1.2.0.4 bundle patch. Oracle Adaptive Access Manager (OAAM) 11.1.2.0.2 bundle patch. Oracle Entitlement Server (OES) 11.1.2.0.2 bundle patch. Oracle Identity Analytics (OIA ) 11.1.1.5.6  bundle patch. Oracle GlassFish Server (OGFS) 2.1.1.22, 3.0.1.8 and 3.1.2.7 bundle patches. Oracle iPlanet Web Server (OiWS) 7.0.18 bundle patch Oracle SOA Suite (SOA) 11.1.1.7.1 bundle patch Oracle WebCenter Portal (WCP) 11.1.1.8.1 bundle patch Sun Role Manager (SRM) 4.1.7 and 5.0.3.2 bundle patches. Patch Set Updates (PSU)Patch Set Updates (PSU)  are collections of well controlled, well tested critical bug fixes for a specific product  that have been proven in customer environments. PSUs  may include security contents but no  enhancements are included. These are cumulative in nature meaning the latest PSU  in a particular series includes the contents of the previous PSUs  released. Oracle Exalogic 2.0.3.0.4 Physical Linux x86-64 and 2.0.4.0.4 Physical Solaris x86-64 PSUs. Oracle WebLogic Server 10.3.6.0.6 and 12.1.1.0.6 PSUs. Critical Patch Update (CPU)The Critical Patch Update program is Oracle's quarterly release of security fixes.The following additional patches were released as part of Oracle's Critical Patch Update program: Oracle JDeveloper 11.1.2.3.0, 11.1.2.4.0 and 12.1.2.0.0 Oracle Outside In Technology 8.4.0 and  8.4.1 Oracle Portal 11.1.1.6.0 Oracle Security Service  11.1.1.6.0, 11.1.1.7.0 and 12.1.2.0.0 Oracle WebCache 11.1.1.6.0 and 11.1.1.7.0 Oracle WebCenter Content 10.1.3.5.1, 11.1.1.6.0, 11.1.1.7.0 and 11.1.1.8.0 Oracle WebServices 10.1.3.5.0 and 11.1.1.6.0 For more information: Master Notes on Fusion Middleware Proactive Patching PSU and CPU October 2013  Availability Document Critical Patch Update Advisory -  October 2013

    Read the article

  • Columnstore Case Study #1: MSIT SONAR Aggregations

    - by aspiringgeek
    Preamble This is the first in a series of posts documenting big wins encountered using columnstore indexes in SQL Server 2012 & 2014.  Many of these can be found in this deck along with details such as internals, best practices, caveats, etc.  The purpose of sharing the case studies in this context is to provide an easy-to-consume quick-reference alternative. Why Columnstore? If we’re looking for a subset of columns from one or a few rows, given the right indexes, SQL Server can do a superlative job of providing an answer. If we’re asking a question which by design needs to hit lots of rows—DW, reporting, aggregations, grouping, scans, etc., SQL Server has never had a good mechanism—until columnstore. Columnstore indexes were introduced in SQL Server 2012. However, they're still largely unknown. Some adoption blockers existed; yet columnstore was nonetheless a game changer for many apps.  In SQL Server 2014, potential blockers have been largely removed & they're going to profoundly change the way we interact with our data.  The purpose of this series is to share the performance benefits of columnstore & documenting columnstore is a compelling reason to upgrade to SQL Server 2014. App: MSIT SONAR Aggregations At MSIT, performance & configuration data is captured by SCOM. We archive much of the data in a partitioned data warehouse table in SQL Server 2012 for reporting via an application called SONAR.  By definition, this is a primary use case for columnstore—report queries requiring aggregation over large numbers of rows.  New data is refreshed each night by an automated table partitioning mechanism—a best practices scenario for columnstore. The Win Compared to performance using classic indexing which resulted in the expected query plan selection including partition elimination vs. SQL Server 2012 nonclustered columnstore, query performance increased significantly.  Logical reads were reduced by over a factor of 50; both CPU & duration improved by factors of 20 or more.  Other than creating the columnstore index, no special modifications or tweaks to the app or databases schema were necessary to achieve the performance improvements.  Existing nonclustered indexes were rendered superfluous & were deleted, thus mitigating maintenance challenges such as defragging as well as conserving disk capacity. Details The table provides the raw data & summarizes the performance deltas. Logical Reads (8K pages) CPU (ms) Durn (ms) Columnstore 160,323 20,360 9,786 Conventional Table & Indexes 9,053,423 549,608 193,903 ? x56 x27 x20 The charts provide additional perspective of this data.  "Conventional vs. Columnstore Metrics" document the raw data.  Note on this linear display the magnitude of the conventional index performance vs. columnstore.  The “Metrics (?)” chart expresses these values as a ratio. Summary For DW, reports, & other BI workloads, columnstore often provides significant performance enhancements relative to conventional indexing.  I have documented here, the first in a series of reports on columnstore implementations, results from an initial implementation at MSIT in which logical reads were reduced by over a factor of 50; both CPU & duration improved by factors of 20 or more.  Subsequent features in this series document performance enhancements that are even more significant. 

    Read the article

  • October 2013 FMW Proactive Patches Released

    - by mustafakaya
    The following Fusion Middleware (FMW) Proactive  patches were released on October 15, 2013. Bundle Patches : Bundle patches are collections of controlled, well tested critical bug fixes for a specific product  which may include security contents and occasionally minor enhancements. These are cumulative in nature meaning the latest bundle patch in a particular series includes the contents of the previous bundle patches released.  A suite bundle patch is an aggregation of multiple product  bundle patches that are part of a product suite. Oracle Identity Management Suite Bundle Patch 11.1.1.5.5 consisting of Oracle Identity Manager (OIM) 11.1.1.5.9 bundle patch Oracle Access Manager (OAM) 11.1.1.5.6 bundle patch. Oracle Adaptive Access Manager (OAAM) 11.1.1.5.2 bundle patch. Oracle Entitlement Server (OES) 11.1.1.5.4 bundle patch. Oracle Identity Management Suite Bundle Patch 11.1.2.0.4 consisting of Oracle Access Manager (OAM) 11.1.2.0.4 bundle patch. Oracle Adaptive Access Manager (OAAM) 11.1.2.0.2 bundle patch. Oracle Entitlement Server (OES) 11.1.2.0.2 bundle patch. Oracle Identity Analytics (OIA ) 11.1.1.5.6  bundle patch. Oracle GlassFish Server (OGFS) 2.1.1.22, 3.0.1.8 and 3.1.2.7 bundle patches. Oracle iPlanet Web Server (OiWS) 7.0.18 bundle patch Oracle SOA Suite (SOA) 11.1.1.7.1 bundle patch Oracle WebCenter Portal (WCP) 11.1.1.8.1 bundle patch Sun Role Manager (SRM) 4.1.7 and 5.0.3.2 bundle patches. Patch Set Updates (PSU) Patch Set Updates (PSU)  are collections of well controlled, well tested critical bug fixes for a specific product  that have been proven in customer environments. PSUs  may include security contents but no  enhancements are included. These are cumulative in nature meaning the latest PSU  in a particular series includes the contents of the previous PSUs  released.  Oracle Exalogic 2.0.3.0.4 Physical Linux x86-64 and 2.0.4.0.4 Physical Solaris x86-64 PSUs. Oracle WebLogic Server 10.3.6.0.6 and 12.1.1.0.6 PSUs. Critical Patch Update (CPU) : The Critical Patch Update program is Oracle's quarterly release of security fixes. The following additional patches were released as part of Oracle's Critical Patch Update program: Oracle JDeveloper 11.1.2.3.0, 11.1.2.4.0 and 12.1.2.0.0 Oracle Outside In Technology 8.4.0 and  8.4.1 Oracle Portal 11.1.1.6.0 Oracle Security Service  11.1.1.6.0, 11.1.1.7.0 and 12.1.2.0.0 Oracle WebCache 11.1.1.6.0 and 11.1.1.7.0 Oracle WebCenter Content 10.1.3.5.1, 11.1.1.6.0, 11.1.1.7.0 and 11.1.1.8.0 Oracle WebServices 10.1.3.5.0 and 11.1.1.6.0 For more information; Master Notes on Fusion Middleware Proactive Patching. PSU and CPU October 2013  Availability Document Critical Patch Update Advisory -  October 2013 

    Read the article

  • Columnstore Case Study #1: MSIT SONAR Aggregations

    - by aspiringgeek
    Preamble This is the first in a series of posts documenting big wins encountered using columnstore indexes in SQL Server 2012 & 2014.  Many of these can be found in this deck along with details such as internals, best practices, caveats, etc.  The purpose of sharing the case studies in this context is to provide an easy-to-consume quick-reference alternative. Why Columnstore? If we’re looking for a subset of columns from one or a few rows, given the right indexes, SQL Server can do a superlative job of providing an answer. If we’re asking a question which by design needs to hit lots of rows—DW, reporting, aggregations, grouping, scans, etc., SQL Server has never had a good mechanism—until columnstore. Columnstore indexes were introduced in SQL Server 2012. However, they're still largely unknown. Some adoption blockers existed; yet columnstore was nonetheless a game changer for many apps.  In SQL Server 2014, potential blockers have been largely removed & they're going to profoundly change the way we interact with our data.  The purpose of this series is to share the performance benefits of columnstore & documenting columnstore is a compelling reason to upgrade to SQL Server 2014. App: MSIT SONAR Aggregations At MSIT, performance & configuration data is captured by SCOM. We archive much of the data in a partitioned data warehouse table in SQL Server 2012 for reporting via an application called SONAR.  By definition, this is a primary use case for columnstore—report queries requiring aggregation over large numbers of rows.  New data is refreshed each night by an automated table partitioning mechanism—a best practices scenario for columnstore. The Win Compared to performance using classic indexing which resulted in the expected query plan selection including partition elimination vs. SQL Server 2012 nonclustered columnstore, query performance increased significantly.  Logical reads were reduced by over a factor of 50; both CPU & duration improved by factors of 20 or more.  Other than creating the columnstore index, no special modifications or tweaks to the app or databases schema were necessary to achieve the performance improvements.  Existing nonclustered indexes were rendered superfluous & were deleted, thus mitigating maintenance challenges such as defragging as well as conserving disk capacity. Details The table provides the raw data & summarizes the performance deltas. Logical Reads (8K pages) CPU (ms) Durn (ms) Columnstore 160,323 20,360 9,786 Conventional Table & Indexes 9,053,423 549,608 193,903 ? x56 x27 x20 The charts provide additional perspective of this data.  "Conventional vs. Columnstore Metrics" document the raw data.  Note on this linear display the magnitude of the conventional index performance vs. columnstore.  The “Metrics (?)” chart expresses these values as a ratio. Summary For DW, reports, & other BI workloads, columnstore often provides significant performance enhancements relative to conventional indexing.  I have documented here, the first in a series of reports on columnstore implementations, results from an initial implementation at MSIT in which logical reads were reduced by over a factor of 50; both CPU & duration improved by factors of 20 or more.  Subsequent features in this series document performance enhancements that are even more significant. 

    Read the article

  • October 2013 Fusion Middleware (FMW) Proactive Patches released

    - by PCat
    We are glad to announce that the following Fusion Middleware (FMW) Proactive  patches were released on October 15, 2013.Bundle PatchesBundle patches are collections of controlled, well tested critical bug fixes for a specific product  which may include security contents and occasionally minor enhancements. These are cumulative in nature meaning the latest bundle patch in a particular series includes the contents of the previous bundle patches released.  A suite bundle patch is an aggregation of multiple product  bundle patches that are part of a product suite. Oracle Identity Management Suite Bundle Patch 11.1.1.5.5 consisting of Oracle Identity Manager (OIM) 11.1.1.5.9 bundle patch Oracle Access Manager (OAM) 11.1.1.5.6 bundle patch. Oracle Adaptive Access Manager (OAAM) 11.1.1.5.2 bundle patch. Oracle Entitlement Server (OES) 11.1.1.5.4 bundle patch. Oracle Identity Management Suite Bundle Patch 11.1.2.0.4 consisting of Oracle Access Manager (OAM) 11.1.2.0.4 bundle patch. Oracle Adaptive Access Manager (OAAM) 11.1.2.0.2 bundle patch. Oracle Entitlement Server (OES) 11.1.2.0.2 bundle patch. Oracle Identity Analytics (OIA ) 11.1.1.5.6  bundle patch. Oracle GlassFish Server (OGFS) 2.1.1.22, 3.0.1.8 and 3.1.2.7 bundle patches. Oracle iPlanet Web Server (OiWS) 7.0.18 bundle patch Oracle SOA Suite (SOA) 11.1.1.7.1 bundle patch Oracle WebCenter Portal (WCP) 11.1.1.8.1 bundle patch Sun Role Manager (SRM) 4.1.7 and 5.0.3.2 bundle patches. Patch Set Updates (PSU)Patch Set Updates (PSU)  are collections of well controlled, well tested critical bug fixes for a specific product  that have been proven in customer environments. PSUs  may include security contents but no  enhancements are included. These are cumulative in nature meaning the latest PSU  in a particular series includes the contents of the previous PSUs  released. Oracle Exalogic 2.0.3.0.4 Physical Linux x86-64 and 2.0.4.0.4 Physical Solaris x86-64 PSUs. Oracle WebLogic Server 10.3.6.0.6 and 12.1.1.0.6 PSUs. Critical Patch Update (CPU)The Critical Patch Update program is Oracle's quarterly release of security fixes.The following additional patches were released as part of Oracle's Critical Patch Update program: Oracle JDeveloper 11.1.2.3.0, 11.1.2.4.0 and 12.1.2.0.0 Oracle Outside In Technology 8.4.0 and  8.4.1 Oracle Portal 11.1.1.6.0 Oracle Security Service  11.1.1.6.0, 11.1.1.7.0 and 12.1.2.0.0 Oracle WebCache 11.1.1.6.0 and 11.1.1.7.0 Oracle WebCenter Content 10.1.3.5.1, 11.1.1.6.0, 11.1.1.7.0 and 11.1.1.8.0 Oracle WebServices 10.1.3.5.0 and 11.1.1.6.0 For more information: Master Notes on Fusion Middleware Proactive Patching PSU and CPU October 2013  Availability Document Critical Patch Update Advisory -  October 2013

    Read the article

  • Using ContentProviderOperation to update and insert contacts

    - by Bogus
    Hello, I faced the problem updating/insertng contacts on Android 2.0+. There is no problem to insert a new contact when phone book is empty but when I did it 2nd time some fileds like TEL, EMAIL are doubled and tripped etc. but N, FN, ORG are ok (one copy). After getting and advice of other member this forum I updated a contact first and then ContentProviderResult[] returned uri's with null then I do an insert action and it went ok but after that I made an update and all contacts are aggregated into one - i got 1 contact insted 3 which existed in phone book. This one was damaged, the contact fields are randomly built. I set Google account. Code: ArrayList<ContentProviderOperation> ops = new ArrayList<ContentProviderOperation>(); ops.add(ContentProviderOperation.newUpdate(ContactsContract.RawContacts.CONTENT_URI) .withValue(RawContacts.AGGREGATION_MODE, RawContacts.AGGREGATION_MODE_DISABLED) .withValue(ContactsContract.RawContacts.ACCOUNT_TYPE, accountType) .withValue(ContactsContract.RawContacts.ACCOUNT_NAME, accountName) .build()); // add name ContentProviderOperation.Builder builder = ContentProviderOperation.newUpdate(ContactsContract.Data.CONTENT_URI); builder.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0); builder.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.StructuredName.CONTENT_ITEM_TYPE); builder.withValue(ContactsContract.CommonDataKinds.StructuredName.PHONETIC_FAMILY_NAME, name); // phones ContentProviderOperation.Builder builder = ContentProviderOperation.newUpdate(ContactsContract.Data.CONTENT_URI); builder.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0); builder.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Phone.CONTENT_ITEM_TYPE); builder.withValue(ContactsContract.CommonDataKinds.Phone.NUMBER, phoneValue); builder.withValue(ContactsContract.CommonDataKinds.Phone.TYPE, phoneType); builder.withValue(ContactsContract.CommonDataKinds.Phone.LABEL, phoneLabel); ops.add(builder.build()); // emails ... // orgs ... try { ContentProviderResult[] result = mContentResolver.applyBatch(ContactsContract.AUTHORITY, ops); } } catch (Exception e) { Log.e(LOG_TAG, "Exception while contact updating: " + e.getMessage()); } What is wrong in this solution ? How does work aggregation engine ? I will be glad for help. Bogus

    Read the article

  • OLAP Web Visualization and Reporting Recommendations

    - by Gok Demir
    I am preparing an offer for a customer. They proide weekly data to different organizations. There is huge amount data suits OLAP that needed to be visualized with charts and pivot tables on web and custom reports will be built by non-it persons (an easy gui). They will enter a date range, location which data columns to be included and generate report and optionally export the data to Excel. They currently prepare reports with MS Excel with Pivot Tables and but they need a better online tool now to show data to their customers. Tables are huge and need of drill-down functionality. My current knowledge Spring, Flex, MySql, Linux. I have some knowledge of PostgreSQL and MSSQL and Windows. What is the easiest way of doing this project. Do you think that SSRP (haven't tried yet) and ASP.NET better suits for this kind of job. Actually I prefer open source solutions. Flex have OLAP Data Grid control which do aggregation on client side. JasperServer seems promising but it seems I need enterprise version (multiple organizations and ad hoc queries). What about Modrian + Flex + PostgreSQL solution? Any previous experience will be appreciated. Yes I am confused with options.

    Read the article

  • Entire Table is pushed to the next page when rendering a SSRS 2005 Report (as .pdf) in SSRS 2008

    - by Pwninstein
    I have a SSRS 2005 report that I'm rendering in SSRS 2008 as a .pdf. The report contains (among other things) a table that's very simple: header row, details, no footer, no aggregation, no grouping, keep together = false, pageBreakAtStart = false, pageBreakAtEnd = false, repeatHeaderOnNewPage = true. I resized the table to be much narrower than the body of the report just to be sure it wasn't extending beyond the bounds of the report, pushing everything down. But, no matter what I try, if some of the detail rows in that table would need to be pushed to the next page, then the ENTIRE TABLE is pushed to the next page, not just the extra rows. So my question is: Is there a workaround for this problem, is this a known issue, or is it even possible to get this 2005 report to render properly in 2008? NOTE: this is related to a question that I previously asked here, and is based on this MSDN forum post started by a coworker. This question is not the same as my previous question, as I'd like to see things work properly in with a 2005 report. If it's not possible, that would be good to know, as it would indicate that we need to upgrade one of our servers to SQL 2008. Thanks!

    Read the article

  • Database warehouse design: fact tables and dimension tables

    - by morpheous
    I am building a poor man's data warehouse using a RDBMS. I have identified the key 'attributes' to be recorded as: sex (true/false) demographic classification (A, B, C etc) place of birth date of birth weight (recorded daily): The fact that is being recorded My requirements are to be able to run 'OLAP' queries that allow me to: 'slice and dice' 'drill up/down' the data and generally, be able to view the data from different perspectives After reading up on this topic area, the general consensus seems to be that this is best implemented using dimension tables rather than normalized tables. Assuming that this assertion is true (i.e. the solution is best implemented using fact and dimension tables), I would like to seek some help in the design of these tables. 'Natural' (or obvious) dimensions are: Date dimension Geographical location Which have hierarchical attributes. However, I am struggling with how to model the following fields: sex (true/false) demographic classification (A, B, C etc) The reason I am struggling with these fields is that: They have no obvious hierarchical attributes which will aid aggregation (AFAIA) - which suggest they should be in a fact table They are mostly static or very rarely change - which suggests they should be in a dimension table. Maybe the heuristic I am using above is too crude? I will give some examples on the type of analysis I would like to carryout on the data warehouse - hopefully that will clarify things further. I would like to aggregate and analyze the data by sex and demographic classification - e.g. answer questions like: How does male and female weights compare across different demographic classifications? Which demographic classification (male AND female), show the most increase in weight this quarter. etc. Can anyone clarify whether sex and demographic classification are part of the fact table, or whether they are (as I suspect) dimension tables.? Also assuming they are dimension tables, could someone elaborate on the table structures (i.e. the fields)? The 'obvious' schema: CREATE TABLE sex_type (is_male int); CREATE TABLE demographic_category (id int, name varchar(4)); may not be the correct one.

    Read the article

  • ai: Determining what tests to run to get most useful data

    - by Sai Emrys
    This is for http://cssfingerprint.com I have a system (see about page on site for details) where: I need to output a ranked list, with confidences, of categories that match a particular feature vector the binary feature vectors are a list of site IDs & whether this session detected a hit feature vectors are, for a given categorization, somewhat noisy (sites will decay out of history, and people will visit sites they don't normally visit) categories are a large, non-closed set (user IDs) my total feature space is approximately 50 million items (URLs) for any given test, I can only query approx. 0.2% of that space I can only make the decision of what to query, based on results so far, ~10-30 times, and must do so in <~100ms (though I can take much longer to do post-processing, relevant aggregation, etc) getting the AI's probability ranking of categories based on results so far is mildly expensive; ideally the decision will depend mostly on a few cheap sql queries I have training data that can say authoritatively that any two feature vectors are the same category but not that they are different (people sometimes forget their codes and use new ones, thereby making a new user id) I need an algorithm to determine what features (sites) are most likely to have a high ROI to query (i.e. to better discriminate between plausible-so-far categories [users], and to increase certainty that it's any given one). This needs to take into balance exploitation (test based on prior test data) and exploration (test stuff that's not been tested enough to find out how it performs). There's another question that deals with a priori ranking; this one is specifically about a posteriori ranking based on results gathered so far. Right now, I have little enough data that I can just always test everything that anyone else has ever gotten a hit for, but eventually that won't be the case, at which point this problem will need to be solved. I imagine that this is a fairly standard problem in AI - having a cheap heuristic for what expensive queries to make - but it wasn't covered in my AI class, so I don't actually know whether there's a standard answer. So, relevant reading that's not too math-heavy would be helpful, as well as suggestions for particular algorithms. What's a good way to approach this problem?

    Read the article

  • Link Maven OSGi to Maven NetBeans Platform Project

    - by mxro
    I am using NetBeans 6.9 Beta and I would like to accomplish the following: Set up a project representing the main application using Maven (for instance "Maven Project", "Maven NetBeans Application") Ideally, the project should only contain the necessary libraries to run in Apache Felix (I would like to be able to right-click the project and select "Run in Felix") I do not want that the project contains all the NetBean Platform APIs I would prefer to implement the modules using OSGi. For instance "Maven OSGi Bundle", "Maven NetBeans Module" + OSGi These are the problems, which I have at the moment: The standard Maven archetype ("Maven NetBeans Application") seems always to select all APIs and I have not found a way to deselect APIs - in normal NetBeans Platform Applications that can be accomplished by going to the project properties and deselected the platform modules) - I guess it has something to do with the NetBeans repository (http://bits.netbeans.org/maven2)? Do I have to create another repository? When creating normal "NetBeans Module" with OSGi support, the modules contain both NetBeans Module and OSGi meta data, which is nice. But the "Maven NetBeans Modules" have only NetBeans meta data and the Maven OSGi Bundles have only OSGi meta data). I figured out how to add modules to the project by using project / new and then placing the modules in the Maven project folder. However, I do not quite know yet how I could link to modules from other locations (NetBeans uses Maven modules, which have to be in the same directory as the project?). Below some useful links for Maven + OSGi in NetBeans wiki.netbeans.org/STS_69_Maven_OSGI NetBeans Maven OSGi Test Specification platform.netbeans.org/tutorials/nbm-maven-quickstart.html NetBeans Platform Quick Start Using Maven (6.9) wiki.netbeans.org/MavenBestPractices NetBeans Maven BestPractices maven.apache.org/pom.html#Aggregation Maven Documentation Multi-Module Projects (sorry about the missing protocol but couldn't post the message otherwise)

    Read the article

  • Database warehoue design: fact tables and dimension tables

    - by morpheous
    I am building a poor man's data warehouse using a RDBMS. I have identified the key 'attributes' to be recorded as: sex (true/false) demographic classification (A, B, C etc) place of birth date of birth weight (recorded daily): The fact that is being recorded My requirements are to be able to run 'OLAP' queries that allow me to: 'slice and dice' 'drill up/down' the data and generally, be able to view the data from different perspectives After reading up on this topic area, the general consensus seems to be that this is best implemented using dimension tables rather than normalized tables. Assuming that this assertion is true (i.e. the solution is best implemented using fact and dimension tables), I would like to see some help in the design of these tables. 'Natural' (or obvious) dimensions are: Date dimension Geographical location Which have hierarchical attributes. However, I am struggling with how to model the following fields: sex (true/false) demographic classification (A, B, C etc) The reason I am struggling with these fields is that: They have no obvious hierarchical attributes which will aid aggregation (AFAIA) - which suggest they should be in a fact table They are mostly static or very rarely change - which suggests they should be in a dimension table. Maybe the heuristic I am using above is too crude? I will give some examples on the type of analysis I would like to carryout on the data warehouse - hopefully that will clarify things further. I would like to aggregate and analyze the data by sex and demographic classification - e.g. answer questions like: How does male and female weights compare across different demographic classifications? Which demographic classification (male AND female), show the most increase in weight this quarter. etc. Can anyone clarify whether sex and demographic classification are part of the fact table, or whether they are (as I suspect) dimension tables.? Also assuming they are dimension tables, could someone elaborate on the table structures (i.e. the fields)? The 'obvious' schema: CREATE TABLE sex_type (is_male int); CREATE TABLE demographic_category (id int, name varchar(4)); may not be the correct one.

    Read the article

  • Entity Framework query

    - by carter-boater
    Hi all, I have a piece of code that I don't know how to improve it. I have two entities: EntityP and EntityC. EntityP is the parent of EntityC. It is 1 to many relationship. EntityP has a property depending on a property of all its attached EntityC. I need to load a list of EntityP with the property set correctly. So I wrote a piece of code to get the EntityP List first.It's called entityP_List. Then as I wrote below, I loop through the entityP_List and for each of them, I query the database with a "any" function which will eventually be translated to "NOT EXIST" sql query. The reason I use this is that I don't want to load all the attached entityC from database to memory, because I only need the aggregation value of their property. But the problem here is, the looping will query the databae many times, for each EntityP! So I am wondering if anybody can help me improve the code to query the database only once to get all the EntityP.IsAll_C_Complete set, without load EntityC to memory. foreach(EntityP p in entityP_List) { isAnyNotComoplete = entities.entityC.Any(c => c.IsComplete==false && c.parent.ID == p.ID); p.IsAll_C_Complete = !isAnyNotComoplete; } Thank you very much!

    Read the article

  • Re-using aggregate level formulas in SQL - any good tactics?

    - by Cade Roux
    Imagine this case, but with a lot more component buckets and a lot more intermediates and outputs. Many of the intermediates are calculated at the detail level, but a few things are calculated at the aggregate level: DECLARE @Profitability AS TABLE ( Cust INT NOT NULL ,Category VARCHAR(10) NOT NULL ,Income DECIMAL(10, 2) NOT NULL ,Expense DECIMAL(10, 2) NOT NULL ) ; INSERT INTO @Profitability VALUES ( 1, 'Software', 100, 50 ) ; INSERT INTO @Profitability VALUES ( 2, 'Software', 100, 20 ) ; INSERT INTO @Profitability VALUES ( 3, 'Software', 100, 60 ) ; INSERT INTO @Profitability VALUES ( 4, 'Software', 500, 400 ) ; INSERT INTO @Profitability VALUES ( 5, 'Hardware', 1000, 550 ) ; INSERT INTO @Profitability VALUES ( 6, 'Hardware', 1000, 250 ) ; INSERT INTO @Profitability VALUES ( 7, 'Hardware', 1000, 700 ) ; INSERT INTO @Profitability VALUES ( 8, 'Hardware', 5000, 4500 ) ; SELECT Cust ,Profit = SUM(Income - Expense) ,Margin = SUM(Income - Expense) / SUM(Income) FROM @Profitability GROUP BY Cust SELECT Category ,Profit = SUM(Income - Expense) ,Margin = SUM(Income - Expense) / SUM(Income) FROM @Profitability GROUP BY Category SELECT Profit = SUM(Income - Expense) ,Margin = SUM(Income - Expense) / SUM(Income) FROM @Profitability Notice how the same formulae have to be used at the different aggregation levels. This results in code duplication. I have thought of using UDFs (either scalar or table valued with an OUTER APPLY, since many of the final results may share intermediates which have to be calculated at the aggregate level), but in my experience the scalar and multi-statement table-valued UDFs perform very poorly. Also thought about using more dynamic SQL and applying the formulas by name, basically. Any other tricks, techniques or tactics to keeping these kinds of formulae which need to be applied at different levels in sync and/or organized?

    Read the article

  • Does query plan optimizer works well with joined/filtered table-valued functions?

    - by smoothdeveloper
    In SQLSERVER 2005, I'm using table-valued function as a convenient way to perform arbitrary aggregation on subset data from large table (passing date range or such parameters). I'm using theses inside larger queries as joined computations and I'm wondering if the query plan optimizer work well with them in every condition or if I'm better to unnest such computation in my larger queries. Does query plan optimizer unnest table-valued functions if it make sense? If it doesn't, what do you recommend to avoid code duplication that would occur by manually unnesting them? If it does, how do you identify that from the execution plan? code sample: create table dbo.customers ( [key] uniqueidentifier , constraint pk_dbo_customers primary key ([key]) ) go /* assume large amount of data */ create table dbo.point_of_sales ( [key] uniqueidentifier , customer_key uniqueidentifier , constraint pk_dbo_point_of_sales primary key ([key]) ) go create table dbo.product_ranges ( [key] uniqueidentifier , constraint pk_dbo_product_ranges primary key ([key]) ) go create table dbo.products ( [key] uniqueidentifier , product_range_key uniqueidentifier , release_date datetime , constraint pk_dbo_products primary key ([key]) , constraint fk_dbo_products_product_range_key foreign key (product_range_key) references dbo.product_ranges ([key]) ) go . /* assume large amount of data */ create table dbo.sales_history ( [key] uniqueidentifier , product_key uniqueidentifier , point_of_sale_key uniqueidentifier , accounting_date datetime , amount money , quantity int , constraint pk_dbo_sales_history primary key ([key]) , constraint fk_dbo_sales_history_product_key foreign key (product_key) references dbo.products ([key]) , constraint fk_dbo_sales_history_point_of_sale_key foreign key (point_of_sale_key) references dbo.point_of_sales ([key]) ) go create function dbo.f_sales_history_..snip.._date_range ( @accountingdatelowerbound datetime, @accountingdateupperbound datetime ) returns table as return ( select pos.customer_key , sh.product_key , sum(sh.amount) amount , sum(sh.quantity) quantity from dbo.point_of_sales pos inner join dbo.sales_history sh on sh.point_of_sale_key = pos.[key] where sh.accounting_date between @accountingdatelowerbound and @accountingdateupperbound group by pos.customer_key , sh.product_key ) go -- TODO: insert some data -- this is a table containing a selection of product ranges declare @selectedproductranges table([key] uniqueidentifier) -- this is a table containing a selection of customers declare @selectedcustomers table([key] uniqueidentifier) declare @low datetime , @up datetime -- TODO: set top query parameters . select saleshistory.customer_key , saleshistory.product_key , saleshistory.amount , saleshistory.quantity from dbo.products p inner join @selectedproductranges productrangeselection on p.product_range_key = productrangeselection.[key] inner join @selectedcustomers customerselection on 1 = 1 inner join dbo.f_sales_history_..snip.._date_range(@low, @up) saleshistory on saleshistory.product_key = p.[key] and saleshistory.customer_key = customerselection.[key] I hope the sample makes sense. Much thanks for your help!

    Read the article

  • MySQL table data transformation -- how can I dis-aggregate MySQL time data?

    - by lighthouse65
    We are coding for a MySQL data warehousing application that stores descriptive data (User ID, Work ID, Machine ID, Start and End Time columns in the first table below) associated with time and production quantity data (Output and Time columns in the first table below) upon which aggregate (SUM, COUNT, AVG) functions are applied. We now wish to dis-aggregate time data for another type of analysis. Our current data table design: +---------+---------+------------+---------------------+---------------------+--------+------+ | User ID | Work ID | Machine ID | Event Start Time | Event End Time | Output | Time | +---------+---------+------------+---------------------+---------------------+--------+------+ | 080025 | ABC123 | M01 | 2008-01-24 16:19:15 | 2008-01-24 16:34:45 | 2120 | 930 | +---------+---------+------------+---------------------+---------------------+--------+------+ Reprocessing dis-aggregation that we would like to do would be to transform table content based on a granularity of minutes, rather than the current production event ("Event Start Time" and "Event End Time") granularity. The resulting reprocessing of existing table rows would look like: +---------+---------+------------+---------------------+--------+ | User ID | Work ID | Machine ID | Production Minute | Output | +---------+---------+------------+---------------------+--------+ | 080025 | ABC123 | M01 | 2010-01-24 16:19 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:20 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:21 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:22 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:23 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:24 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:25 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:26 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:27 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:28 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:29 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:30 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:31 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:22 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:33 | 133 | | 080025 | ABC123 | M01 | 2010-01-24 16:34 | 133 | +---------+---------+------------+---------------------+--------+ So the reprocessing would take an existing row of data created at the granularity of production event and modify the granularity to minutes, eliminating redundant (Event End Time, Time) columns while doing so. It assumes a constant rate of production and divides output by the difference in minutes plus one to populate the new table's Output column. I know this can be done in code...but can it be done entirely in a MySQL insert statement (or otherwise entirely in MySQL)? I am thinking of a INSERT ... INTO construction but keep getting stuck. An additional complexity is that there are hundreds of machines to include in the operation so there will be multiple rows (one for each machine) for each minute of the day. Any ideas would be much appreciated. Thanks.

    Read the article

  • java class creation dynamically and make it accessible across the network different jvms i.e. serial

    - by inj.rav
    Hi. I have a requirement of creating java classes dynamically and make it accessible different jvms across the network. I tried to use reflection and javassist tool,but nothing worked. Let me explain the scenario we are using Coherence distributed cache. It has a power of doing aggregation/filtering in parallel across the cluster. For example if a class has [dynamic class] has amount variable and getAmount/setAmount methods. Then if we execute COHERENCE queries, it will start process in parallel across the cluster. I tried to create classes at run time by using javassist and reflection. I am able to access it from single JVM, but when I tried to access the same class from other jvm [through coherence cluster]. I am getting exception of class not found [as remote jvm is not having idea of this class].I can over come this by creating same class dynamically on remote jvm also and access the methods. But coherence in built methods/functions are not able to find the class. could some one help me on this matter

    Read the article

  • Examples of monoids/semigroups in programming

    - by jkff
    It is well-known that monoids are stunningly ubiquitous in programing. They are so ubiquitous and so useful that I, as a 'hobby project', am working on a system that is completely based on their properties (distributed data aggregation). To make the system useful I need useful monoids :) I already know of these: Numeric or matrix sum Numeric or matrix product Minimum or maximum under a total order with a top or bottom element (more generally, join or meet in a bounded lattice, or even more generally, product or coproduct in a category) Set union Map union where conflicting values are joined using a monoid Intersection of subsets of a finite set (or just set intersection if we speak about semigroups) Intersection of maps with a bounded key domain (same here) Merge of sorted sequences, perhaps with joining key-equal values in a different monoid/semigroup Bounded merge of sorted lists (same as above, but we take the top N of the result) Cartesian product of two monoids or semigroups List concatenation Endomorphism composition. Now, let us define a quasi-property of an operation as a property that holds up to an equivalence relation. For example, list concatenation is quasi-commutative if we consider lists of equal length or with identical contents up to permutation to be equivalent. Here are some quasi-monoids and quasi-commutative monoids and semigroups: Any (a+b = a or b, if we consider all elements of the carrier set to be equivalent) Any satisfying predicate (a+b = the one of a and b that is non-null and satisfies some predicate P, if none does then null; if we consider all elements satisfying P equivalent) Bounded mixture of random samples (xs+ys = a random sample of size N from the concatenation of xs and ys; if we consider any two samples with the same distribution as the whole dataset to be equivalent) Bounded mixture of weighted random samples Which others do exist?

    Read the article

  • Patterns for non-layered applications

    - by Paul Stovell
    In Patterns of Enterprise Application Architecture, Martin Fowler writes: This book is thus about how you decompose an enterprise application into layers and how those layers work together. Most nontrivial enterprise applications use a layered architecture of some form, but in some situations other approaches, such as pipes and filters, are valuable. I don't go into those situations, focussing instead on the context of a layered architecture because it's the most widely useful. What patterns exist for building non-layered applications/parts of an application? Take a statistical modelling engine for a financial institution. There might be a layer for data access, but I expect that most of the code would be in a single layer. Would you still expect to see Gang of Four patterns in such a layer? How about a domain model? Would you use OO at all, or would it be purely functional? The quote mentions pipes and filters as alternate models to layers. I can easily imagine a such an engine using pipes as a way to break down the data processing. What other patterns exist? Are there common patterns for areas like task scheduling, results aggregation, or work distribution? What are some alternatives to MapReduce?

    Read the article

  • Count rows against to SQL server (2005) table?

    - by David.Chu.ca
    I have a simple question with two options to get count of rows in a SQL server (2005). I am using VS 2005. There are two options to get the count: SELECT id FROM Table1 WHERE dt >= startDt AND dt < endDt;; I get a list of ids from above call in cache and then I get count by List.Count. Another option is SELECT COUNT(*) FROM Table1 WHERE dt >= startDt AND dt < endDt; The above call will get the count directly. The issue is that I had several cases of exceptions with the second method: timeout. What I found is that the table1 is too big with millions of data. When I used the first option, it seems OK. I am confused by the fact that Count() takes more time than getting all the rows(is that true?). Not sure if the aggregation call with Count() would cause SQL server to create temporary table or cache on server side and it would result in slow performance when table is too big? I am not sure what is the best way to get the count?

    Read the article

< Previous Page | 5 6 7 8 9 10 11 12  | Next Page >