Using information to navigate uncertain times – a case study

by Bill 12. February 2010 17:48

One of the most valuable uses of good business information is to help navigate a business in times of uncertainty and change. Many of our clients are doing exactly that and seeing huge benefits from having a data mart of solid, reliable and up to date information about their business performance.

A little while ago I had lunch with one of our Retail clients and they provided a great example. Their bank had been hit hard by the financial services crash and decided that over a period of months they would scale back our client’s line of credit for working capital. Rather than panicking our client went back to a new multi-dimensional analysis that they’d been using to analyze their inventory levels. They realized they could more closely manage inventory now that they had more accurate and timely seasonal sales information. Over a period of about nine months they reduced their inventory levels and working capital by almost 50% and were able to easily accommodate a smaller line of credit. In fact, they are seeing other operational benefits from not storing and managing so much inventory. Since their customer’s economic situation has also been dramatically changed they are buying different products. Having less on-hand inventory has kept our client better positioned to respond to these product buying shifts and able to move quickly into new higher-margin products rather than having to heavily discount large inventories of older products.

Having good business information delivered in a means that facilitates analysis allows our clients to see leading indicators of change well in advance. It also allows them to explore multiple possible solutions and anticipate how these solutions will impact their business. These are essential capabilities in these times of economic change!

Tags:

Virtual Applications in Windows 7 Virtual PC

by Ellen 9. February 2010 19:45

Recently, I inadvertently discovered a useful feature of Windows 7 Virtual PC that I'd like to share, just in case there are folks out there who haven't experienced this particular functionality.

On my host laptop (a Windows 7 box), I've installed SQL Server 2008 and Visual Studio 2008, although my previous XP box hosted SQL Server/Visual Studio 2005. I use Visio for Enterprise Architects as my data modeling tool and as I discovered during the move to the new Windows 7 laptop, Visio for Enterprise Architects requires Visual Studio 2005 in order to install properly - it won't play with VS2008.

Okay, fine.

This was annoying, but not catastrophic. Since most of our clients are still on SQL Server 2005, I have a Virtual PC with SQL Server/Visual Studio 2005 installed. I installed Visio on this VPC, grumbling since I knew I'd have to launch this VPC whenever I wanted to do any data modeling.

Or so I thought.

One day, I experienced a momentary lapse of consciousness and double-clicked a Visio (.vsd) file on my host machine. Just as I was cursing myself for doing something so dopey, the Virtual PC credentials login for my SQL Server/VS2005 VPC popped up. I entered my credentials, and the VPC started to launch, except instead of opening the VPC window, it launched Visio as a "virtual application" within the context of the host machine.

How cool is that?

I was able to access files from inside the VPC as well as the host machine from the Visio application menus.  The host machine treated it as a remote connection to the VPC through the local network.

I've poked around with this a little more since the first time it surprised me and have discovered a few more things:

  • My VPC is a member of a domain.  If no network is available (even a network that is not the domain network), the remote application cannot launch.  For instance, I was in an airport recently with no internet/network connectivity and was unable to start the virtual application.
  • For this domain-enabled VPC, it doesn’t seem to matter if Integration Features are enabled or not.  The remote application can still launch.
  • Occasionally, Internet Explorer tries to take over and pretend that it’s the application of choice to host .vsd file.  You have to take a firm stand on this – right click the .vsd file and choose Open With.  On my host laptop, the option to open with the remote Visio program appears in the popup menu (SQLServer2005 is the name of my VPC):

open_with_visio_remote

I haven’t tried this with a VPC that isn’t a member of a domain, so I’m not sure what the behavior for that would be like.  I’ll have to give that a try for the next blog post!

Tags: ,

Linking TFS Changesets to Work Items After Code Check In

by Serena 22. December 2009 01:41

One of the helpful features of Team Foundation Server (TFS) is the ability to associate work items with the code that was changed or created for them during the check in process.  In the event that you forget to make this association during code check in, TFS allows you to link a code changeset to a work item after the fact.

links_tab 

After opening the work item you want to associate with a changeset, select the Links tab and click on the Add button.

add_link_cropped

 

Changeset is one of the options you can select in the Link type drop down box.

After you select Changeset, click on the Browse button to the right of Changeset text box inside Link details.  

 

find_changeset_cropped 

TFS gives you a few different ways to locate the changeset you want to associate with a work item, including a specific file contained within the changeset, the user who checked the code in, the changeset number, and a date/time range.  

After you click Find, TFS presents you with a list of changesets meeting your criteria.  If you would like to review the files in a changeset before clicking OK, there is a Details button at the bottom left of the dialog box that will retrieve a list of the files in the changeset.

Tags:

Software Development

Simplifying date logic in IBM Cognos 8 – Part 2

by Bill 14. December 2009 18:35

In Part 1 of this post we described a dilemma in reporting environments where report developers need to do complex date calculations. These frequently involve relative dates – periods that are relative to the current date such as the last 13 months or the currently closed financial periods.

The most common approaches we see to these issues are:

  1. Add more prompting to the report so that the user has the burden of defining what the last x months are or what financial periods are closed. In addition to being a burden this is subject to misinterpretation. For example, what if the person executing the report doesn’t know what financial periods are closed? Do they guess?
  2. Put complicated date functions and logic into each report. This usually performs badly – often the logic cannot be evaluated completely within the database engine and requires moving large amounts of data between the database server and the IBM Cognos application server. It also runs the risk that different report developers will use slightly different sets of complicated date functions to achieve what they believe are the same thing. For example, one developer may calculate how many working days (not Saturday, Sunday or Holidays) have elapsed already in the current month and think that is the same thing as another developer who simply calculated the total elapsed days in the month.

The first part of this article illustrated how we can unload the heavy lifting of building date related calculations to our overnight ETL process. This part will discuss using a very cool feature of IBM Cognos 8 called Parameter Maps to expose that logic for report developers.

A parameter map is a way to expose a scalar (single item) data value. This is a subtle point but if you think about it – all queries return sets. Even if only one row is returned – it is a set. Many useful functions and calculations cannot consume a set (even if it only has one column and one row) and they must have a scalar value.

One of the great features of IBM Cognos 8 is that Parameter Maps not only allow you to retrieve these scalar values but they allow you to reference them by an key value of your choosing. You are also allowed to define all of this centrally in the Framework Manager metadata and consume these scalar values from Parameter Maps in other centrally defined Prompts and Filters.

To create a new Parameter Map from within Framework Manager simply right-click on Parameter Maps and select Create > Parameter Map from the pop-up menu. You are given a choice to create the parameter map manually but that would make it static (which could be useful for some things like your company name!) Select the option, “Base the parameter map on existing Query Items.” instead and give it a name. This will make the Parameter Map be driven by our control table which is dynamic since our ETL process updates it nightly.

Since the control table in Part 1 of this post had columns for Integer, Character, Smalldatetime and Float data types I’d probably name this one something like pmStatus_Integer and expect to create three additional parameter maps – one for each of the data types.

Select the control table that we created in Part 1 and use the Set as Key and Set as Value buttons to make the parameter name the key and the Integer Value the value.

create parameter map

Finish up the Create Parameter Map wizard and you’ve successfully exposed your control values as a parameter map. Next let’s look at how you might use this within Framework Manager.

The first thing that leaps to mind is to use it with Filters. Creating Filters within Framework Manager is a really great technique because they are defined centrally and can be easily reused everywhere in reports, queries, etc. This helps you define that “single version of the truth.” Here is a snapshot from Framework Manager showing a few Filters and Calculations.

Filters and calculations

Let’s look at how the Parameter Map is used in one of these calculations. Let’s look at Last 13 Months as an example.

Filter definition Last 13 Months

What this Filter is doing is filtering the date dimension so that only dates that are greater than the date 13 months ago and less than our current Reporting Effective Date are returned. The parameter map pmStatus_Smalldatetime is being referenced and I’m retrieving specific values via the two keys, “13 months prior to Reporting Effective datetime” and “Reporting Effective Datetime”. Notice that I like to use very descriptive names as keys.

If you actually look at the SQL generated when this Filter is used you’ll see it is efficiently filtering the dimension table (not the fact which may have many more rows) and that all of the filtering is being done in the SQL sent to the database server.

Here is a really simple Query Studio query before I apply the Filter. Notice how many months of data are being returned.

query studio before

Now I simply drag the Last 13 Months filter to my query surface and…

query studio after

Voila! Now I’ve used the power of a control table in my data mart and parameter maps in Framework Manager to deliver a very easy to use and fully dynamic set of filters that can be reused over and over.

Because the underlying control table is dynamic the date 13 months prior to the effective reporting date is being updated every night automatically. So the report developer or end user doesn’t have to do this calculation.

Tags:

Simplifying date logic in IBM Cognos 8 – Part 1

by Bill 14. December 2009 17:23

A common challenge that we see with all of our many IBM Cognos 8 clients is dealing with date logic. The most common forms for this challenge are around relative dates:

  • How can I filter this report to only show the last x months of data?
  • Can we display only the closed financial periods?
  • I want to display the name of the current month in the header of my report

Unfortunately the most common approach to solving these sorts of issues is to either make the person running the report respond to a prompt to determine the needed filtering values or to put very complex date logic into functions in the report. Making the user provide the information is annoying to the user (why can’t this report just know what the last x months are?) and prone to misinterpretation at runtime (e.g. I thought I was supposed to pick the invoice dates that I wanted – not the ship dates!) Putting complex date logic in the reports is difficult to maintain, poorly performing and runs the risk that each report developer may take a slightly different and inconsistent approach to delivering a solution.

Our recommended approach is to have the Extraction, Transformation and Loading (ETL) process do the heavy lifting of date calculations and then leverage a powerful IBM Cognos feature called Parameter Maps to easily expose those calculations for use in reports.

The ETL process is crucial because it runs on a regular time interval (often on a nightly basis) so it can easily deliver calculations that involve relative dates – dates that are relative to now. We create a SQL table in our data mart that is very simple like this:

CREATE TABLE [dbo].[pcon_ctl_status](
    [status_key] [int] IDENTITY(1,1) NOT NULL,
    [status_parameter_name] [varchar](50) NOT NULL,
    [status_parameter_description] [varchar](100) NULL,
    [status_int_value] [int] NULL,
    [status_char_value] [varchar](100) NULL,
    [status_smalldatetime_value] [smalldatetime] NULL,
    [status_float_value] [float] NULL,
CONSTRAINT [pcon_ctl_status_PK] PRIMARY KEY CLUSTERED
(
    [status_key] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

The idea is that we populate this table with one row for every useful scalar value that we can think of – some examples include:

  • The last time our ETL process was executed (so we know how current our data is)
  • The date 13 months before the current date (if we do a lot of rolling 13 month reporting.)
  • Any sort of calculated time period factors (what percentage of the working days in the current month have already elapsed?)

So the data in the resulting table looks something like this:

ctl_status table

 

Every night the ETL process runs and updates the values in the status_xxx_value columns of this table. These are the four columns to the far right in the screen shot above. The most common way for the ETL process to do this is via a simple SQL script that gets executed at the end of the ETL process.

data manager screenshot 

The screen shot above shows some of the update statements executing from within the IBM Cognos DecisionStream ETL tool but we also commonly do this from within Microsoft SQL Server SSIS jobs.

The next post in this series will explain how we now use this data inside our IBM Cognos 8 reporting environment.

Tags:

Business Intelligence | Data Warehousing

PDC 2009 Recap

by Rick Glos 7. December 2009 20:10

We do quite abit of .NET Development here at Perkins Consulting and like to stay on top of current technology, software architecture, and new technology.  I was lucky enough to be able to attend the Microsoft PDC 2009 Conference in Los Angeles, CA.  Below are some notes and details.

Recap

These are notes condensed from 4 days of note taking in OneNote.

Stuff that got announced

Main Theme

3 screens and a cloud  (next 'wave' is Cloud computing [Mainframe (1970's) --> client-server (1980's) --> web (1990's) --> soa (2000's) --> cloud (2010's)]

  • Phone, TV, PC
  • Silverlight being the client mechanism (they are really pushing this to be the main UI mechanism)

Data playing increasingly larger role - Vivek Kundra - Chief Information Officer for the USA live via video feed

  • make information public - for public consumption and transparency of data
  • data.gov - searchable data catalogs

Conference Sessions Attended (11) + 1 pre-conference workshop

Workshop

  1. Developing Microsoft BI Applications - The How and the Why
    1. Note that I struggled with this decision.  I signed up for Getting the Most out of Microsoft Silverlight 3 but having spent the last 9 months on a Silverlight project for one of our clients, I did not think it was going to be that beneficial.  There’s a ton of info on the web for Silverlight.  However, there’s not much for BI.  The Silverlight application I worked on sits on top of a Analysis Services Cube – the API is completely different than just throwing down a Entity Framework model on top of a relational database (which I did as well for some parts of it).  This app has an AdoMdDataReader, CellSet and MDX statements.  I was curious how others approached this area.

Sessions

  1. Data Programming and Modeling for the .NET Developer - SQL Server Modeling Services
  2. Overview of SharePoint 2010 Programmability
  3. Evolving ADO.NET Entity Framework in Microsoft.NET Framework 4 and Beyond
  4. SketchFlow: Prototyping to the Rescue
  5. Microsoft Perspectives on the Future of Programming
  6. Should I Use Silverlight, MVC, or Web Forms for Web User Interface Development?
  7. Networking and Web Services in Silverlight
  8. Advanced Topics for Building Large-Scale Applications with Microsoft Silverlight
  9. Automating the App Lifecycle with Windows Azure
  10. SQL Server Modeling Services: Using Metadata to Drive Application Design, Development and Management
  11. Mastering Microsoft WCF RIA Services

Conference Sessions I wanted to attend, 44.

There was alot of concurrent session angst for me.  At any given time slot, there were 11 sessions and many of those I wanted to attend more than one in that time slot.  Luckily it was all being recorded and available at PDC, http://microsoftpdc.com/.  It will be interesting to see how making this all available online affects conference attendance in the future.

I might add that one of my favorite sessions, ‘Should I Use Silverlight, MVC, or Web Forms for Web User Interface Development?’, was an audience participation session where I got to talk quite alot because out of the mass of people in the session, I was one of the few using Silverlight in a LOB application for one of our clients.  However since it was audience participation, there is no video or audio available – an experience you can only get by being there.

Conclusion

Overall I thought this conference extremely worthwhile.  To be able to see the direction of the software industry and to see how much data is going to play a part in the future.  I think Perkins, having years of experience with Data Warehouse projects, is sitting in a perfect position to take advantage.

Tags:

Software Development

Using SSIS Package Explorer to Eliminate Undisplayed Package Objects

by Ellen 4. December 2009 20:11

Last night I saw upon the stair

A little man who wasn’t there

He wasn’t there again today

Oh, how I wish he’d go away

--from Antigonish by Hughes Mearns

 

Recently, I’ve been assisting a client in moving their data mart databases and SSIS packages to a new server.  Prior to decommissioning the old server, I was comparing package execution results, and came across an anomaly as exposed by our standard audit infrastructure tables.  In reviewing the package execution table, it appeared that several of the packages were being logged more than once (same package guid and execution guid but different execution keys).

package_exec_listing 

I could tell by reviewing the SQL Agent job history that each of these packages only executed once, and the rows in the package execution table were not completely identical – the columns that are normally updated at the end of the execution were null for one of the rows in each duplcate set.  Hmmmmm…. Obviously, something goofy was happening in the package, related to the instantiation and update of the package execution record.

When I opened the packages in question, I saw that the Control Flow pane included the three normal audit infrastructure sequence containers that are part of all our packages.  However, I noticed something.  Two of the infrastructure sequence containers had the numeral one appended to the container name.

 

exposed_sequence_containers

Usually, this means that the SSIS developer has copied and pasted an object with a name that already exists in the package context.  BIDS appends sequential numerals to objects in these cases, to make sure the object names are unique.  I searched the Control Flow pane, but could find no duplicates for the containers.  Sure, it was possible that the developer who created the package had copied from an open package and hadn’t renamed the objects in his destination package, but I had the evidence of the package execution table screaming at me that the code was firing more than once.  Also, I wasn’t able to remove the trailing numerals from the container names without BIDS throwing an error, so SSIS still believed that the containers existed within the package. 

Invisible code.  Oh, yeah.  We hate that.

I was reduced to observing what was right in front of my nose – the Package Explorer tab.  I’m embarrassed to say that in over five years of working with SSIS, I have never looked at that tab before.  However, the Package Explorer tab does not hold a grudge – even though I’d ignored it all this time, it helped me out anyway.

When I expanded the Executables node, I could see that the two infrastructure containers were indeed duplicated.  By right-clicking on the ones that were not displayed on the Control Flow tab (the items without the trailing numeral), I was able to delete them.  I was then able to rename the visible containers, removing the trailing numeral without error.

 

package_explorer_listing

Tags: , ,

Data Warehousing | Business Intelligence

Technoxenophobia

by Ellen 23. November 2009 18:02

 

“Klaatu barada nikto”

--The Day the Earth Stood Still

 

A few weeks ago, I posted a blog entry about the phenomenon I call the Cheshire Data Mart - a data mart that disappears from the perception of the end user whose only interaction with the data is through a presentation tool.

Today I want to talk about the opposite effect - the inclination of the user to distrust the data mart data in all instances when the data does not tie perfectly to the user's program of choice. I call this "technoxenophobia" - the fear of alien technology.alien_flying_saucer

I'm not talking about the need to validate the data mart loads - that's a necessary and understood process (at least from our perspective). I'm talking about the resistance that business users can experience when asked to work with data or tools that are outside their normal comfort zone.

None of our clients, to my knowledge, has ever had an ERP or OLTP system that perfectly matches their business. They're always forced to do some kind of work-around or to store supplemental data in odd corners (cough***Excel***cough) in order to meet their reporting needs. These "data cubbies" are not usually supported by a tight business process or (which would be even more preferable) enforced by the API of the OLTP system. The more manually-maintained and/or distributed these data cubbies are, the more likely it is one or more maintenance steps could be missed.

When we design and build a data mart, we try to incorporate all of these special cases and additional data, so that the data mart actually does align with the business's reporting expectations and requirements.

Since the data mart data is normally distributed to a broader business community via a business intelligence architecture than functionally specific OLTP applications (point-of-sale systems or accounting applications, for instance), any errors or omissions in the data cubby maintenance are exposed in this larger environment, frequently in a much more rapid life cycle than the users responsible for their manual maintenance expect. Additionally, the extended business community may know nothing about the supplemental data manually maintained by users outside their own sphere.

Result? Any presentation of unexpected data is blamed on the data mart, since it's the only new guy in town.

The data mart is the only place where all these disparate parts are brought together. The results can sometimes be startling for the end user, exposing data usage from different parts of the organization that can be either redundant or conflicting.

In my very earliest days in data mart development, I always accepted these accusations at face value and tried to find the errors in my code. I've learned over the years, however, that the first place to check is any source file that is heavily dependent on human intervention. The gradual exposure to the end users of these points of fragility in their own business systems is, in my opinion, one of the cool things about data mart implementation. The business community is given the opportunity to tighten their own procedures by observing the results of those procedures as defined by their own data.

The true secret to a successful data mart invasion is not conquest but self-knowledge and evolution. Not all aliens are hostile, after all.

Tags:

Data Warehousing | Business Intelligence

64-bit SQL Server and iSeries ODBC

by Ellen 12. November 2009 00:31

This week, I started work on a data mart for a client whose source data resides on an AS400. The data access for the source system is handled via the iSeries ODBC driver from IBM. Getting the DSNs set up correctly to work with the client's 64-bit SQL Server installation was a little irritating, so I thought I'd share some lessons learned.

As you may already know, even though the database engine for 64-bit SQL Server is a 64-bit application (with a 64-bit dtexec executable for SSIS packages), the BIDS designer is a 32-bit application. This presents a challenge in package development and deployment when data connections either do not have a 64-bit driver (as in the case of Jet for Excel or Access) or when the 64-bit and 32-bit drivers exist, but require different executable versions (as with ODBC).

In this case, I needed to connect to the AS400 with the iSeries ODBC driver from within SSIS. The client had created the DSN for me and believed he had created a 32-bit DSN (since the driver description within the ODBC Administrator was labeled as 32-bit). However, the DSN did not appear in the DSN selection drop-down in the ODBC connection manager in SSIS.

The problem was that, regardless of what the driver was labeled, the DSN had to be created with the 32-bit ODBC Administrator in order for it to exist as a 32-bit DSN and become accessible to SSIS. Furthermore, in order for the package to run both in debug mode from within the designer (32-bit) and as a scheduled job in SQL Server Agent using the Integration Services step type (64-bit), you have to create DSNs with the identical name in both the 32-bit ODBC Administrator and the 64-bit Administrator.

So, in the Program menu for the iSeries access, you'll see a 64-bit and 32-bit ODBC Administrator:

 

iseries_menu

You need to create identical system DSNs using both Administration modules.  The DSN defined in the 32-bit Administrator will be exposed and utilized from with BIDS; the DSN defined in the 64-bit Administrator will be utilized by SQL Server Agent and 64-bit dtexec.

To add a little more excitement to your day, SSIS has no ODBC source adapter for the Data Flow task, and in SQL Server 2008, the Data Reader source adapter has been retired and replaced with the ADO.NET source adapter.  In order to use the iSeries DSN, you need to create your connection manager as an ADO.NET provider (.Net Providers\Odbc Data Provider) rather than as an ODBC connection manager type.

Tags: , , , ,

Business Intelligence | Data Warehousing

How Old Is That in Techno-Years?

by Ellen 4. November 2009 16:56

"You have to be very, very careful with this, because it's thousands and thousands of years old…from when Mommy was little."

--Hana, age 5

 

After my daughter uttered the fateful words above, my husband took entirely too much pleasure in calling me at work and telling me about it. After all, he's not that much younger than I am. Considering that Hana is now a junior at the University of Oregon, I suspect that another three or four millennia have been added to my age from her perspective.

Nothing makes you feel older than comments from your children.

Or so I thought.

This week, 71.43% of PCon is attending the annual PASS (Professional Association for SQL Server) conference in Seattle. Yesterday was the first day of the conference, and over dinner, Toni, Kathy, Eric, Serena and I discussed some of the sessions we'd attended. No life-changing sessions yet (something I always hope for at PASS), but we'd each seen sessions on features upcoming in the next SQL Server release. Yay! New stuff! We love it!

However, Eric also related a conversation he had with a young man in the elevator at his hotel. They discovered they were both in town to attend a tech conference. The other fellow asked if Eric was attending the iPhone conference on the second floor. Eric replied that he was here for PASS. The response:

"Oh, you're an old school geek."

It was very lowering.

According to this whippersnapper (I failed to ask Eric if the youngster was still wearing his Cub Scout uniform, although it wouldn't surprise me), PASS is the technological equivalent of AARP.

Well.

I refuse to be dismayed by this. I'm looking forward on my session today on SSIS design patterns, a Microsoft chalk talk on metadata-driven ETL and the session tomorrow on using agile development techniques in SSIS.

However, when I get back to Portland next week, I'm going to have to discuss my job title with Bill. In another month or so, "Senior Consultant" might be construed as a politically incorrect term in my case. I'll draw the line at "Consultant Emeritus" though. Even at my advanced technological age, I still look forward to learning cool new stuff – even if I have to look through my bifocals to do it.

Tags:

Powered by BlogEngine.NET 1.5.0.7
Theme by Perkins Consulting Content Copyright 2009 Perkins Consulting, LLC All rights reserved.