T-SQL End of Previous Week, Month, Quarter, Year

I recently had a requirement to capture periodic snapshots of an incident/activity type OLTP table into a fact table. I already had an accumulating snapshot type fact table where I was capturing the same facts on a daily basis. But, in addition to this, the client wanted the ability to perform period over period (weekly, monthly, quarterly, yearly) reporting.

My approach was simple enough and in case anyone was thinking about an overly complicated approach I figured I'd share . . .

After my daily load of the fact table I check to see if snapshot data exists for the prior week, month, quarter, and year. If any of these snapshots do not exist I simply do an INSERT selecting all data from the accumulating snapshot table and adding the As-Of date along with the level of periodicity (W,M,Q,Y).

I used 4 T-SQL if statements to determine if the pior period data exists and if it doesn't I call the INSERT.

Below are the statements used to capture the pior period as-of dates in an INT key format. The same format you would get converting a date to a VARCHAR using the 112 style (e.g. 20091231 for December 31st, 2009). I'll leave it up to you to apply this logic to your own INSERT statements.

--Previous end of month
DECLARE @EndOfWeekKey AS INT
SET @EndOfWeekKey = (SELECT CAST(CONVERT(VARCHAR,DATEADD(wk,-1,DATEADD(dd,-(DATEPART(dw, GETDATE()) - 7), GETDATE())),112) AS INT))

--Previous end of month
DECLARE @EndOfMonthKey AS INT
SET @EndOfMonthKey = (SELECT CAST(CONVERT(VARCHAR,DATEADD(d,-1,CAST(CAST(MONTH(GETDATE()) AS VARCHAR) + '/1/' + CAST(YEAR(GETDATE()) AS VARCHAR) AS DATETIME)),112) AS INT))

--Previous end of quarter
DECLARE @EndOfQuarterKey AS INT
SET @EndOfQuarterKey = (
SELECT
    CASE
        WHEN MONTH(GETDATE()) IN (1,2,3) THEN CAST(YEAR(GETDATE()) - 1 AS VARCHAR) + '1231'
        WHEN MONTH(GETDATE()) IN (4,5,6) THEN CAST(YEAR(GETDATE()) AS VARCHAR) + '0331'
        WHEN MONTH(GETDATE()) IN (7,8,9) THEN CAST(YEAR(GETDATE()) AS VARCHAR) + '0630'
        WHEN MONTH(GETDATE()) IN (10,11,12) THEN CAST(YEAR(GETDATE()) AS VARCHAR) + '0930'
END)

--Previous end of year
DECLARE @EndOfYearKey AS INT
SET @EndOfYearKey = (SELECT CAST(CAST(YEAR(GETDATE()) -1 AS VARCHAR) + '1231' AS INT))

Cognos 8.3 -- Framework Manager -- Model objects and shortcuts

I was recently tasked with creating a relational model in Cognos Framework Manager and got a little confused when deciding what kind of objects to use. I found myself asking if I should use a Query Subject or a Shortcut. Furthermore, it seemed there had been some changes to the Shortcut functionality since I had last used Cognos. I found the following that made it rather clear which object(s) I should use when creating my model.

Taken from the Cognos documentation . . .

The main difference between model objects and shortcuts is that model objects give you the freedom to include or exclude items and to rename them. You may choose to use model objects instead of shortcuts if you need to limit the query items included or to change the names of items.

Shortcuts are less flexible from a presentation perspective than model objects, but they require much less maintenance because they are automatically updated when the target object is updated. If maintenance is a key concern and there is no need to customize the appearance of the query subject, use shortcuts.

Framework Manager has two types of shortcuts:
● regular shortcuts, which are a simple reference to the target object.

● alias shortcuts, which behave as if they were a copy of the original object with completely independent behavior.

Alias shortcuts are available only for query subjects and dimensions. Regular shortcuts are typically used as conformed dimensions with star schema groups, creating multiple references with the exact same name and appearance in multiple places. Alias shortcuts are typically used in role-playing dimensions or shared tables.

Being able to specify the behavior of shortcuts is new to Cognos 8.3. When you open a model from a previous release, the Shortcut Processing governor is set to Automatic. When Automatic is used, shortcuts work the same as in previous releases, that is, a shortcut that exists in the same folder as its target behaves as an alias, or independent instance, whereas a shortcut existing elsewhere in the model behaves as a reference to the original. To take advantage of the Treat As property, it is recommended that you verify the model and, when repairing, change the governor to Explicit. The repair operation changes all shortcuts to the correct value from the Treat As property based on the rules followed by the Automatic setting, this means that there should be no change in behavior of your model unless you choose to make one or more changes to the Treat As properties of your shortcuts.

When you create a new model, the Shortcut Processing governor is always set to Explicit. When the governor is set to Explicit, the shortcut behavior is taken from the Treat As property and you have complete control over how shortcuts behave without being concerned about where in the model they are located.

SSIS Error Codes (Hresults)

Ever get an Hresult error code and have no idea what it meant or how to figure out what it's related to?

The SSIS run time engine and the data flow engine are written in native code. Therefore, many errors are returned as Hresults. In case your unfamiliar with what an Hresult is, below is a Wikipedia definition for Hresult.

In the field of computer programming, the HRESULT is a data type used in many types of Microsoft technology. HRESULTs are used as function parameters and return values to describe errors and warnings in a program.

An HRESULT value has 32 bits divided into three fields: a severity code, a facility code, and an error code. The severity code indicates whether the return value represents information, warning, or error. The facility code identifies the area of the system responsible for the error. The error code is a unique number that is assigned to represent the exception. Each exception is mapped to a distinct HRESULT.

Now that you know what Hresults are, you can look up their mappings (for SSIS) using the link below.
http://msdn.microsoft.com/en-us/library/ms345164.aspx

SQL Server 2005/2008 Maximum Server Memory - What should it be?

While questions like this don't have black or white answers there is some simple guidance here . . .

You probably want to change it from the default configuration of 2,147,483,647 MB which is over 2 terabytes. I believe the idea here is to allow SQL Server to use as much memory as it wants. That's all well and good until it begins to starve the OS of memory which will in turn bring SQL Server to its knees.

There are implications related to OS editions, SQL Server editions, 32 versus 64 bit, AWE, etc. I won't get into that here but I will say that it's a good idea to set a maximum value and leave at least a half of a GB and if possible 1+ GB for the operating system.

For example, if you're on a budget and running 32 bit Windows Server 2003 Enterprise with 4GB of RAM (the max ram allowed on this OS) I would suggest setting this value to somewhere between 3,072 – 3,584 MB (3 - 3.5 GB).

Here are some recommendations from Glenn Berry's blog.

Cognos 8.3 and SSAS 2005 - Not So Good

It's a rare day that I concede defeat but today was one of those rare days. I threw in the towel on migrating existing Cognos Report Studio reports from a Framework Manager model that used a SQL Server relational data source to a new model based on a SQL Server Analysis Services 2005 data source.

There were two major reasons:

1. The inability to make any modifications to a Framework Manager model using objects from the SSAS cubes. This includes the creation of filters based on an SSAS cube.

2. Performance issues related to this architecture.

Let me explain why I am using a hybrid (Microsoft/Cognos) architecture . . .

A couple of years ago I took on a job to create a reporting solution for a large CRM implementation at a financial services firm. What I found was that the CRM solution that they were having customized had a plug-in module for Cognos-based "analytics". What this meant was that they had a Cognos model built on top of their OLTP system and some reports to go with it. Additionally, they provided an authentication module that allowed for users that were authenticated on the CRM system (using a combination of AD and forms based login) to have their credentials passed on to this CRM application. This was important because some of the users were on a different domain and were not able to authenticate using AD. I tried to sell them on creating a custom authentication provider for SSRS and scrap Cognos. Not because I refuse to work with non-Microsoft technology but because Cognos didn't seem like the right solution for the problem (I won't get into that here).

I couldn't sell them so I proceeded to build Cognos reports using Report Studio, against a Framework Manager model that was on top of the live OLTP database. The OLTP schema was complex, the model was buggy and full of gaps. It was not pretty. In the end we had 30+ reports. Some of which were very complex and written against a very complex and cumbersome model. The queries being produced by Cognos were unbelievably complex and at times rather difficult to tune and troubleshoot.

Fast forward a couple of years and the maker of the CRM software is now using SQL Server as their reporting and analytics platform and my client is struggling with supporting their existing Cognos implementation as they don't quite have the expertise to extend (modify existing and create new reports) or tune their solution and they would like to incorporate cubes.

Initially I thought I would build them a data mart. From that I would create a relational Cognos model as well as build out an SSAS database and cube(s). Then I thought, wouldn't it be great if I could simply build the cubes and have Framework Manager, Report Studio, and Analysis Studio use the cube(s) instead of having to maintain both the cubes and the relational model.

I got as far as building the mart, ETL, configuring an SSAS 2005 data source and creating a model on the SSAS cube. Then I began to port the existing reports. Well these reports used some filters in complex filters. I attempted to recreate these filters in the new model. No can do. It simply isn't supported. Then I started playing around with basic reports using the new model. At times the report was unresponsive. I then began looking around the message boards only to find that there were quite a number of people attempting to resolve performance issues with Cognos 8.3 and Analysis Services cubes with no positive responses. This was the point at which I threw in the towel.

So tomorrow I move on. I have some nice cubes created that will be used with Excel 2007 and hopefully a little bit with Analysis Studio for those that can't authenticate using AD. But, I'll have to define a relational model for the report conversion process.

An alternative would be to implement SSRS and recreate all the reports as well as create a custom authentication extension but that would be too costly at this time. Another alternative would be to create Cognos Transformer cubes but the client does not have a license for this product and do not want to pay for it, nor do they have the expertise to support it.

It's not an ideal solution (a relational FM model and AS cubes) and will require additional maintenance (two metadata models) but it will provide a much higher performing system with analytics capabilities and a much simpler and higher performing model from which to write reports.

Process More Than One SSAS Object At a Time in SSMS

Durrr . . . sometimes I discover something so simple all I can say is "durrrrrr". I've been working with SSAS for over 2 years now and never knew that you could process more than one SSAS object at a time using SQL Server Management Studio. In case anyone else is as dense as me . . .

Simply open the Object Explorer Details window if it's not already open. Click on the folder containing the objects to be processes. Select multiple objects in the browser and initiate processing. That simple. I'm not sure why SSMS doesn't allow the selection of multiple objects in the Object Explorer tree control.

What's That Confounded Character?? (ASCII and CHAR Functions)

I was on my way back from the kitchen at my client's site when I passed the office of one of their developers. We made momentary eye contact and I could see that look of desperation. She immediately called out my name and said something like, "Dave! I need your help. You're smart." I can't resist that kind of flattery so I immediately put down my yogurt and pulled up a chair.

It seemed like a simple problem. She just wanted to take part of a string from a larger one - a substring. Below are examples of the complete strings. They were created from some kind of a flat file extract from a legacy system. Assume the column name is PRODVAL.

ymkez 11/30/2009 456.8
nipin+ 11/30/2009 432.90
gapter- 11/30/2009 543.12

She simply wanted the first part of the string - up to the "space". I don't recall her logic at this point but it used the T-SQL SUBSTRING and PATINDEX functions.

I began to re-write her query. I came up with the following.
    SELECT SUBSTRING(PRODVAL, 1, CHARINDEX(' ', PRODVAL) - 1) FROM table

Well, it didn't work. To troubleshoot the problem I rewrote the above query to as follows.
     SELECT CHARINDEX(' ', PRODVAL) FROM table

The query returned a whole long list of zeros. Something was obviously wrong. I could see the space. So why couldn't the CHRINDEX function? Next step, figure out what the "space" character actually is. The next query helped determine that.
    SELECT ASCII(SUBSTRING(PRODVAL,6,1) FROM table

The first row in the table had a PRODVAL value with a "space" in the 6th position so the above query returned the ASCII value of this 6th character. It returned a 9 - Tab.

Ok great. So now I knew we were looking for a  tab, not a space. So, going back to our original query, we can modify it to use the CHAR function to look for the tab.
    SELECT SUBSTRING(PRODVAL, 1, CHARINDEX(CHAR(9), PRODVAL) - 1) FROM table

Voila! Using the ASCII function we identified the mysterious character and then used the CHAR function to convert the ASCII code for Tab (9) to a character so that we could use it in a SUBSTRING function.

Is Everything OK?

Is it just me or is there still a bit of a lack of an overall SQL Server BI ecosystem? Don't get me wrong, I am a big proponent of SQL Server BI stack and there are a lot of nice integration points but I think there is still a lot of work to be done to tie it all together.

One of my biggest gripes is related to where to look when there seems to be an issue with a BI solution that uses SSIS, SSAS, and SSRS. And what if you just want to know if things have been humming along. We can implement all the email notifications in the world but what if your email system is unreliable (as it was with a recent client). Hence the question, "Is everything OK?"

I'm pretty well versed in exception handling and logging and can probably set my brain to autopilot and find an error message in a log on one of 10 servers without actually thinking about it but what about everyone else? There are other stakeholders such as more junior developers or even senior level people that are unfamiliar with SQL Server. How about the business? An analyst might see something fishy in the data and want to know if last night's load was successful. Do they have to call me or someone else on the development team? In many cases yes. But why? That kind of a dependency is unhealthy.

That's where the SQL Server BI Health Monitor Dashboard comes in. The SQL Server Health what? Don't fret if you haven't heard of this. I'd be suprised, no maybe disturbed, if you had. The SQL Server Health Monitor Dashboard was something I recently put together for a client that was unable to obtain a seasoned SQL Server BI developer but needed to continue to support the solution I built for them without having to rely on getting a hold of me.

The dashboard uses all out-of-the-box data sources including some system tables, the SSIS log table created by the SQL Server log provider, and the SSRS tables found in the ReportServer database to tie together a majority of the data needed to determine where in the SQL Server BI ecosystem your problem lives thus helping fix the problem that much faster.

I've posted the source code on Codeplex. You can find it @ http://bihealthdashboard.codeplex.com/. It's not exactly plug and play at this point. Connection strings need to be modifed and you might have to create the DimDate table if you don't already have a table of dates accessible. It's also geared towards a BI solution that has a single SQL Server Agent job step that calls a master SSIS package that in turn calls all other SSIS packages and a single SQL Server Agent job step that process your SSAS database. If that doesn't sound like your system, the solution can still be valuable in that it shows you how to extract very useful process metadata from your SQL Server BI Solution.

Which leads me to my next endeavor  ->  The SQL Server Metadata Manager . . .

Schedule a Cognos Report Outside of the Cognos Platform

Download Source Code


I know, I know . . . this isn't a Cognos blog. I apologize. However, I do think this is a good little tid-bit and it can also be applied to SQL Server based on some (very high level) similarities in their architecture.

The Cognos scheduler is not extremely robust (nor is SQL Job Agent at times) and even if it was, there are times when it is appropriate to run a job using an enterprise scheduler application such as AppWorx or Tidal Enterprise Scheduler.

By exposing most functionality through web services, Cognos (and SSRS) allow you do to things such as execute a report. So, if your scheduler can make web services calls or call executables (wrap the web service calls in a console app) you can bypass the Cognos scheduler and leverage your enterprise scheduler by running the report directly (or through a proxy app such as a .NET exe wrapped web service) by your scheduler of choice.

Below is an example of this using .NET and Cognos.


using System;

using System.IO;
using System.Web.Services.Protocols;
using cognosdotnet_2_0;
using System.Configuration;

namespace InformationCollaboration.Cognos.Utilities

{

class ExecuteCognos8BIJob
{

[STAThread]
static int Main(string[] args)
{
    string serverHost = System.Configuration.ConfigurationSettings.AppSettings["serverHost"];
    string serverPort = System.Configuration.ConfigurationSettings.AppSettings["serverPort"];

searchPathSingleObject jobPath = new searchPathSingleObject();
jobPath.Value = System.Configuration.ConfigurationSettings.AppSettings["jobPath"];

// Parse the command-line arguments.
bool exit = false;
string jobName = string.Empty;

for (int i = 0; i < args.Length; i++)
{
if (args[i].CompareTo("-job") == 0){
i++;
jobName = args[i];
jobPath.Value += "jobDefinition[@name='" + jobName + "']";
} else {
Console.WriteLine("Unknown argument: {0}\n", args[i]);
showUsage(jobPath.Value);
exit = true;
}
}

if (!exit)
{
try
{
string Server_URL = "http://" + serverHost + ":" + serverPort + "/p2pd/servlet/dispatch";

jobService1 jobService = new jobService1();
jobService.Url = Server_URL;

Console.WriteLine("Logging on.");
setUpHeader(jobService);

parameterValue[] parameters = new parameterValue[] { };
option[] runOptions = new option[] { };

asynchReply res = jobService.run(jobPath, parameters, runOptions);
Console.WriteLine("Show usage.");
Console.WriteLine("Successfully executed job " + jobName);
writeMessageToLog("Successfully executed job " + jobName, false);
return 0;
}

catch (SoapException ex)
{
Console.WriteLine("SOAP exception!\n");
Console.WriteLine(ex.Message);
writeMessageToLog(ex.Message, true);
return -1;
}
catch (Exception ex)
{
Console.WriteLine("Unhandled exception!");
Console.WriteLine("Message: {0}", ex.Message);
Console.WriteLine("Stack trace:\n{0}", ex.StackTrace);

writeMessageToLog("Unhandled exception!\n" + "Message: " + ex.Message + "\n" + "Stack trace:\n" + ex.StackTrace + "\n", true);
return -1;
}
} // if (!exit)
else
{
return -1;
}
} // Main

///

/// This function will prepare the biBusHeader
///
/// If no error was reported, then logon was successful.
///

static void setUpHeader(jobService1 jobService)
{
// Scrub the header to remove the conversation context.
if (jobService.biBusHeaderValue != null)
{
if (jobService.biBusHeaderValue.tracking != null)
{
if (jobService.biBusHeaderValue.tracking.conversationContext != null)
{
jobService.biBusHeaderValue.tracking.conversationContext = null;
}
}
return;
}

// Get authentication info from config file
string onyxUserId = System.Configuration.ConfigurationSettings.AppSettings["onyxUserId"];
string onyxPassword = System.Configuration.ConfigurationSettings.AppSettings["onyxPassword"];
string onyxApplication = System.Configuration.ConfigurationSettings.AppSettings["onyxApplication"];
string onyxSite = System.Configuration.ConfigurationSettings.AppSettings["onyxSite"];
string CAMNamespace = System.Configuration.ConfigurationSettings.AppSettings["CAMNamespace"];

// Set up a new biBusHeader for the "logon" action.
jobService.biBusHeaderValue = new biBusHeader();
jobService.biBusHeaderValue.CAM = new CAM();
jobService.biBusHeaderValue.CAM.action = "logonAs";
jobService.biBusHeaderValue.hdrSession = new hdrSession();

formFieldVar[] ffs = new formFieldVar[5];
ffs[0] = new formFieldVar();
ffs[0].name = "onyxUserId";
ffs[0].value = onyxUserId;
ffs[0].format = formatEnum.not_encrypted;
ffs[1] = new formFieldVar();
ffs[1].name = "onyxPassword";
ffs[1].value = onyxPassword;
ffs[1].format = formatEnum.not_encrypted;
ffs[2] = new formFieldVar();
ffs[2].name = "onyxApplication";
ffs[2].value = onyxApplication;
ffs[2].format = formatEnum.not_encrypted;
ffs[3] = new formFieldVar();
ffs[3].name = "onyxSite";
ffs[3].value = onyxSite;
ffs[3].format = formatEnum.not_encrypted;
ffs[4] = new formFieldVar();
ffs[4].name = "CAMNamespace";
ffs[4].value = CAMNamespace;
ffs[4].format = formatEnum.not_encrypted;

jobService.biBusHeaderValue.hdrSession.formFieldVars = ffs;
}

static void showUsage(string jobPath)
{
Console.WriteLine("Run a Cognos 8 BI job.\n");
Console.WriteLine("usage:\n");
Console.WriteLine("-job searchPath Search path in Cognos Connection to a job.");
Console.WriteLine(" " + jobPath);
}

static void writeMessageToLog(string message, bool error)
{
try
{
string errorLogPath = System.Configuration.ConfigurationSettings.AppSettings["errorLogPath"];
errorLogPath += "CognosJobServiceLog_" + DateTime.Today.ToString("dd-mm-yy") + ".txt";
if (!File.Exists(errorLogPath))
{
File.Create(errorLogPath).Close();
}
using (StreamWriter w = File.AppendText(errorLogPath))
{
string logText;
if (!error)
{
logText = message;
}
else
{
logText = "ERROR: " + message;
}
w.WriteLine("{0} {1}",DateTime.Now.ToString(), logText);
w.Flush();
w.Close();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
}

Download Source Code

SQL Server - What's My Version/Edition?

Here's a quick way to find out what version of the SQL Server engine you have installed.

SELECT 
    SERVERPROPERTY('productversion') AS ProductVersion,
    SERVERPROPERTY ('productlevel') AS ProductLevel,
    SERVERPROPERTY ('edition') AS Edition

Keep in mind that it is possible to have a SQL Server running the Enterprise edition of the SQL Server engine but the Standard edition of Analysis Services.