SQL

SQL

Wednesday, February 17, 2016

No Data Dictionary, No Problem!!!!

        So, you know how this goes, management gets dazzled into buying a new application. All they know is that it creates beautiful reports that will revolutionize their business. What they don't know from the 50,000 foot view from the top is that the majority of the reports contain absolutely no detail, and to get to the detail takes numerous clicks of the mouse. And to add an extra degree of difficulty, the reports are PDFs.

        So you have an idea, you are pretty savvy at SQL, why don't you get access and create some scripts so you can pull the data directly from the database. After jumping through a few hoops with the DBA, you finally get access. You get in, and what do you see, about 100 tables with naming conventions you know nothing about. Yep, this ain't no data warehouse with names that align to your normal business terminology.  No No No, you're in developer water now, the deep end with no resources. So now you have to figure out how the tables relate to one another.

       You could do numerous trial and errors, but you're smart. You know there are already people out there like me that have already gone through this exact dilemma, and you are exactly right. So below is a script that I use when I know nothing about a database. The script below will provide the Foreign Keys in the tables that will allow you to determine how they relate. I got this from Pinal Dave.


SELECT t.name AS FKTableName
   , fk.name AS NameOfForeignKey
   , pc.name AS FKColumn
   , rt.name AS ReferencedTable
   , c.name AS ReferencedColumn
FROM sys.foreign_key_columns AS fkc
INNER JOIN sys.foreign_keys AS fk ON fkc.constraint_object_id = fk.object_id
INNER JOIN sys.tables AS t ON fkc.parent_object_id = t.object_id
INNER JOIN sys.tables AS rt ON fkc.referenced_object_id = rt.object_id
INNER JOIN sys.columns AS pc ON fkc.parent_object_id = pc.object_id
   AND fkc.parent_column_id = pc.column_id
INNER JOIN sys.columns AS c ON fkc.referenced_object_id = c.object_id
   AND fkc.referenced_column_id = c.column_id

  in addition, use this script below to identify tables with no data, so you don't have to waste time with them.

SELECT
    t.NAME AS TableName,
    p.rows AS RowCounts
FROM
    sys.tables t
INNER JOIN
    sys.partitions p ON t.object_id = p.OBJECT_ID
WHERE
    t.NAME NOT LIKE 'dt%'
    AND t.is_ms_shipped = 0
    AND p.rows = 0
GROUP BY
    t.Name, p.Rows
ORDER BY

    t.Name

Thursday, July 16, 2015

TRANSFORM AND VISUALIZE DATA WITH POWER BI

Transform and Visualize Data with Power BI
Most of us are overwhelmed with data from all the different applications that we use on a daily basis. Bringing all the data together is often a very time-consuming and sometimes a challenging process. Even further, attempting to analyze and visualize the data poses new challenges that is sometime difficult or impossible to overcome. Now with Power BI this can all be made very simple. Individuals, ranging from novice information workers to advanced IT professionals can quickly and easily transform, analyze and visualize data using a single tool.
Location
Date
Discount Code
Registration Link
Atlanta, GA
September 26, 2015
FHT_SQLSAT1
Orlando, FL
October 9, 2015

Details coming soon
Houston, TX
To Be Announced

Details coming soon




In this course we will work through four main topics:
  • Shaping and Transform Data from Different Sources
  • Create a Single Data Model
  • Visualizing Data with Reports
  • Using the Power BI Service to Create and Share Dashboards
In this full-day workshop, not only will you gain insight into all the new features of Power BI, but you will also walk-through a hands-on lab that helps you experience many facets of the product. The following is the full agenda:
Agenda
8:30 – 9:00           Breakfast and Networking
9:00 – 9:30           Introduction to Power BI
9:30 – 10:45         Importing Data from:
  •                                Excel Files
  •                                Access Databases
  •                                Comma-delimited Files
10:45 – 11:00      Break
11:00 – Noon      Shaping and Transforming Data
  •                               Renaming Queries and Columns
  •                               Formatting Data
  •                               Adding columns
  •                               Merging Data Sources
Noon – 12:45      Working Lunch (Power BI Data Model and Reporting Concepts)
12:45 – 2:00        Building the Data Model
  •                               Hiding Tables and Columns
  •                               Adding Calculated Columns
  •                               Adding Measures
  •                               Creating Relationships
2:00 – 3:15           Visualizing Your Data
  •                               Creating the First Report (Bar Charts, Line Charts, Slicers, Tree Maps, etc…)
  •                               Filtering Data (Page and Visualization)
  •                               Formatting the Report (Titles, Headers, Adding Images, Color Scheming, etc…)
3:15 – 3:30           Break
3:30 – 4:30           Using the Power BI Service
  •                               Uploading the Power BI Desktop File
  •                               Creating a Report
  •                               Creating and Organizing a Dashboard
  •                               Using Q&A
4:30 – 5:00           Questions and What’s Next
In addition, you will be given access to videos that walk you through each module of the lab step-by-step that can be viewed later at your convenience.
Registration is open for the Atlanta event, but room is limited so sign up today!
See you in class,


Wednesday, November 6, 2013

Working with DTS Packages in SQL 2005

Once you have a repetitive task, that requires the same manipulation tasks to clean up your data, you will need to start using a combination of DTS packages and Stored Procedures.  A store procedure is nothing more than a query that is saved that you can call when needed. Stored Procedures are located in the Programmability portion of the SQL database.  Your procedure can be as simple or complex as you need it to be.  The Stored Procedure below is used to Delete data from a table

STORED PROCEDURE
USE [Dashboard]
GO
/****** Object:  StoredProcedure [dbo].[sp_Clear_Current_Dashboard]    Script Date: 11/06/2013 17:38:13 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO




CREATE  PROCEDURE [dbo].[sp_Clear_Current_Dashboard]
AS

SET NOCOUNT ON

Delete from DATA_IMPORT_Q4_2012
Where not indicator_id in('13','21','34','35','61','200')

Once the Stored Procedure is created, you don’t need to save it. If at another time you need to update the procedure, right click on the Stored Procedure and select modify. It will open as ALTER PROCEDURE instead of CREATE.

CALL STORED PROCEDURE
To execute the Stored Procedure you simply write. That sp is for Stored Procedure.
execute [dbo].[sp_Clear_Current_Dashboard]



Thursday, April 4, 2013

Statistical Sampling in SQL


 It’s been awhile since I have blogged, I have been extremely busy working with a few other SQL Server enthusiast working on the next great application, but that’s for another time. Today I want to share with you a simple way to pull a statistical sample using SQL. Using the code below, you can pull a quick sample


Select TOP 10  - The number of samples you want to pull back of your criteria


From – The Table your information is coming from

Order By NEWID() – I don’t want to cause any confusion by going into this function, just know that you need it. If you want to know more, start at this site below http://sqlmag.com/sql-server/change-behavior-rand-and-newid-sql-server-2005


Good luck

Curt

Saturday, February 2, 2013

Working with Dates


As you begin to work more with data, you begin to understand the importance of working with dates. Data really has no meaning unless it is specific. As you begin to work with WHERE clauses, you are going to want to drill down to specific time frames.
You may want to see data within a specific date range, in those cases your WHERE clauses would look like this:
Where [Service Date] between '01-01-2013' and '12-01-2013'
Or you could have simply used DATEPART
Where DATEPART(YEAR,[Service Date]) = '2013'

If you will be using a script on a regular basis that may require a few date changes in several parts of the code, you may want to use DECLARATIONS. Variables are declared in the body of a batch or procedure with the DECLARE statement and are assigned values by using either a SET or SELECT statement. 
In our example declarations will allow you to set what the date will be at the beginning of the statement, and that one date will be referenced when ever a date range is needed.
DECLARE @FromDos datetime,
        @ThruDOS datetime

SET @FromDos        = '01/01/2012' 
SET @ThruDOS        = '12/31/2012' 

So now, when a date is needed, you simply insert @FromDos and/or @ThruDOS.

Where [Service Date] between   @FromDos  and @ThruDOS               

Try it and see how it works for you.

Thursday, January 24, 2013

Working with Bad Data

When you are dealing with data from multiple sources, it will not always be in the correct format you need it. In this example we will look at converting data types. There are many different data types, but those I come in the most contact with dealing with healthcare data are varchar, nvarchar and datetime.
When you create or import a table, you can see what you data types are when you expand the fields in your table. Its very important that your data types are correct, especially if those are the fields that will be used in your joins.
In this example we will be changing our date from nvarchar to datetime, and using the replace function.
In the dataset below, we need the date in a more usable format and be removing the decimal in the diagnosis codes.




















Using CAST allows you to select what you want the data in the field to be. In this instance we want to change the data to DATETIME. At the same time, we are using the REPLACE function to strip our diagnosis codes of their decimal.

Select Distinct

 CAST ([ServiceDate] as Datetime) as [From_Service_Date]

,[Diag Code] = CASE WHEN ltrim(rtrim(DXCode)) Is NULL THEN ''                
               ELSE REPLACE (ltrim(rtrim(DXCode)),'.','') END    


From YOUR TABLE


 Try this and see how it works for you.

Tuesday, January 22, 2013

Adding in filters

    We left off with a very simple Select Statement that will pull all of the data without exception. But what about when you want to see specific data. At that point you want to add a filter, or in the case of SQL, you need to add a WHERE clause. The WHERE is the place in your statement you can specify exactly what you want SQL to show. There are many ways to filter your information, this is one of many we will cover. This is where you first start t get your feet wet with specifics.

    In the statement below we will add a very simple where condition. Once again, healthcare data is all about specifics, so lets start with a few of the details we like to see in healthcare analytics. (I plan on building a database with some examples of the type of data we want to see, but it takes time to make up phantom data)


SELECT  HEALTHPLAN

,[MEMBERIDTOUSE]
,[First Name]
,[Last Name]
,CONVERT (varchar (10),[enc_Date], 101) as [From Date]
,CONVERT (varchar (10),[enc_Date], 101) as [To Date]
,ProviderType1= ltrim(rtrim('20')),Diag1= CASE WHEN ltrim(rtrim(DX_Code)) Is NULL THEN ''
ELSE REPLACE (ltrim(rtrim(DX_Code)),'.','') END
,ProviderType2=''
,Diag2= ''
,ProviderType3=''
,Diag3= ''
,ProviderType4=''
,Diag4= ''
,ProviderType5=''
,Diag5= ''
,ProviderType6=''
,Diag6= ''
,ProviderType7=''
,Diag7= ''
,ProviderType8=''
,Diag8= ''
,ProviderType9=''
,Diag9= ''
FROM DATABASE_NAME.dbo.TABLE_NAMEwhere MEMBERIDTOUSE is not null and DX_Code is not null


 So in this statement we only want to those members that have data in the fields. This statement will exclude emmty field spaces for MEMBERIDTOUSE and DX_Code. There is alot in this script that we can cover, so we will continue to point back to it. Thats it for today, short and sweet. (That's what she said)


 FYI, we use [ ] when the field names are not connected, although I sometimes use them regardless of how the field is labeled.