Years ago I was studying how to use Power Pivot and wanted a large dataset to test. I discovered the Public Use Microdata Sample data (Public Use Microdata Sample (PUMS)). This data provides curated results from millions of questionnaires sent out each year by the US Census bureau. These files are generated from the survey […]
Public Use Microdata Sample (PUMS) for data engineers Read More »
In a previous post, I introduced my readers to the data.gov (Data.gov a catalog for USA data). Why should a data engineer be interested in government data? Our plates are full with our own company’s data. We also have so much technology that we need to learn. Where is the time for such a side
Why should a data engineer be interested in government data? Read More »
The USA government produces and gathers a lot of data, but how to find it? Over the last decade, the United States executive branch and later Congress have required the government to publish the data on the web. data.gov provides a search engine for accessing that data. The OPEN Government Data Act makes Data.gov a
Data.gov a catalog for USA data Read More »
This week I set up a a user in our SQL Server. To save time, I simply assigned all of the permission that user needed. Then I remembered how many times I have been burned by this shortcut. It is so much easier in the long run to create a role in SQL Server, assign
SQL Permission Roles versus User Read More »
I take time for granted, especially time zones. When asked if a certain time works for a meeting, I assume that the time zone is the same as the one I am in. That logic began to fail when I began working remote with people all over the United States and the world. I am
Time in application and database design Read More »
I am a relative newbie with SSAS Tabular. I needed to find the Row Level Security rules for a role in our SSAS instance. I have experimented with Row Level Security using Visual Studio, but Visual Studio is not set up yet in our environment. However, I learned it is quite easy to find this
Reading SSAS Tabular Row Level Security in SSMS Read More »
A URL can be broken up into multiple parts. 1 If you break up the URL into its parts, how do you put all the pieces together to form the full URL in Azure Synapse Analytics (#AzureSynapse) pipelines or Azure Data Factory? This post gives an example of how to create a valid URL from
Azure Synapse Analytics pipeline expression to create dynamic URL Read More »
One of the things that messes up processing delimited files is when the delimiter character is actually part of the data rather than acting as a delimiter. So look at the following: The second line has 4 commas but the first has only 3 commas. The second comma is actually part of the data: One
Selective text qualifiers in SSIS Read More »
When using dynamic SQL there is always the possibility that someone will inject SQL that does harm. They can put queries that can drop objects or can create sysadmin logins. Running the scripts as low permission users can reduce the potential harm. But that takes time to research and is easy to cut. You can
Limiting permissions with “Execute as” when using dynamic SQL Read More »