开发者

load a comma delimited flat file into SQL server table using ssis script task

开发者 https://www.devze.com 2023-03-10 00:51 出处:网络
I want to load the data from a comma delimited table into a temp table on sql server. I am using this code and it is working great. But since it is a \",\" delimited file, if any field in the file con

I want to load the data from a comma delimited table into a temp table on sql server. I am using this code and it is working great. But since it is a "," delimited file, if any field in the file contains ',' then this code is not working. as in the replace function that "," is also replace. Any help

Imports System Imports System.Data Imports System.Math Imports Microsoft.SqlServer.Dts.Runtime Imports System.IO Imports system.Data.OleDb Imports Microsoft.SqlServer.DTSRuntimeWrap

Public Class ScriptMain

' The execution engine calls this method when the task executes.
' To access the object model, use the Dts object. Connections, variables, events,
' and logging features are available as static members of the Dts class.
' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
' 
' To open Code and Text Editor Help, press F1.
' To open Object Browser, press Ctrl+Alt+J.

Public Sub Main()
    Dts.TaskResult = Dts.Results.Failure
    Dim strFilePath As String = Dts.Variables("FilePath").Value.ToString
    Dim strCurrentZipFile As String = Dts.Variables("CurrentZipFile").Value.ToString
    Dim strConn As String = IO.Path.GetFileNameWithoutExtension(Dts.Variables("FilePath").Value.ToString)
    Dim strFields() As String = Dts.Variables("FilePath").Value.ToString.Split(",".ToCharArray())

    'Dts.Connections.Item(strConn).ConnectionString = strFilePath
    Dts.Connections.Item("EmpInfo").ConnectionString = strFilePath
    Dts.Variables("CurrentRawFile").Value = IO.Path.GetFileName(strCurrentZipFile)
    ' MsgBox(Dts.Variables("CurrentRawFile").Value)
    Dts.TaskResult = Dts.Results.Success


    ' The execution engine calls this method when the task executes.
    ' To access the object model, use the Dts object. Connections, variables, events,
    ' and logging features are available as static members of the Dts class.
    ' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
    ' 
    ' To open Code and Text Editor Help, press F1.
    ' To open Object Browser, press Ctrl+Alt+J.


    Dim cm As Con开发者_如何学JAVAnectionManager

    Dim con As OleDbConnection
    Dim cmd As New OleDbCommand()
    ' myADONETConnection = DirectCast(TryCast(Dts.Connections("Polldata").AcquireConnection(Dts.Transaction), SqlConnection), SqlConnection)

    '  MsgBox(myADONETConnection.ConnectionString, "PollData")

    Dim line1 As String = ""
    'Reading file names one by one
    Dim SourceDirectory As String = Dts.Variables("FilePath").Value.ToString
    cm = Dts.Connections("Polldata")
    Dim cmParam As Wrapper.IDTSConnectionManagerDatabaseParameters90
    cmParam = CType(cm.InnerObject, Wrapper.IDTSConnectionManagerDatabaseParameters90)
    con = CType(cmParam.GetConnectionForSchema(), OleDb.OleDbConnection)


    cmd.Connection = con
    'MsgBox(Dts.Variables("FilePath").Value.ToString)
    ' TODO: Add your code here
    '  Dim fileEntries As IO.DirectoryInfo = New IO.DirectoryInfo(SourceDirectory)
    ' MsgBox(fileEntries)
    ' For Each fileName As String In fileEntries.GetFiles()
    ' do something with fileName
    ' MsgBox(fileName)
    Dim columname As String = ""


    'Reading first line of each file and assign to variable
    Dim file2 As New System.IO.StreamReader(Dts.Variables("FilePath").Value.ToString) '(fileName)

    'Dim filenameonly As String = (((fileName.Replace(SourceDirectory, "")).Replace(".txt", "")).Replace("\", ""))
    'Create a temporary table 
    line1 = (" IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].tmp_empinfo" & "') AND type in (N'U'))DROP TABLE [dbo].tmp_empinfo" & " Create Table dbo.tmp_empinfo" & "(" & file2.ReadLine().Replace(",", " VARCHAR(100),") & " VARCHAR(100))").Replace(".txt", "")

    file2.Close()

    ' MsgBox(line1.ToString())
    cmd.CommandText = line1
    cmd.ExecuteNonQuery()


    'MsgBox("TABLE IS CREATED")

    'Writing Data of File Into Table
    Dim counter As Integer = 0
    Dim line As String = ""

    Dim SourceFile As New System.IO.StreamReader(Dts.Variables("FilePath").Value.ToString) '(fileName)
    While (InlineAssignHelper(line, SourceFile.ReadLine())) IsNot Nothing

        If counter = 0 Then
            columname = line.ToString()
            ' MsgBox("INside IF")
        Else

            ' MsgBox("Inside ELSE")
            Dim query As String = "Insert into dbo.tmp_empinfo" & "(" & columname & "  VALUES('" & line.Replace(",", "','").Replace("""", "") & "')"

            'Dim query As String = "Insert into dbo.tmp_empinfo" & "(" & columname & "  VALUES(" & strFields.ToString & ")"

            ' Dim query As String = "BULK INSERT dbo.tmp_empinfo FROM '" & strFilePath & "' WITH " & " ( " & " FIELDTERMINATOR = '|', " & " ROWTERMINATOR = '\n' " & " )"


            MsgBox(query.ToString())

            cmd.CommandText = query
            cmd.ExecuteNonQuery()
        End If


"I want to load the data from a comma delimited table into a temp table on sql server". Are you saying you basically already have a column in a table in a database that contains the data in a comma separated list? For instance,

SELECT column_name FROM schema.table

outputs something like some_data, more_data, even_more_data, even,more_data? And your problem is that the text isn't quoted, so some of the rows end up having extra phantom columns when you try to load it in your destination?

If this is the problem, then I would recommend introducing quoted identifiers in your source data before it is loaded into your source table. Meaning, whatever process that imports the data into that table needs to be fixed so that you don't have to deal with this kind of problem. If that can't be done, then you will have to build logic into your script component or sql select statement to split it out appropriately. The only way to fix the problem at this point would be to fix the data.

Did I misunderstand your intent or does this answer your question?

0

精彩评论

暂无评论...
验证码 换一张
取 消